I was able to reduce the size of my texture folder from 850 meg to 350 meg by resizing texture pics on monsters and objects. Not on icon atlas or pics for cinematics though. Resizing something from 1024x1024 to 400 by 400 pixels reduces file size alot and still maintains good detail.
My overal dungeon folder size is about 800 meg. I have 3 gig of ram and I still can not export it into dat file. It says not enough memory.
If RAM were the issue, virtual memory would solve it. I would adjust those settings first. Adjust both directions, too much and hd runs out of space (thus memory), too little, and you just run out of memory. If that is not the problem we likely need to hear from the Almost Human guys about possible limitations in the engine. I've been through this loooong and arduous process with you, and you WILL get it exported and playable or a part of me will die. You have a vision so much larger than the rest of us, that I INSIST it come to fruition. I refuse to meet my maker without seeing what you have pulled off. (I went too deep didn't I?, Bah! It's the truth!)
Currently conspiring with many modders on the "Legends of the Northern Realms"project.
"You have been captured by a psychopathic diety who needs a new plaything to torture." Hotel Hades
I expect you're running out of graphics memory, not main system memory. If your texture folder is 400Mb say, then those textures will be a little bit bigger in memory. Your laptop doesn't have a dedicated graphics card (it looks this way from the crash log), and I believe the X4500 on your CPU has a maximum of 1024Mb of shared GPU memory.
Suggestions / Thoughts
1. Each dimension of every texture should be to a power of two (2,4,8,16,32,64,128,256,512,1024). I don't know for sure, but most games will use the standard DirectX texture loading mechanism, which will automatically resize it to a power of 2 anyway, so if your texture is 200x400 pixels, it will actually be loaded as a 256x512 texture in RAM (using more memory than your DDS file takes up on disk).
2. The game will obviously be using a lot of memory for all of the objects you have loaded in the game. I don't know if it does anything clever with the wallsets on the other levels or if it just loads everything in your dungeon into memory, but you can probably imagine that if you have a lot of different monsters and objects and wall pieces etc all on a given level, that's a lot of memory already used up by the game assets.
3. While your DDS files may total 300Mb, and your meshes 30Mb say, there's likely a bit of overhead at least on your GPU memory (especially if textures are not a power of 2). For that 330Mb of files, it might be using 360Mb of GPU memory for random reasons - possibly even more.
4. I can only really suggest that you downgrade some textures further (and possibly any meshes if you have custom models with LOADS of vertices, although, they don't tend to chew up much memory in comparison to textures). Small items that you pick up might not need normal maps and textures could probably be 128x128 in size. Wallsets and such are what really eat up memory - as you need big ass normal maps for them to look good. You might want to consider using one normal map across multiple wallsets if you're simply recolouring walls etc.
In a sense - it's actually good that you're running into these problems on your machine. Otherwise, you'd start getting bug reports and not know why!
Of course, I could be barking up the wrong tree, but I'd place a small bet on this is what's happening.
My Grimrock Projects Page with links to the Grimrock Model Toolkit, GrimFBX, Atlas Toolkit, QuickBar, NoteBook and the Oriental Weapons Pack.