Since the modding forum isn't open yet, I thought I might place this here. I've worked a bit on dissecting the Grimrock.dat file, and I've got it mostly figured out - only downside is that the devs uses hashes of filenames instead of the filenames themselves, so it's hard to find out what the files are actually named.
Here's an extractor, written in Python: http://pastebin.com/w6anxSjY
Incomplete list of file names: http://pastebin.com/hupic6F4
But you might not be interested in that, you are probably more interested in custom levels. So far I've only found a very bad and stupid method of running custom levels:
1. Make a backup copy of your Grimrock.dat file
2. Use a hex editor to search for the hex bytes AF F4 D1 88 in Grimrock.dat and replace them with 00 00 00 00 (This makes the game unable to find the first level inside the .dat and forces it to search the file system instead.
3. Make the directories "assets\dungeons\grimrock" in your Grimrock game dir, and place the level named as level01.lua there.
4. Start up the game, start a new game, and play.
To get started, here's the manually decompiled level 1 map, which shows how some things are made: link removed