Common Serialization (Saving) Pitfalls

Ask for help about creating mods and scripts for Grimrock 2 or share your tips, scripts, tools and assets with other modders here. Warning: forum contains spoilers!
minmay
Posts: 2768
Joined: Mon Sep 23, 2013 2:24 am

Common Serialization (Saving) Pitfalls

Post by minmay »

I have seen a lot of mods in LoG2 and LoG1 that are broken because of a lack of understanding of serialization - what happens when the player saves the game. This guide is written for Grimrock 2, but mistakes 1 and 2 also apply to Grimrock 1. (There are other things to be aware of with serialization in Grimrock 1, maybe I'll cover them in a different post someday.)

The 3 serialization mistakes that will break your mod

1. Storing variables in your scripts that can't be serialized
The most common - and easily avoidable - error is attempting to serialize a type that cannot be serialized. This has its own page in the official modding guide. If you define a variable in a script without using the local keyword - even if your definition is inside a function or other block - then it will be "global" to that script, and be permanently stored. If the player attempts to save while an unserializable type is stored in this way, the game will crash. Often the crash includes this specific, rather confusing error message:

Code: Select all

[string "Script.lua"]:0: attempt to concatenate field 'id' (a nil value)
stack traceback:
	[string "Script.lua"]: in function 'saveValue'
	[string "Script.lua"]: in function 'saveState'
	[string "GameObject.lua"]: in function 'saveState'
	[string "Map.lua"]: in function 'saveState'
	[string "GameMode.lua"]: in function 'saveGame'
...
This means you have a reference to an unserializable type somewhere.
Other possible errors you can cause with this, that make a little more sense, include:
"cannot serialize table with metatable" - All of Grimrock's classes, such as Champion and Map, have metatables, and can't be saved.
"could not look up class name"
"cannot serialize variable of type X"
"cannot serialize a function with upvalues" - See the "What's an upvalue?" section below.
"unknown value type" - It's hard to get this one unless you try to get it on purpose, but I'm listing it for completeness.

As of version 2.2.4, the Grimrock editor will try to catch these errors whenever you run the dungeon preview. However, it can't catch 100% of problems. Always test saving and loading in your mods!

It's worth repeating that you only need to worry about this for non-local variables. Variables defined as local are temporary variables and Grimrock will never try to save them.

"What's an upvalue?"
In Lua, if you define a function that uses a local variable from an enclosing scope, that function will be a closure. For example, let's say you have this in a ScriptComponent:

Code: Select all

local message = "eating whole raw fish is fine"

function showMessage()
	hudPrint(message)
end
The "showMessage" function is a closure, and the "message" variable is now an upvalue of it. Now, there are plenty of things that are convenient to program using closures, but Grimrock cannot save closures. Grimrock cannot save functions with upvalues, and it will crash if it tries to. The script shown above will crash. If you want to use a closure in a ScriptComponent, make sure the closure itself is defined as local.
If you want to store a closure in a ScriptComponent so you can use it later, too bad, you can't. You'll have to come up with a different way to implement whatever you're implementing.

2. Storing a function in multiple script environments when it needs one specific environment
There is another, easier-to-miss issue that can arise from serialization if you have references to a function in more than one place. For example, if you have two script entities, script_entity_1 and script_entity_2:

Code: Select all

--Script 1
function abc() print("stuff") end

Code: Select all

-- Script 2
def = script_entity_1.script.abc
then you have a function with two references to it; one where it was defined in the ScriptComponent of script_entity_1, and another where it was defined in the ScriptComponent of script_entity_2. The possible problem that can arise is that functions referenced in multiple places aren't guaranteed to preserve their environment across save/load. If the save code reaches script_entity_2 before script_entity_1, it will save the function with an environment of script_entity_2's ScriptComponent, not script_entity_1's ScriptComponent as it originally did. Here's a slightly more detailed example.
So if the function in script_entity_1 references any variables that belong to script_entity_1, it could break if the game is saved and reloaded; if its environment changes to script_entity_2's ScriptComponent, it won't be able to find the variables. If you're lucky it'll break in a way that causes a crash.
To keep your code safe from this problem, just follow these two rules:
1. If you want a function to use variables other than its own parameters and completely global variables, treat it similar to you would treat an unserializable item: except for its original, permanent definition, don't store any permanent references to it. Temporary local references are still fine, obviously.
2. If you want to store references to a function in more than one place, only reference the global environment in it. If you really need the function to reference variables in a specific ScriptComponent, write it like this:

Code: Select all

function example()
  local varFromGlobalEnvironment = script_entity_1.script.var
end
but this is obviously inelegant and difficult to maintain. Note that variables in ScriptComponents are read-only to functions that are not in that ScriptComponent's environment. So if you want to change variables in your script from the global environment you also have to write setter functions for them.

You may have noticed that this behaviour can also be used to detect when the player reloads the game. However, there's a better method explained in section 3 below.

3. Not knowing what minimalSaveState does, why it's important, and which objects have it
Objects defined as having "minimalSaveState = true" will have only minimal properties saved: their name, id, x, y, elevation, and facing. When the game is loaded, they are simply re-spawned at that position with that name and id, in their default state; any changes that were made to them are lost. Because the object is spawned again, all its components are created again, meaning that all their onInit hooks will run again.
Objects that don't have minimalSaveState will have their entire state saved, including all their components and the state of all their components, so that they are preserved exactly across save/load even if changes have been made to them. Their components' onInit hooks will not run again.
minimalSaveState should be used on objects such as walls and floors that won't change during gameplay. This is important because if you don't use minimalSaveState on these objects, the game will have to save the full state of every single instance of them - which could be tens of thousands of objects. The game can easily run out of memory when saving in this case, and even if it doesn't, it will take an unnecessarily long time.
If you are changing the object in any way - using setWorldPosition, setWorldRotation, doing anything to its components such as enabling/disabling them (even in the editor), setting wall text, etc. - and the object has minimalSaveState, then these changes won't be saved, and your mod will 100% catastrophically break as soon as the player saves and loads the game. I cannot stress this enough. Tons of released Grimrock 2 mods are broken because of their authors not knowing this. Don't become one of them!
The asset pack is a good guide for this: note how objects like trees, floors, etc. that are used frequently and will never change state have minimalSaveState (if they didn't saving would be too expensive), but objects like altars and monsters don't have minimalSaveState because their state can change during gameplay.
Before you use an object, look at its definition and especially whether it has minimalSaveState or not, so that you don't make the mistake of trying to change the state of a minimalSaveState object. Remember that if an object's base_object has minimalSaveState, then that object has minimalSaveState too, inherited from the base_object!

Finally, minimalSaveState offers an easy way to detect when a player reloads the game: define a minimalSaveState object that has a component with an onInit hook and place one instance of it anywhere in your dungeon. Then, whenever that onInit hook runs, you know that the player just reloaded or just started the game.
Last edited by minmay on Wed Mar 22, 2023 10:35 pm, edited 12 times in total.
Grimrock 1 dungeon
Grimrock 2 resources
I no longer answer scripting questions in private messages. Please ask in a forum topic or this Discord server.
Azel
Posts: 808
Joined: Thu Nov 06, 2014 10:40 pm

Re: Common Serialization (Saving) Pitfalls

Post by Azel »

Good info, minmay.

I originally thought that defining a variable without the "local" notation meant that it was "global" to all Scripts on that map/level. Your post corrected me on that. I just ran a test and sure enough...

I had a script that I titled "global_script" which define a counter as: myPuzzleCounter = 0.

Then in another script entity (puzzle_script_1) a function is called with the logic: " if myPuzzleCounter == nil then doSomething() end "

I originally thought that the reason there was no error in that "if" statement was due to the fact that myPuzzleCounter was globally defined in another script entity. Yet after reading your post I went and renamed the myPuzzleCounter in the global script... and then the "if" statement in the second script still worked.

Looks like I should do this better (by using the Counter object instead), since sloppy code is bad all around - even if the game isn't crashing on a Save/Load. Good stuff.
User avatar
TheLastOrder
Posts: 104
Joined: Wed Oct 17, 2012 1:56 am

Re: Common Serialization (Saving) Pitfalls

Post by TheLastOrder »

Thanks a lot for this info minmay, really appreciated!!! :mrgreen:
MrChoke
Posts: 324
Joined: Sat Oct 25, 2014 7:20 pm

Re: Common Serialization (Saving) Pitfalls

Post by MrChoke »

Azel wrote:Good info, minmay.

I originally thought that defining a variable without the "local" notation meant that it was "global" to all Scripts on that map/level. Your post corrected me on that. I just ran a test and sure enough...

I had a script that I titled "global_script" which define a counter as: myPuzzleCounter = 0.

Then in another script entity (puzzle_script_1) a function is called with the logic: " if myPuzzleCounter == nil then doSomething() end "

I originally thought that the reason there was no error in that "if" statement was due to the fact that myPuzzleCounter was globally defined in another script entity. Yet after reading your post I went and renamed the myPuzzleCounter in the global script... and then the "if" statement in the second script still worked.

Looks like I should do this better (by using the Counter object instead), since sloppy code is bad all around - even if the game isn't crashing on a Save/Load. Good stuff.
From the research I have done there is no way to define a truly global scope variable. I tried defining one in multiple places including init.lua for example and none worked. A variable is either local scope or "scrpt_entity" scope, thus requiring reference thru the script entity ID to access outside of that script (<script entity id>.script.<var name>).
minmay
Posts: 2768
Joined: Mon Sep 23, 2013 2:24 am

Re: Common Serialization (Saving) Pitfalls

Post by minmay »

Yeah, Grimrock does not allow you to directly modify the global environment; it would be easy to screw things up. You can add fields to the global tables such as GameMode, but I really don't recommend it.
Grimrock 1 dungeon
Grimrock 2 resources
I no longer answer scripting questions in private messages. Please ask in a forum topic or this Discord server.
Azel
Posts: 808
Joined: Thu Nov 06, 2014 10:40 pm

Re: Common Serialization (Saving) Pitfalls

Post by Azel »

Right, I wouldn't recommend that either after working in this development environment. I don't even feel that accessing variables/functions from other script entities via direct calls to those entities a "good thing." I'm sure those who are more comfortable with LUA and the Grimrock Editor can expertly do these things as needed. For me, however, I am now looking at smarter (ie, simpler) ways to achieve things.

For example, in the situation I described above (with the global counter variable), I ended up using an actual Counter object, which is Global to the Level. Much easier, much smarter.
User avatar
Drakkan
Posts: 1318
Joined: Mon Dec 31, 2012 12:25 am

Re: Common Serialization (Saving) Pitfalls

Post by Drakkan »

oh crap. just got [string "Script.lua"]:0: attempt to concatenate field 'id' (a nil value) when testing.

any ideas how to recognize where the problem is ? I have many copied scripts form forum. I would welcome some default and basic commands which are probably causing the error often so I can try find it out. For now I have searched for something like: x = spawn("dagger") but not suscess
Breath from the unpromising waters.
Eye of the Atlantis
Azel
Posts: 808
Joined: Thu Nov 06, 2014 10:40 pm

Re: Common Serialization (Saving) Pitfalls

Post by Azel »

I have received similar errors and the culprit was typically the difference between:

local myItemID = item.id
vs
local myItemID = item.go.id


I haven't delved much in to the situational difference, I just know that sometimes I can use "x.ID" and other times I need "x.go.ID"

Lastly, this error creeps up on me when I use SPAWN(entity) without the final Optional Parameter that assigns an actual ID. So I will spawn an entity via Script Function, and then try to local it via FindEntity, which errors because the ID I thought I assigned was never provided.

Not sure if this applies to you, but thought it could help point in the right direction.
User avatar
Drakkan
Posts: 1318
Joined: Mon Dec 31, 2012 12:25 am

Re: Common Serialization (Saving) Pitfalls

Post by Drakkan »

ok I found when game is save-crashing, but it makes no sense to me. When I kill creature named gaint_snake_4 or green_slime_2 (regular monster placed, not spawned or antyhing) and then save, game crash with string "Script.lua"]:0: attempt to concatenate field 'id' (a nil value). I do not use these monsters in any script or any trigger...
any clues ?
I have noticed I have in my dungeon monster green_slime_2, green_slime_3, green_slime_4 - missing number one. the same "issue" for snakes. could this be the problem ?
Breath from the unpromising waters.
Eye of the Atlantis
minmay
Posts: 2768
Joined: Mon Sep 23, 2013 2:24 am

Re: Common Serialization (Saving) Pitfalls

Post by minmay »

Drakkan wrote:ok I found when game is save-crashing, but it makes no sense to me. When I kill creature named gaint_snake_4 or green_slime_2 (regular monster placed, not spawned or antyhing) and then save, game crash with string "Script.lua"]:0: attempt to concatenate field 'id' (a nil value). I do not use these monsters in any script or any trigger...
any clues ?
As the original post says, this means you have a reference to an unserializable type. Have you changed the monsters' definitions in any way? Do they have any connectors? Look at the scripts they're connected to and see if you're creating any unserializable references there. Also, make sure that it's actually killing the monsters that causes it, not something else.
If you're having trouble keeping track of what you're doing in scripts, you might want to adopt a coding convention such as Hungarian notation that will make it easy to automatically search for specific variable types.
Drakkan wrote:I have noticed I have in my dungeon monster green_slime_2, green_slime_3, green_slime_4 - missing number one. the same "issue" for snakes. could this be the problem ?
Definitely not.
Grimrock 1 dungeon
Grimrock 2 resources
I no longer answer scripting questions in private messages. Please ask in a forum topic or this Discord server.
Post Reply