Common Serialization (Saving) Pitfalls

Ask for help about creating mods and scripts for Grimrock 2 or share your tips, scripts, tools and assets with other modders here. Warning: forum contains spoilers!
minmay
Posts: 2657
Joined: Mon Sep 23, 2013 2:24 am

Common Serialization (Saving) Pitfalls

Post by minmay » Tue Jan 27, 2015 4:10 am

I have seen a lot of mods in LoG2 and LoG1 that are broken because of a lack of understanding of serialization - what happens when the player saves the game. This guide is written for Grimrock 2, but mistakes 1 and 2 also apply to Grimrock 1. (There are other things to be aware of with serialization in Grimrock 1, maybe I'll cover them in a different post someday.)

The 3 serialization mistakes that will break your mod

1. Storing variables in your scripts that can't be serialized
The most common - and easily avoidable - error is attempting to serialize a type that cannot be serialized. This is has its own page in the official modding guide, so there's really not much excuse for getting it wrong. If you define a variable in a script without using the local keyword - even if your definition is inside a function or other block - then it will be "global" to that script, and be permanently stored. If the player attempts to save while an unserializable type is stored in this way, the game will crash. The most common error message to result from this is:

Code: Select all

[string "Script.lua"]:0: attempt to concatenate field 'id' (a nil value)
stack traceback:
	[string "Script.lua"]: in function 'saveValue'
	[string "Script.lua"]: in function 'saveState'
	[string "GameObject.lua"]: in function 'saveState'
	[string "Map.lua"]: in function 'saveState'
	[string "GameMode.lua"]: in function 'saveGame'
...
This means you have a reference to an unserializable type somewhere. Here is one possible (not necessarily correct) explanation for why this unhelpful message appears.
Other possible errors include:
"cannot serialize table with metatable" - probably a reference to an instance of a class (like Champion)
"could not look up class name" - same as above
"cannot serialize variable of type X" - self-explanatory
"cannot serialize a function with upvalues" - generally this means you used the "local" keyword on a permanent variable which turns it into an upvalue; in Grimrock "local" may only be used for true temporary variables. See Addendum: Upvalues.
"unknown value type" - If you manage to get this error to appear at all, you should already know what it means.

As of version 2.2.4, the Grimrock editor will try to catch these errors whenever you run the dungeon preview. However, it can't catch 100% of problems. Always test saving and loading in your mods!

2. Referencing a function that needs an environment, in multiple environments
There is another, easier-to-miss issue that can arise from serialization if you have references to a function in more than one place. For example, if you have two script entities, script_entity_1 and script_entity_2:

Code: Select all

--Script 1
function abc() print("stuff") end

Code: Select all

-- Script 2
def = script_entity_1.script.abc
then you have a function with two references to it; one where it was defined in the ScriptComponent of script_entity_1, and another where it was defined in the ScriptComponent of script_entity_2. The possible problem that can arise is that functions and tables referenced in multiple places aren't guaranteed to preserve their environment across save/load. If the save code reaches script_entity_2 before script_entity_1, it will save the function with an environment of script_entity_2's ScriptComponent, not script_entity_1's ScriptComponent as it originally did. Here's a slightly more detailed example.
So if the function in script_entity_1 references any variables that belong to script_entity_1, it could break if the game is saved and reloaded; if its environment changes to script_entity_2's ScriptComponent, it won't be able to find the variables. If you're lucky it'll break in a way that causes a crash.
To keep your code safe from this problem, just follow these two rules:
1. If you want a function to use variables other than its own parameters and completely global variables, treat it similar to you would treat an unserializable item: except for its original, permanent definition, don't store any references to it. Temporary local references are still fine, obviously.
2. If you want to store references to a function in more than one place, only reference the global environment in it. If you really need the function to reference variables in a specific ScriptComponent, write it like this:

Code: Select all

function example()
  local varFromGlobalEnvironment = script_entity_1.script.var
end
but this is obviously inelegant and difficult to maintain. Note that variables in ScriptComponents are read-only to functions that are not in that ScriptComponent's environment. So if you want to change variables in your script from the global environment you also have to write setter functions for them.

You may have noticed that this behaviour can also be used to detect when the player reloads the game. The details of that are left as an exercise for the reader. (It is not a very good way to do it, however; for a better method, see minimalSaveState below.)

I mentioned that this also happens for tables. However, the environment of a table is only relevant if there is a function inside the table that requires it. This is not something you are likely to encounter.

3. Not knowing what minimalSaveState does, why it's important, and which objects have it
Objects defined as having "minimalSaveState = true" will have only minimal properties saved: their name, id, x, y, elevation, and facing. When the game is loaded, they are simply re-spawned at that position with that name and id, in their default state; any changes that were made to them are lost. Because the object is spawned again, all its components are created again, meaning that all their onInit hooks will run again.
Objects that don't have minimalSaveState will have their entire state saved, including all their components and the state of all their components, so that they are preserved exactly across save/load even if changes have been made to them. Their components' onInit hooks will not run again.
minimalSaveState should be used on objects such as walls and floors that won't change during gameplay. This is important because if you don't use minimalSaveState on these objects, the game will have to save the full state of every single instance of them - which could be tens of thousands of objects. The game can easily run out of memory when saving in this case, and even if it doesn't, it will take an unnecessarily long time.
If you are changing the object in any way - using setWorldPosition, setWorldRotation, doing anything to its components such as enabling/disabling them (even in the editor), setting wall text, etc. - and the object has minimalSaveState, then these changes won't be saved, and your mod will 100% catastrophically break as soon as the player saves and loads the game. I cannot stress this enough. Almost all released Grimrock 2 mods are broken because of their authors ignoring this simple fact. Don't become one of those authors!
If you're still confused, look at the asset pack and use it as a guide: note how objects like trees, floors, etc. that will never change state have minimalSaveState (if they didn't saving would be too expensive), and note how objects like altars and monsters don't have minimalSaveState because their state can change during gameplay.
Most importantly, before you use an object, look at its definition and especially whether it has minimalSaveState or not, so that you don't make the mistake of trying to change the state of a minimalSaveState object, or the mistake of placing thousands of instances of an object that lacks minimalSaveState. Remember that if an object's base_object has minimalSaveState, then that object has minimalSaveState too, inherited from the base_object!

Finally, minimalSaveState offers an easy way to detect when a player reloads the game: define a minimalSaveState object that has a component with an onInit hook and place one instance of it anywhere in your dungeon. Then, whenever that onInit hook runs, you know that the player just reloaded or just started the game. This is useful when you are manipulating sounds and music.

Addendum: Upvalues
As the official page says, upvalues cannot be serialized, and if the saving code reaches an upvalue, it will throw an error. The implication of this is that you should ensure any local variable is not used after its block exits. In other words, you need to avoid constructs like:

Code: Select all

function makePrinter(str)
  return function()
    print(str)
  end
end
because the function returned by makePrinter requires the upvalue str.
As long as you use local and function parameters only for actual local variables, and not for upvalues, you don't need to worry about this. Note that I can only think of two situations in either Grimrock game where the user can create an upvalue:
1. using the local keyword inside a ScriptComponent, but not inside any block. This is the example given on the official page. There is no good reason to use local here in the first place, so just don't do it.
2. a block that creates a function that uses a local variable from inside the block, like the example I gave above. There is no good reason to do this in Grimrock mods, either.
Last edited by minmay on Mon Dec 19, 2016 1:55 am, edited 11 times in total.
Grimrock 1 dungeon
Grimrock 2 resources
I no longer answer scripting questions in private messages. Please ask in a forum topic or this Discord server.

Azel
Posts: 808
Joined: Thu Nov 06, 2014 10:40 pm

Re: Common Serialization (Saving) Pitfalls

Post by Azel » Tue Jan 27, 2015 8:45 am

Good info, minmay.

I originally thought that defining a variable without the "local" notation meant that it was "global" to all Scripts on that map/level. Your post corrected me on that. I just ran a test and sure enough...

I had a script that I titled "global_script" which define a counter as: myPuzzleCounter = 0.

Then in another script entity (puzzle_script_1) a function is called with the logic: " if myPuzzleCounter == nil then doSomething() end "

I originally thought that the reason there was no error in that "if" statement was due to the fact that myPuzzleCounter was globally defined in another script entity. Yet after reading your post I went and renamed the myPuzzleCounter in the global script... and then the "if" statement in the second script still worked.

Looks like I should do this better (by using the Counter object instead), since sloppy code is bad all around - even if the game isn't crashing on a Save/Load. Good stuff.

User avatar
TheLastOrder
Posts: 104
Joined: Wed Oct 17, 2012 1:56 am

Re: Common Serialization (Saving) Pitfalls

Post by TheLastOrder » Fri Jan 30, 2015 7:06 am

Thanks a lot for this info minmay, really appreciated!!! :mrgreen:

MrChoke
Posts: 324
Joined: Sat Oct 25, 2014 7:20 pm

Re: Common Serialization (Saving) Pitfalls

Post by MrChoke » Sat Jan 31, 2015 5:55 pm

Azel wrote:Good info, minmay.

I originally thought that defining a variable without the "local" notation meant that it was "global" to all Scripts on that map/level. Your post corrected me on that. I just ran a test and sure enough...

I had a script that I titled "global_script" which define a counter as: myPuzzleCounter = 0.

Then in another script entity (puzzle_script_1) a function is called with the logic: " if myPuzzleCounter == nil then doSomething() end "

I originally thought that the reason there was no error in that "if" statement was due to the fact that myPuzzleCounter was globally defined in another script entity. Yet after reading your post I went and renamed the myPuzzleCounter in the global script... and then the "if" statement in the second script still worked.

Looks like I should do this better (by using the Counter object instead), since sloppy code is bad all around - even if the game isn't crashing on a Save/Load. Good stuff.
From the research I have done there is no way to define a truly global scope variable. I tried defining one in multiple places including init.lua for example and none worked. A variable is either local scope or "scrpt_entity" scope, thus requiring reference thru the script entity ID to access outside of that script (<script entity id>.script.<var name>).

minmay
Posts: 2657
Joined: Mon Sep 23, 2013 2:24 am

Re: Common Serialization (Saving) Pitfalls

Post by minmay » Sat Jan 31, 2015 7:24 pm

Yeah, Grimrock does not allow you to directly modify the global environment; it would be easy to screw things up. You can add fields to the global tables such as GameMode, but I really don't recommend it.
Grimrock 1 dungeon
Grimrock 2 resources
I no longer answer scripting questions in private messages. Please ask in a forum topic or this Discord server.

Azel
Posts: 808
Joined: Thu Nov 06, 2014 10:40 pm

Re: Common Serialization (Saving) Pitfalls

Post by Azel » Sun Feb 01, 2015 1:29 am

Right, I wouldn't recommend that either after working in this development environment. I don't even feel that accessing variables/functions from other script entities via direct calls to those entities a "good thing." I'm sure those who are more comfortable with LUA and the Grimrock Editor can expertly do these things as needed. For me, however, I am now looking at smarter (ie, simpler) ways to achieve things.

For example, in the situation I described above (with the global counter variable), I ended up using an actual Counter object, which is Global to the Level. Much easier, much smarter.

User avatar
Drakkan
Posts: 1317
Joined: Mon Dec 31, 2012 12:25 am

Re: Common Serialization (Saving) Pitfalls

Post by Drakkan » Thu Feb 12, 2015 9:08 pm

oh crap. just got [string "Script.lua"]:0: attempt to concatenate field 'id' (a nil value) when testing.

any ideas how to recognize where the problem is ? I have many copied scripts form forum. I would welcome some default and basic commands which are probably causing the error often so I can try find it out. For now I have searched for something like: x = spawn("dagger") but not suscess
Breath from the unpromising waters.
Eye of the Atlantis

Azel
Posts: 808
Joined: Thu Nov 06, 2014 10:40 pm

Re: Common Serialization (Saving) Pitfalls

Post by Azel » Thu Feb 12, 2015 10:01 pm

I have received similar errors and the culprit was typically the difference between:

local myItemID = item.id
vs
local myItemID = item.go.id


I haven't delved much in to the situational difference, I just know that sometimes I can use "x.ID" and other times I need "x.go.ID"

Lastly, this error creeps up on me when I use SPAWN(entity) without the final Optional Parameter that assigns an actual ID. So I will spawn an entity via Script Function, and then try to local it via FindEntity, which errors because the ID I thought I assigned was never provided.

Not sure if this applies to you, but thought it could help point in the right direction.

User avatar
Drakkan
Posts: 1317
Joined: Mon Dec 31, 2012 12:25 am

Re: Common Serialization (Saving) Pitfalls

Post by Drakkan » Thu Feb 12, 2015 11:37 pm

ok I found when game is save-crashing, but it makes no sense to me. When I kill creature named gaint_snake_4 or green_slime_2 (regular monster placed, not spawned or antyhing) and then save, game crash with string "Script.lua"]:0: attempt to concatenate field 'id' (a nil value). I do not use these monsters in any script or any trigger...
any clues ?
I have noticed I have in my dungeon monster green_slime_2, green_slime_3, green_slime_4 - missing number one. the same "issue" for snakes. could this be the problem ?
Breath from the unpromising waters.
Eye of the Atlantis

minmay
Posts: 2657
Joined: Mon Sep 23, 2013 2:24 am

Re: Common Serialization (Saving) Pitfalls

Post by minmay » Thu Feb 12, 2015 11:50 pm

Drakkan wrote:ok I found when game is save-crashing, but it makes no sense to me. When I kill creature named gaint_snake_4 or green_slime_2 (regular monster placed, not spawned or antyhing) and then save, game crash with string "Script.lua"]:0: attempt to concatenate field 'id' (a nil value). I do not use these monsters in any script or any trigger...
any clues ?
As the original post says, this means you have a reference to an unserializable type. Have you changed the monsters' definitions in any way? Do they have any connectors? Look at the scripts they're connected to and see if you're creating any unserializable references there. Also, make sure that it's actually killing the monsters that causes it, not something else.
If you're having trouble keeping track of what you're doing in scripts, you might want to adopt a coding convention such as Hungarian notation that will make it easy to automatically search for specific variable types.
Drakkan wrote:I have noticed I have in my dungeon monster green_slime_2, green_slime_3, green_slime_4 - missing number one. the same "issue" for snakes. could this be the problem ?
Definitely not.
Grimrock 1 dungeon
Grimrock 2 resources
I no longer answer scripting questions in private messages. Please ask in a forum topic or this Discord server.

Post Reply