Script input data should be flexible

As mentioned in the last devblog, solving the script binding problem is the first focus of 1.18. To solve this problem a design on how “glue” game objects to scripts must be developed. This is a very important, core design decision of the system, maybe even more important than the scripting language itself. If the access to script data is slow compared to the core C++ code (for example, mediated by hash tables, like most interpreted languages) it could erase all the work on making NimbyScript as fast as plain C.

What is “script binding data”? Basically any data required by the script which does not come from the core C++ game objects. The C++ objects can already be exposed to NimbyScript as plain C pointers to structs, and accessed as such (with all the static safety features of NimbyScript to avoid the usual pointer problems). The C++ side will call NimbyScript functions and pass one or more of these safe pointers for the script as function parameters, but the passed data is determined by the C++ side. What if the script needs additional data? For example, what if some signal code needs to check if a given train has a tag? A first impulse would be to add such an API to the provided C++ interfaces:

// DB is the database of all player-created objects
// imagine train is passed by the C++ side for the purpose of this example
fn some_signal_script_fn(db: &DB, train: &Train, signal: &Signal) {
    let tag &= db.tags.find("Express") else { return; };
    let speed mut= 50;
    if train.tags.contains(tag) {
        speed = 100;
    }
    ...
}

The script is interested in a specific object, the tag named “Express”, so it looks it up by name, then uses it. So far this looks okay. But now the player creates a second signal, and it wants to check for a different tag, so they need to copy paste the script code and change the tag name:

fn some_signal_script_fn(db: &DB, train: &Train, signal: &Signal) {
    let tag &= db.tags.find("Local") else { return; };
    let speed mut= 50;
    if train.tags.contains(tag) {
        speed = 20;
    }
}

This not good but it could be survivable, since there’s usually only a few dozens of these functional tags, so the amount of duplication is bad but not a disaster. But what if the object to lookup is a station? Or a different signal? Medium and large saves have thousands of stations and signals, so you now need thousands of script copies, all edited to fit a very specific purpose. In the case of signals it can degenerate into having an individual script text for every signal in a save. It would also mean that renaming objects makes these scripts stop running properly, since they are looking up things by the old name.

If you had in mind the idea that NimbyScript is the same as having a little text area to input script code for every signal you create, you should very quickly dispel that notion. Recompiling this code is not performance free, and making players copy paste code thousands of times with tiny edits and having it all go to waste when something is renamed (or deleted or moved) is not workable. Therefore I made the decision it should be impossible for NimbyScript code to “find” game objects by name or explicit ID.

Instead, NimbyScript will be passed a set of function parameters it can use to call methods to access that data, and another set to serve as parameters of the script as configured by the player. The previous example then becomes:

// a null ptag could mean the player neglected to select a tag somewhere
fn some_signal_script_fn(db: &DB, train: &Train, signal: &Signal, ptag: *Tag, tag_speed: f64) {
    let tag &= ptag else { return; };
    let speed mut= 50;
    if train.tags.contains(tag) {
        speed = tag_speed;
    }
    ...
}

Then, every time the player selects this script to run for a signal, they must also select a tag in the UI, and input some speed value. I don’t have an UI for this exact pattern since as I will explain later, this particular iteration of the design was discarded, but imagine a drop down to select a script, and depending of the selected script, the UI then shows additional controls to provide values to the parameters of the function. In this case it would show a tag picker. The game remembers both the selected script and the input parameters as save data, just like any other signal option.

This idea alone of forbidding hardcoded names and IDs in scripts and forcing them to be parameters solves the issue of copy pasting scripts, but it has problems of its own. In particular it is too rigid, in how the only way to pass data to scripts from the UI is as parameters to some function call. It is very often the case some piece of data is not really associated to some function call, but to some game object instead. Something more flexible was needed, and I had a very cool idea, but before it was possible to implement, I needed to finally design and implement script defined structs.

Script data structures and the migration problem

As I mentioned earlier, most scripting languages just provide some kind of variable type hash table as the “object” type, do the language designer equivalent of ¯\(ツ)/¯, and move on. Then every access to an object field becomes hundreds of instructions and multiple L1 cache misses. Instead, NimbyScript is going to provide C like structs, with typed fields, O(1) access and memory locality.

What are structs? A struct is a sequence of named fields to store data somewhere in memory. For example:

struct Params {
    tag_id: ID<Tag>,
    speed: f64,
}

Then, the previous function example could take a Params reference as a parameter:

fn some_signal_script_fn(db: &DB, train: &Train, signal: &Signal, params: &Params) {
    let tag &= db.tags.view(params.tag_id) else { return; };
    let speed mut= 50;
    if train.tags.contains(tag) {
        speed = params.speed;
    }
    ...
}

The potential UI for this would identical to the first proposed design of function parameters. Indeed there’s little advantage with this design. It’s tidier since you can keep adding more parameters to the struct and it does not explode the function signature, but that’s pretty much it. Implementing this was quite a bit work and it has no clear benefit, and it actually has a huge (technical) downside. An advantage of the classic scripting design of “objects are hashtables” is making it much easier to support data migration, when the programmer decides to add fields to the object. But NimbyScript structs are C-like pure data in memory, not YOLO hash tables.

This means that any edit to the struct definition changes the in-memory offsets of not just the affected fields, but every other following field. It should be possible to delete and reorder fields, not just add them to the end. This what I call the “migration problem”. It is very familiar to me since the game itself is at this point a huge migration problem itself, with every new design, feature, change and bugfix completely and utterly determined by remaining compatible to existing saves. And dear future script authors, this is you too. You will need to consider and measure every change you make to a data structure. Really think about field naming and what goes into them. Realizing that deleting something means you are automatically deleting data players manually input in thousands of places in thousands of saves.

To make this easier I ended up making NimbyScript struct a bit of an hybrid. It keeps the relatively compact memory layout of a C struct and the O(1) access, but it also adds metadata to every instance of the struct to describe which names and types were used in each field. This data is never used for script execution. But when a script changes the definition of a struct, the game will automatically perform a migration of every instance of that struct in the game data. It can preserve data when fields are reordered, deleting fields only loses the data of the deleted fields, and new fields can added at any point in the struct. It cannot preserve data on field renaming, so really think about field naming before you release your creation to the world, because it’s then set in stone.

Binding script data to game objects

At this point the game was able to call create instances of script-defined data structs and pass them to script calls as a parameter, and keep those instances preserved as game state. But this capability was just a placeholder for the big thing: being able to extend C++ game objects with script-defined data structs. Jumping straight into it:

struct SpeedChange extend Signal {
    tag_id: ID<Tag>,
    speed: f64,
}

SpeedChange is similar to the previous Params struct. But it now includes a new syntax: extend Signal. This means 0 or 1 instances of this struct can be associated to an instance of the C++ type Signal. It also automatically adds a new script-level method to Signal, called view_SpeedChange(), which will return *SpeedChange if it exists. For example:

fn some_signal_script_fn(db: &DB, train: &Train, signal: &Signal) {
    let sc &= signal.view_SpeedChange() else { return; };
    let tag &= db.tags.view(sc.tag_id) else { return; };
    let speed mut= 50;
    if train.tags.contains(tag) {
        speed = sc.speed;
    }
    ...
}

This moves the parametrization of scripts from function calls to the game objects themselves. Although parametrization by function parameter is obvious and logical, in the context of the game it is much more often that those parameters are going to be associated with game objects. With this syntax it is now possible to associate arbitrary script data structures to (some) C++ game objects, and retrieve them for reading in the script. For reading. Who writes to them? The player does, from the UI:

This means that, when script structs declare themselves to be extensions of some built-in game object, like Signal, the relevant UI editor panel will now also display extra controls to enable these extensions into the object, and allow the player to fill in these data fields (this will be better organized than in the test UI of the screenshot).

Since struct fields are going to be shown on the UI, some extra data is needed to make it more usable. The SpeedChange example from the UI screenshot actually looks like this:

struct SpeedChange extend Signal {
    tag_id: ID<Tag> meta {
        label: "Tag",
    },
    speed: f64 meta {
        label: "Speed",
        min: 0,
        default: 50,
        max: 100,
    },
}

meta is optional, but it will improve the UI of your scripts if you put some effort into it.

Scripts can also extend some objects which are not going to be “conceptually” scriptable, like Line or Train (Motion is the simulation state of trains). The data binding UI will then show up in the line editor and allow the player to input data, and the script will then be able to read it back if it also gets a reference to the line from somewhere (like simulation state). This is where the real power of the feature shows up.

Before you were forced to hardcode data in the script, or pass it over every call into the script (from UI controls). Indirect and conditional data is hard can cumbersome to express, like for instance “some signals should change some trains speed” being mediated by the fact of a train having a tag, and then having the speed value also as a parameter. Wouldn’t it be also good to be able to directly attach the concept of this specific script speed change to the Train object, and directly read it from the signal script? With this new design it is possible:

struct SpeedChange extend Train {
    speed: f64,
}
fn some_signal_script_fn(db: &DB, train: &Train, signal: &Signal) {
    let sc &= train.view_SpeedChange() else { return; };
    let speed mut= sc.speed;
    ...
}

The script succinctly declares what data it is adding to Train, then the code is now super clear: some optional data might exist in the Train. If it exists, read and act on it (not shown). If it does not, do nothing. The UI would show the controls to attach this data to trains as it did to the previous signal example (no screenshot, all of this is still too green for most of the UI to be done).

Binding arbitrary instances of script data structs to (some) game objects, and automatically having the game editor UI allow the editing of that data is very powerful and it neatly solves many problems at once, like script parameters, object extensions, excessive copy-pasting, etc.

Script execution and signal reconceptualization

I’m being evasive around the fact of how and when scripts are going to be executed, and it is on purpose, because I’ve changed my mind. In the 1.17 signal prototype the answer is “always, and the script can modify everything in the simulation”. I now believe this is an error. I now think scripts should be more declarative and more data driven, and I finally have the tools to make this possible, unlike in 1.17. For this reason I am not going to explain the prototyped new signal systems of 1.18 in this post, since I conceptualized them before this realization, and both their code and ideas assume this “script owns the entire simulation world” concept.

This change of mind goes back to making sure scripts do not nuke the game, both in the crashing sense and in the design sense. The crashing sense is obvious and that’s why I am making a babby’s first Rust clone as the game scripting language. The design sense is not so obvious but very clearly emerges as soon as you try to change anything in the signal system beyond basic path checking: every script gets to define what is a path check, what is an occupied path, what are trains supposed to do. Different scripts have zero mutual coordination (they coordinate with the C++ core because they have to, but since they have full write access, they will make logical state errors which nuke the game logic and which the author cannot anticipate). I now think this is a bad idea, and I want to avoid it as much as possible. Therefore the 1.17 idea is wrong and a new concept for signal script execution is required, centered on coordination.