The Spatials V3 devblog 2015-10-05

Internal development notes, very slightly cleaned up and commented a week later.

TL;DR: the new generic job state machine has been implemented and most of the week was devoted to fixing everything that broke down. The end result is a much more robust, yet generic and extensible job system. There's nothing new to show on screen, this week was all about logic coding and bugfixes.

- litter: walking and nonwalking integrity
    use dumb transition nodes for enter/exit-like actions for now
- litter picked spc offset? how to fix on cleanup? in general case?
    new PositionModifierComponent with info on offsets for iso, screen, Z
    remember markDirty on PosC on removal
    track at MoverSystem level, so it works even for custom display systems

A few bugs from the litter system state machine were fixed, leaving it ready to serve as the base for a new general purpose, extensible state machine for general jobs.

- job-logistic-sm for replacing the current JobsSystem
- replace uses of JobsSystem with a state machine
    building objects
        remove state components after build
    dismantling objects
    building flooring
        keep C++ side tilemap scan, create work ent but without JobComponent for the work area
        check for invalid floor on work area too
        scheme side state machine looks for the work area
        the existing TileConstructionJobComponent holds the completion flag
    pallet logistics
        support -1 amount in job sm
        support MaxCarry
- job sm: pick only minimal amount
- whole state machine extension via hashtable merge
- infinite loop is happening related to state machines and find_symbol_unchecked
    (outlet (curlet)) used outside of a foreach lambda inside (init-state-machine)
        so it was modifying lexical mod scope or similar, and all sm instances used the same slot...
    nope! still happens, just build a bunch of beds at the same time
        a non-break cycle happens when no items can be found
        always (break) on cleanup
- job sm is leaving stacks busy despite not using them
    it always leaves the stack busy when not fully consumed
    fix by making sure if the stack survives picking, it is de-busied
- build sm and others are leaving workers busy
    happens when the extended sm is not cleaning up busy states on owned entities, no cleanup is called
    cleanup wont be called because the extended finalize (break)s
    cleanup cannot be called since finalize cleans too much stuff, which must remain clean (job and construction comps for example)
    just dupe the req cleaning, is just 2-3 calls, extend is not meant as a black box
- job sm endless loop when walk-to-item happens to be in the same tile as the worker is at the moment, AND the not enough not found triggers
    very core flaw in ECS complies checks
        they dont take into account same-frame component edits, so an item stack is not a child for the same frame
        fix: never use getComponents directly for complies checking
    now centralized in Entity::complies
        it can now take into account same-frame removed and added comps
        this introduces inconsistencies in query/complies vs in-frame actual accesible components
            getByKind was already patched to return in-frame added
- job sm doesnt survive a forced repathing of the worker
    there are two (to cleanup X) rules in walk-to-job-giver
    the node struct is a hashtable, so one overrides the other
    decide: use a list, or make it a rule to never have more than one to rule for a node inside a another node
        list cons: no direct access to vars/extend/etc members
        using or cons: this means different evals for the same path cannot have different actions, which is very useful
    -> use assoc list, and make sure extend and vars are always at the front when coding the sm
- job sm keeps incrementing even when worker is away from work area
    track position, make work condition stronger
- job sm error when impossible to find path
    ecs-preferred-center returns floor value of tile offset, so it needs +0.5 to avoid impassables
    also ecs-preferred-center, for the non-walkable preferred center, is non determinist, so the over-center check keeps failing
    add another version of the random neigh finder that accepts entropy as a parameter, use (e #@entropy) -> now deterministic
- ecs-query wasnt properly filtering path-from results

This was the bulk of the work week, and there's no easy way to cut it down to smaller pieces. A new general purpose job state machine mas introduced, and the old JobsSystem/WorkerSystem was removed. Along the way many, many bugs were fixed in both the state machine and its runtime library. There's not much visual result to show from this work, other than a node graph view of the state machine (open image in new tab for bigger, it's a SVG file):

Job state machine

Along the way the SM runtime gained the capability of extending state machines into a new state machine, while overriding one or more states. This work was vital in order to reach the goal of not having a monolithic base system that does everything relared to jobs. Now it's possible to fully customize the job SM for every possible job kind (actually it is mandatory, because the base, generic SM doesn't know what to do after a job is done). For jobs or actions that require it, it will be also possible to opt out from the base job SM and implement a new SM from scratch while playing nice with it since it's all still based on the BusyComponent / BusySystem duo.

- bug: same-frame expectations for spatial query in ItemHeldBy for a newly spawned entity
    change to a world query
- pallet sm: need to support toggles for pallet, floor picking, avoid pallet rect
    extend find-item state to support the custom query
- bug: changing pallet storage prefs mid-job lock-leaks the busy worker
    the current debug editor removes BusyC from the entity, so yeah
- bug: dumpings of picked items on pallets that don't go to the central, existing stack
    happens because of the child/not child ping-pong for the min-stack rule
    also creates "ninja scenario" for workers to race for an available stack in the up to 3secs that may take the pallet system to own it
    unify stacks while counting
    check for in-storage items on pallet job finalize and stack on them
    the fixes make this better but the ninja scenario is still possible when the drop is the first stack
- ugly pallet cliping on walls: fixedZ hack is not required at all

Having more complex item-related semantics, pallets and their delivery jobs were a bit more complicated to port to the new system, but it all worked out in the end.

- remove WorkerS, WorkerC, JobC, JobsS
- re-introduce Officer class, slim down from V2 AgentOfficer
    partial V2 model removal, first of many

There's a large amount of V2 C++ code awaiting deletion. I'm removing it piece by piece, as it becomes obsolete by newer V3 systems.

- actually start using scheme defined room specs
- move resource rules/model to scheme
    problem: a lot of the existing C++ rules init depends on having resources already init, and scheme rules init happens after C++
        comment out the C++ stuff, it's pending deletion anyway

Resources are the kind of items that can be seen in the V3 screenshots/videos, and they represent countable amounts of stuff that will be later required for building or manufacturing. For now they are the only kind of item supported, and prior to this week they were defined using the old V2 data rules. This has now been replaced with a resource description made from Scheme in V3, so resource (re)declaration is now available to mods.

- switch again to a worker system to enable future priority system
    worker-system (new): find jobs matching every worker-component entity ('one), pick closest path-able non-busy one
    if found, it sets itself busy with the job giver
    somehow the job giver ent sm has to notice a worker wants to work it
        worker creates+sets BusyC subject to itself on the giver, marks #@removeCompOnInvalid #t
        job sm recoginizes this and moves to next state
        fix job sm so it stops using 'worker-ref, uses #@subject instead
- stop using the "-object" job components, replace by the "-job" components
- bug: worker busy leak in delivery jobs
    make sure to clean spurious BusyC when not interested in deliveries
    always activate reciprocal check in the worker BusyC

The new job SM assumed the task of finding available workers as an intermediate porting step (previously it was handled WorkerSystem). This is very limiting since it means it's impossible to implement job priorities, for example. After the job SM was proven robust, a new worker system was re-implemented to be prepared for this functionality in the future.