PTF: family descriptors, provenance-based eligibility, deterministic order -- and level data

[denwere]

PTF: family descriptors, provenance-based eligibility, deterministic order — and level data

Two commits, and the first stands on its own. If you want the generic engine work and not the
level hook, take commit 1 and stop; nothing in it changes behaviour for the files PTFextender
merges today.

The three config files currently merged are untouched by all of this. The shipped
seven-attribute identity is still the fallback for any file no descriptor describes, so
ActionMaps.xml and friends resolve exactly as they do now.

---

Commit 1 — PTF: family descriptors, provenance-based eligibility, deterministic order

Four changes to ApplyPtfPatches, each fixing something that only shows up once a mod patches
something bigger than a keybind file.

1. Extension-less base paths

wildcard.rfind('.') returning npos meant "give up", so a file without a dot could never be
patched. levels/<lvl>/whdata_1 is one — the registry that says which profile streams a layer.
A file should not be unpatchable because of how it is spelled.

2. Eligibility is PROVENANCE, not just the name

Every <stem>__*<ext> the pak filesystem returns was merged, and that filesystem is the base
game plus every mod, flattened. The base game ships a great many files that already carry a
__<suffix> — counted on a retail 1.5.6 install, restricted to the directories a content mod
patches:

5,902 files across 3,245 distinct suffixes
muz 411   zena 371   kuman 200   bandita 96   straz 68   autotests 20

Warhorse publish these as reserved modids for exactly this reason. A player who installs a mod
called muz should not thereby merge 411 of Warhorse's own build fixtures into their tables.

ModIndex (new, mod_index.h / .cpp) answers it exactly: read mod_order.txt, walk each
enabled mod's paks, record which names each ships. A part is eligible when a MOD ships it AND
its suffix names an enabled mod. No reserved-name list to maintain, nothing to go stale at a
patch.

The index reads only zip central directories and needs no external library. It cannot be taken
from the enumeration instead — measured in game: ICryPak::FindFirst fills
attrib / times / size / name with no pak field, and attrib is 0x80000001 for all 184 hits
tested across the base game and three separate mods. It separates pak from loose and nothing
finer.

3. Order is mod_order.txt, not enumeration order

Patches were applied in whatever order FindFirst returned them, so two mods touching one row
resolved differently on different machines, and differently from what the player's load order
says. Candidates are now collected, filtered, stable-sorted by the modid's position in
mod_order.txt, and the position is logged.

4. Identity comes from a family descriptor

The shipped rule compares seven hardcoded attribute names, which spans the three keybind files
it was written for and cannot express anything else: whdata_1 keys on a CHILD ELEMENT,
waitinglinks on a pair of attributes plus a child, and a Libs/Tables row on that table's
primary key. A fixed list is silently wrong rather than absent — skill_check_difficulty.xml
carries none of the seven, so every row keys identically and rows pair by position.

ptf_families.h is generated from a table checked offline against shipped artifacts:
merge(vanilla, diff(vanilla, override)) must reproduce that override byte for byte, over 12
files and 11 negative controls. The rules arrive here already falsified against something real
rather than as a design.

Also

Glob matching where * never crosses a separator. With fnmatch semantics a rule written for a
graph NODE also catches every EDGE inside it, the edges key on an attribute they do not carry,
and a contested rewiring comes out silently applied.

IXmlNode.h: getContent declared at slot [49] and [47] annotated as getParent, both
measured in game. [49] is the trivial getter of the content member at +0x48, confirmed by
READING real element text rather than by elimination — the <Name> leaf under
/Root/GameProfileManager/GameProfiles/GameProfile returned apolena_abandonedCampsEnviro. The
node header is 0x58, not the 0x48 the clone implementation's allocation size suggests.

---

Commit 2 — Apply PTF parts to level data, not only the config files

ApplyPtfPatches runs from the table loader, so the only files that could ever be patched were
the ones that loader reads. Level data comes in through IXmlUtils::LoadXmlFromFile and never
passed through it — which is why a content mod has no choice but to ship a whole-file override
of leveldata.xml, whdata_1, waitinglinks.xml and objects_mission0.xml, and why two such
mods cannot coexist: the last one in mod_order wins the file outright and the other mod's rows
simply cease to exist.

One hook reaches all of them. Measured: 37,768 calls over 28,019 distinct paths in a single
launch-load-quit, and every family a level-content mod claims comes through this one function.
So this is a hook plus a path lookup, and every future family is a descriptor rather than a
reverse-engineering job.

Slot [1] of IXmlUtils, not ISystem [131] — the latter is a thin forwarder into the former,
so hooking IXmlUtils catches both routes and no direct caller slips past. The target is read out
of the live vtable via gEnv->pSystem->GetXmlUtils(), which is why it arms on a KCSE
PreDataLoaded message rather than at plugin load: the pointer does not exist yet when the DLL
loads.

bEnablePatching is 1 on all 37,768 calls — the stock XML patcher already runs inside this
function, on every file. A merge placed here sits exactly where the engine already expects one.

A thread_local depth counter guards re-entrancy, because the merge loads each part through the
very function being hooked. Thread-local rather than a plain flag: two loader threads were
observed interleaving in the same session.

The engine does not already do this — tested by effect rather than assumed. The same build,
shipping the parts and no override, plays with the plugin installed and does not play with the
DLL renamed away and KCSE otherwise untouched (Total plugins found: 0, and no plugin lines in
kcd.log). So C_ObjectTreeDatabase::LoadFromXML's LoadPartFromXML sibling is not reached for
whdata_1, and neither is anything for leveldata.xml.

Path spellings vary by caller, so matching is done on a normalised form — lowercased, separators
folded, a leading data/ stripped:

levels/kutnohorsko/LevelData.xml            mixed case, no prefix
levels/kutnohorsko/Objects_Mission0.xml
levels/kutnohorsko/whdata_1                 no extension at all
data/levels/kutnohorsko/WaitingLinks.xml    ...and this one carries data/

A glob written one way would silently miss three of the four. A load whose path matches no
family descriptor returns the engine's own pointer untouched, before any work is done.

---

How it was tested

A nine-cutscene mod spanning two levels (kutnohorsko, trosecko), 105 new layers, shipped
twice — once with the parts BESIDE the whole-file overrides, once with the overrides removed and
the parts as the only source. The plugin's own counters come out inverted, which is what makes
this a measurement rather than a look:

                            overrides present   overrides removed
kutnohorsko  leveldata        added 0  mod 87     added  87  mod 0
             objects_mission0 added 0  mod 437    added 175  mod 0
             whdata_1         added 0  mod 609    added  87  mod 0
             waitinglinks     added 0  mod 350    added 175  mod 0
trosecko     leveldata                            added  18  mod 0
             objects_mission0                     added  37  mod 0
             whdata_1                             added  18  mod 0
             waitinglinks                         added  37  mod 0

With the overrides present every part row keys onto the row already there and nothing is
added — the check that the identity rule recognises the mod's own rows rather than duplicating
them. With the overrides gone every row is added, and each count is exactly the part's row
count. In game: Kuttenberg plays with the overrides present, and both regions play with them
removed — the configuration where the rows can only have come from the merge.

Level pak sizes for that mod: 19.4 MB and 7.1 MB as overrides, 7,428 and 2,928 bytes as
parts. Whole download 36.9 MB to 10.2 MB.

Cost at level load, bounded from log timestamps: <= 6 s for a 63 MB objects_mission0.xml
and a 25 MB waitinglinks.xml, <= 1 s for the other six parts. Those windows include the
engine's own read and parse, so they are an upper bound on the merge rather than a measurement
of it.

The descriptor table is also cross-checked offline against a Python implementation of the same
merge, run over every enabled mod's claimed vanilla paths on this machine rather than only the
author's own mod — because a table tuned to one author's habits would describe their rows and
nothing else.

What this does NOT do

That survey is also where the limits show up, so they are worth stating rather than leaving to
be discovered:

• A part can only replace or append, never remove. Seven files across three installed mods
  delete rows from the vanilla file, and those mods must keep their whole-file override. This is
  inherent to the mechanism, not a gap in the table.
• Six files the descriptor table does not yet describe correctly — four Skald graphs and
  skill_check_difficulty.xml, which carries none of the seven fallback attributes, so every row
  keys identically and rows pair by position. They are foreign content, and they are the reason
  the descriptor table is data rather than code: each is a table entry, not a patch.
• The level hook is not free on very large files. A 63 MB objects_mission0.xml merge sits
  inside a <= 6 s window at level load. If that is unacceptable in a keybind merger, commit 2 is
  the one to leave out — commit 1 stands alone and changes nothing for the files PTFextender
  merges today.
• Nothing warns a player whose plugin failed to load. A mod that ships parts INSTEAD of
  overrides goes silent rather than degrading. The mod this was built for ships parts BESIDE its
  overrides for that reason, and only drops them once the log has confirmed the merge.