perf(kite3d): serve the manifest from memory and drop .kite3d from it

[pythonlearner1025]

Stacked on #20. Base branch is perf/dev-server-manifest-hash-cache, not main. Review #20 first, then this diff.

What part this touches

kite3d dev starts a local HTTP server. The editor is its only client.

GET /api/files returns the project manifest. The editor calls it on project load, on every scene save, and on every Play.

The server already runs a chokidar watcher over the project. On add, change and unlink it hashes the changed file, updates knownHashes, and broadcasts an SSE event.

One predicate, isIncludedPath, did two unrelated jobs. It chose what the manifest lists and what the watcher reports. It was also the authorization gate for every /files route, through safeProjectPath:

if (!relativePath || !isIncludedPath(relativePath)) throw new Error(`Invalid project path: ${relativePath}`)

That thrown message becomes HTTP 403. So the manifest filter and the file permission were the same line.

The problem

#20 stopped the byte reads. The walk survived.

buildManifest still ran one readdir per directory and one stat per file, all serial, on every request. About 12,700 entries at roughly 30 microseconds each is the whole cost:

{ "label": "step-0 (PR 20)", "filesSamplesMs": [ 374.2, 368.1, 355.0 ], "entryCount": 12677 }

Startup still hashed the whole tree, 8,985 ms, because the boot pass seeds knownHashes.

More than half of that tree is not the project. .kite3d is the server's own scratch area, and isIncludedPath let it in on purpose:

return !parts.some((part) => part.startsWith('.') && part !== '.kite3d')

In the owner's project that directory held 9,181 files and 1,870 MB, 54.6 percent of everything scanned: agent logs, video references, Blender rounds, capture rounds.

The fix

The server holds the manifest in memory, keyed by path.

The watcher maintains it. The same debounced timer that broadcasts an event now reads the path once and writes the result into the manifest, instead of hashing only to broadcast.

scheduleEvent marks the path stale when it fires. Every route that changes a file reaches that function, so a read landing inside the 150 ms debounce still sees the truth.

GET /api/files resolves the marked paths, then returns the map. Cost follows what changed, not project size.

Without a watcher no event ever arrives, so a request walks the tree and rebuilds the map from it. That is the same code path as #20, and the map doubles as its hash cache.

isIncludedPath splits into two honest names. isServableProjectPath keeps the old rule and still gates the file routes and the watcher. isManifestPath adds one condition: not .kite3d.

The remaining walk reads eight files at a time.

Answering the startup question from the report. An in-memory manifest does answer presence, so knownHashes.has(path) is no longer the only way to tell a real unlink from a removed directory. The startup hash still has to stay, for a different reason:

if (await hashFile(target) === knownHashes.get(path)) return

That line drops a watcher event when the bytes did not change. With no seeded hash the comparison never matches, so every first touch of a file would reload it in the editor even when nothing changed. A build step that rewrites 500 unchanged assets would fire 500 reloads. A hashless boot index would also just move the full hash to the first request, since /api/files needs a sha256 on every row.

So startup still hashes. It now hashes 1,558 MB instead of 3,428 MB, with eight readers instead of one, which is the 4.3x below.

The risk trade

The manifest now trusts the watcher. A missed event persists until the next event on that path, where before a page reload re-walked and repaired it.

That trust already existed. Live reload has always depended on the watcher alone, so a missed event already meant a stale editor. This widens the blast radius from one file's content to one file's manifest row.

The watcher failing to start is the one case that is fully covered. createDevServer already catches it and warns, and currentManifest walks whenever watcher is undefined.

Dropping .kite3d from the manifest is the riskier half. What I checked before taking it:

The file routes are unaffected. They gate on isServableProjectPath, which is unchanged, so GET, PUT and DELETE of .kite3d/state.json, .kite3d/console.log and .kite3d/check.json all still work. This was the trap: deleting the .kite3d exception outright would have 403'd the editor's own three writers.

The watcher still reports .kite3d. So the editor still receives change events for those paths, and ViewerInstanceManager re-seeds its hash from the event payload. Two editor tabs still agree.

console.log and check.json writes re-read the file for their If-Match before writing, so they never depended on the manifest. state.json falls back to If-Match: * on its first write and uses the write response after that.

Play builds fileRevisions from the manifest. No runtime asset lives in .kite3d, so nothing loses a ?v=.

The Files panel filters dot paths out of its visible grid already, but its second pass did not. .kite3d/journal.jsonl and friends were rendered as stray aria-label buttons. They are gone now.

I rejected the wider version of this change: making the watcher skip .kite3d too. It would save watcher work, but it would cut the SSE events that keep a second editor tab's hash for state.json in sync, and repairing that needs an editor-side change to adopt the conflict hash. Not worth it for a directory that fsevents watches for free.

Tests

Suites run, mirroring .github/workflows/ci.yml:

npm run build       OK
npm run typecheck   OK
npm run lint        OK
npm run test:kite3d   Test Files  6 passed (6)    Tests  11 passed (11)
npm run test:runtime  1 passed (1.8s)
npm run test:editor   8 passed (22.1s)

The 8 editor tests are the proof for the .kite3d half. They drive real dev servers and poll .kite3d/state.json, .kite3d/console.log and .kite3d/check.json over HTTP throughout, and one of them fails the run on any response of 400 or more.

One test is new, in packages/kite3d/test/server.test.ts:

lists an externally added file and forgets an externally deleted one

It writes a file outside the editor, waits for the add event, asserts the manifest lists it with the right hash, deletes it, waits for the unlink event, and asserts the manifest has dropped it.

Removing the two lines in refreshManifestPath that write the manifest fails it, and fails the test from #20 as well:

⎯⎯⎯⎯⎯⎯⎯ Failed Tests 2 ⎯⎯⎯⎯⎯⎯⎯
   × reports a new manifest hash after a same size rewrite that keeps the modification time 271ms
   × lists an externally added file and forgets an externally deleted one 223ms
      Tests  2 failed | 9 passed (11)

Said plainly, because the report asked: this test failed correctly on its first version. The test in #20 did not. Its first version restored the modification time with a Date, which drops the sub-millisecond part and moved the time by itself, so it passed without the fix. That one was rewritten to pin a whole second.

Correctness on the real project. The whole /api/files response was captured from #20 and from this branch against the same 12,701 file clone. The manifest deliberately changed, so the digests cannot match outright. Removing the .kite3d rows from #20's response makes them match exactly:

entries: pr20=12677  pr20-minus-kite3d=3496  new=3496
3c5a2bd843fab3197a032656ad8ba8229462bbc3c367151165c72f10ebf52f8c  m-pr20-nokite.json
3c5a2bd843fab3197a032656ad8ba8229462bbc3c367151165c72f10ebf52f8c  m-step23-c.json
IDENTICAL: every surviving entry has the same path, size, sha256 and mtime
--- dropped paths are all .kite3d ---
9181

Speed on the same clone:

{
  "label": "all-three",
  "bootMs": 2069.5,
  "filesSamplesMs": [ 15.4, 4.2, 2.4 ],
  "entryCount": 3496,
  "hashedMegabytes": 1557.7
}

GET /api/files fell from 374 ms to 2.4 ms. Startup fell from 8,985 ms to 2,069 ms.

The concurrency limit was chosen by measurement, not taste. Same tree, same hashing work, readers varied:

limit 1   5885 ms
limit 8   2188 ms
limit 32  2101 ms
limit 64  2121 ms

Manual walkthrough, headless Chromium, real editor. A fresh kite3d init carrying tools/, docs/, and three capture directories under .kite3d/, the same shapes as the owner's project. Same fixture for both runs:

[stacked-before] dev server boot: 3642 ms
[stacked-before] editor open to Project loaded: 5858 ms
[stacked-before] /api/files on load: [985] ms
[stacked-before] Play click to running game: 660 ms
[stacked-before] /api/files during Play: [170] ms

[stacked-after]  dev server boot: 1282 ms
[stacked-after]  editor open to Project loaded: 3453 ms
[stacked-after]  /api/files on load: [979] ms
[stacked-after]  Play click to running game: 165 ms
[stacked-after]  /api/files during Play: [4] ms

Play fell from 660 ms to 165 ms. I opened the editor, viewed the Files panel, pressed Play, and compared the screenshots at each step. The scene, the Objects list, the Inspector and the Files tree render the same.

One number did not move, and it is not the manifest. /api/files on load stayed at about 980 ms in both runs. That request queues behind the 2.2 MB editor bundle on the same single-threaded server. It is worth a separate look.

Edges checked:

A file added, changed or deleted outside the editor still reaches the manifest. That is the new test.

A checkpoint restore still works. It suppresses watcher events and rewrites paths the server never wrote, so it keeps a real walk on the after side, and the map is rebuilt from that walk.

.kite3d files are still readable and writable over HTTP. The 8 editor tests exercise all three of them.

.kite3d change events still broadcast. refreshManifestPath hashes any servable path and only declines to store the ones the manifest does not list.

Publish is untouched. It uses a different buildManifest, in packages/kite3d/src/manifest.ts.

Not covered: a watcher that silently drops an event. The manifest row for that path stays stale until the next event on it. Restarting kite3d dev repairs it.

Deploy

The kite3d npm package ships it, together with #20. The change is one file in packages/kite3d/src, compiled by npm run build:kite3d into packages/kite3d/dist.

No editor bundle change. No engine change. No backend change. No migration.

Merge #20 first, then this branch. Release with the repository script:

npm run release:patch

Rollback: revert this commit alone to keep #20's hash cache, then release again.

git revert f797611
npm run release:patch

Users on an older kite3d are unaffected. The dev server is local, so nothing is deployed to a host.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T623SnndzSVwCSmuQrVRj2