Skip to content

Drop the output re-parse and the full pre-run snapshot - #11

Open
IPerception wants to merge 4 commits into
mainfrom
perf/drop-reparse-and-full-snapshot
Open

Drop the output re-parse and the full pre-run snapshot#11
IPerception wants to merge 4 commits into
mainfrom
perf/drop-reparse-and-full-snapshot

Conversation

@IPerception

Copy link
Copy Markdown
Owner

web/PERFORMANCE.md backlog items 2 and 3. Four commits, each reviewable on its own.

What changed

Item 2 — stop re-parsing the output. render() rebuilt the browsable document
with parse(result.output), a second full parse producing a second full set of
segment objects for a document processText() already held. It now returns that
document and render() takes it.

Item 3 — snapshot only what changes. processText() cloned every segment before
the rules ran so it could diff afterwards. Each rule now copies a segment on its
first write to it and carries the copy in the mark it already reported, so only
changed segments are ever copied. All three rules had a single mutation site to hook.

processText keeps the first copy per segment: two rules can touch one segment,
and the second rule's copy already contains the first rule's edit, so last-writer-wins
would show a before that never existed in the input file.

sameSegment was kept, as a filter over marked segments rather than a sweep over
all of them. StringReplaceRule counts a match even when the replacement equals it,
so ANYTOWNANYTOWN marks every hit and changes nothing; without the filter each
would surface in the Changes tab as a change with identical before and after.

Results

Measured against the same tree with only the change absent, not against the numbers
in the doc — those were taken on another machine.

before after
item 2, 50 MB run 2565 ms 1848 ms −28%
item 3, 100 MB run 5502 ms 4057 ms −26%
item 3, 125 MB run 10812 ms 6968 ms −36%

Both were predicted as memory savings and delivered time savings instead. Item 2
was expected to remove "roughly a third of peak heap" and moved the heap column ~2%.
The reasoning error was treating the doc's five copies as five things held at once,
when the two removed were each built and discarded inside a single run. The
out-of-memory ceiling has not moved: 150 MB failed before and still does.

Two things in PERFORMANCE.md were wrong rather than stale, and are corrected with
the measurements: the heap column understated the app (the benchmark discarded the
browsable document the app keeps in state.doc), and it is heap after a forced GC
rather than peak, which is precisely why it could not see either copy.

Correctness

Item 3 moves the correctness of the change list — and of the Limited Data Set's only
record of what it rewrote — onto the accuracy of each rule's marks. Nothing checked
that before, so web/tests/_marks.mjs was added and committed first, passing
against the old behaviour, before anything was removed. It audits both directions for
all three rules, from the suites that already have each rule wired up.

Two tests were verified load-bearing by breaking the code and watching them fail:

  • restoring the re-parse fails render.mjs [6]
  • flipping to last-writer-wins fails parity.mjs [14] with a fabricated before
node web/tests/all.mjs     # 9 suites, 91 + 132 + 31 checks in the three touched
python -m unittest         # 11 tests

Parity with the Python engine is byte-for-byte unchanged throughout, which is the
property that matters most here: none of this may alter a single byte of output.

🤖 Generated with Claude Code

render() rebuilt the browsable document with parse(result.output) -- a second
full parse, producing a second full set of segment objects, for a document
processText already held. It now returns that document and render() takes it.

Measured on 10/25/50 MB synthetic 837P, like for like: run time falls 26-37%
(50 MB: 2565 ms -> 1848 ms). Retained heap is essentially flat, about 2%.

PERFORMANCE.md predicted "roughly a third of peak heap" and that is wrong.
The prediction treated the re-parse as a copy held alongside the mutated
document, but the mutated document became unreachable the moment processText
returned, so only one of the two was ever retained. What the re-parse actually
cost was the time to do it. The doc is corrected in a later commit, with the
measurements.

The benchmark stopped counting a re-parse that no longer happens. Note it had
been discarding its re-parse immediately while the app keeps it in state.doc
forever, so its heap column understated the app; the baseline above was
re-measured with that corrected.

render.mjs section [6] covers what changed -- the document you browse, which
parity.mjs does not see because the output string was never at risk. Verified
load-bearing by restoring the re-parse, which fails it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01AfabdBfKu77QsNYVZkc3nn
Preparation for dropping the full pre-run snapshot from processText. Once the
change list is built from marks instead of from a clone-and-compare over every
segment, a change a rule makes without marking it is unrecoverable -- the
before-value is gone the moment the rule overwrites it. Nothing checked that.

_marks.mjs audits both directions and is called from the suites that already
have each rule wired up, rather than becoming a tenth suite that would have to
re-lift the engine to reach them: parity.mjs [13] for the two engine rules,
deid.mjs [18] for DeidentifyRule.

The two directions are not equally serious. "unmarked" must always be empty.
"overclaimed" is expected and harmless -- StringReplaceRule counts a match even
when the replacement equals it, so ANYTOWN -> ANYTOWN marks every hit and
changes nothing. That case is now pinned explicitly, because it is the reason
processText keeps sameSegment as a filter over marked segments instead of
trusting the marks outright.

All three rules pass as they stand; this commit changes no behaviour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01AfabdBfKu77QsNYVZkc3nn
processText cloned every segment in the document before the rules ran, so it
could diff against that copy afterwards and work out what moved. On a 5M-segment
file that is a second full working set built to find a few thousand changes.

Each rule now copies a segment on its first write to it and carries the copy in
the mark it already reported, so only changed segments are ever copied. All
three rules had a single mutation site to hook -- put() in DeidentifyRule, and
one assignment each in the other two -- so the capture is a few lines apiece.

processText keeps the FIRST copy it sees per segment. Two rules can touch one
segment, and the second rule's copy already contains the first rule's edit; the
change list has to show what arrived in the file, not what the second rule
happened to find. parity.mjs [14] pins this, and fails with a fabricated
before value if the rule is flipped to last-writer-wins.

sameSegment stays, as a filter over marked segments rather than a sweep over
all of them. The marks say what a rule touched, not what it managed to change,
and StringReplaceRule counts a match even when the replacement equals it.
Dropping the filter would surface those as changes with identical before and
after. What it no longer catches -- a rule editing without marking -- is now
covered directly by the mark audits added in the previous commit.

Measured on synthetic 837P, against the same tree with only this change absent:

  100 MB   5502 ms -> 4057 ms   (-26%)
  125 MB  10812 ms -> 6968 ms   (-36%)
   50 MB   1848 ms -> 1770 ms    (-4%)

Retained heap is unchanged and the out-of-memory ceiling has not moved: the
clone was transient, so it never showed up in a post-GC measurement. What it
cost was the work of building it and the GC pressure of carrying it, which is
why the gain grows with the file -- at 125 MB the extra copy was crowding the
heap limit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01AfabdBfKu77QsNYVZkc3nn
…bers

Two things in this file were wrong rather than merely stale.

The heap column understated the app. The benchmark discarded its copy of the
browsable document immediately; the app keeps it in state.doc for as long as the
file is open. The 125 MB row moving from 1.99 GB to 2.90 GB is that correction,
not a regression.

And the column is heap after a forced GC, not peak heap, though it was labelled
peak. That is what made the predictions for both items wrong: they were sold as
memory savings on the theory that five copies were held at once, when the two
being removed each lived and died inside a single run. Both delivered time
instead -- together roughly halving run time at 100-125 MB -- and moved the heap
column by about 2%. The reasoning error is written down next to the result,
since the same mistake is available to anyone reading the copy table.

Also records that the ceiling did not move: 150 MB was out of memory before and
still is. Only the streaming parse moves that.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01AfabdBfKu77QsNYVZkc3nn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant