Skip to content

Examples, quickstart, the live measurement — and a parallel branch that lost its assets - #8

Merged
inhuman merged 8 commits into
mainfrom
examples-docs-parallel
Aug 4, 2026
Merged

Examples, quickstart, the live measurement — and a parallel branch that lost its assets#8
inhuman merged 8 commits into
mainfrom
examples-docs-parallel

Conversation

@inhuman

@inhuman inhuman commented Aug 4, 2026

Copy link
Copy Markdown
Owner

What

Two example applications, a quickstart, a README written for a reader who
arrives cold, the live measurement that replaces three unbacked numbers — and
one real bug found along the way.

Format 2.2.2 → 2.2.3 (an engine fix; no field added, changed or removed).

The bug: a parallel branch lost six of the flow's fields

A branch's state was assembled by naming fields, and six were not named: the
assets, their resolver, cache and context, working memory, and the application's
vocabulary.

Nothing failed loudly, which is why it survived. An unknown asset expands to an
empty string by contract — that contract exists so a marker never reaches a
model — so {{asset:x}} inside a branch became "" and the tool call that
needed it lost a required argument. The error named the argument, not the
substitution.

Reproduced all three failure modes separately before fixing; they break
differently and all silently:

field what it looks like
assets a call loses a required argument
memory a step reads a preview instead of the whole value
vocabulary one_of loses its tie-breaker and stores nothing

It stayed hidden because in a live catalogue of 29 skills, not one call step
with an asset had ever sat inside a parallel branch.

Fixed by forking the parent state rather than rebuilding it, with the few
branch-local fields reset right after. A list has to be extended by whoever adds
a field to the engine, and that person is not thinking about parallel; forking
inverts the default, and what does NOT reach a branch is visible in one place.

The asset cache is shared with branches rather than cloned, so an asset three
branches need is fetched once — which makes it concurrent state, so it has a
lock, held across the resolve rather than around the map alone, and the mutex is
a pointer because a forked state copies the struct.

CI now runs -race. That was the widest hole here: parallel is goroutines
over shared state, and such bugs give a wrong answer under load rather than a
crash in the quiet. The first thing the detector caught was a test of my own —
see below.

Two properties documented, not changed

The step callbacks are concurrent. OnStep and OnStepStart fire from the
goroutine that ran the step, so inside a parallel they fire from several at
once. An embedder appending to a slice without a lock has a data race in
production. Not serialised inside the engine on purpose: a lock there would hold
up a branch for as long as somebody else's telemetry write takes.

Outcome.Steps stops at a parallel, Outcome.Skipped does not. Branch
steps reach OnStep as they happen, so nothing is lost — but the two fields
disagree, and that is invisible until somebody checks one and assumes the other.
Left as it is rather than "fixed": merging branch traces in would add entries for
every embedder reading that field, which is a behaviour change, not a bug fix. A
test pins the property and says what to do if it ever starts failing.

Examples

examples/ now holds the format AND how to embed it:

examples/skills/          the skill files, where they were
examples/simple-llm-app/  the engine embedded in ~200 lines, net/http only
examples/eino-llm-app/    the same, with the model reached through a framework

Both are separate modules, which is what keeps eino out of the engine's
go.modgo list -deps ./... confirms it. Two guards hold that: the
self-containment walk skips nested modules, and a new test fails if a directory
with Go code loses its go.mod. Proven by deleting one — eino is reported as a
dependency of the engine on the next run.

CI vets and tests each example module separately, because go test ./... at the
top never descends into them.

Docs

QUICKSTART.md — fifteen minutes ending in an A/B the reader
runs themselves: one skill carrying both descriptions, same request, one flag
changed. 227 tokens as prose against 102 as steps, and on a request naming
nothing, prose answers while steps exit honestly. Steps 1–5 need no model:
the test stub charges by prompt length, and a test asserts the direction.

README gained what a cold reader looks for and never found: go get,
badges, a Status section (library v0.5.x, format 2.2.3, where it runs, and
the caveat that 1.x skills do not load until Migrate runs), and a "What this is
not" section — not a process orchestrator, not an agent framework, zero
dependencies as a consequence of being embedded, and when NOT to use it.

The measurement is real now. The three unbacked pairs of numbers are gone,
replaced by the event log of a working installation recomputed from the raw
rows: 23 skills, 5 280 turns, 20 significantly cheaper, 1 more expensive, 2
unchanged; with Holm–Bonferroni 16 of 23 survive. Stated as observational,
not an experiment — the periods are split by a date, not randomised, and other
things changed in those days. The skill that got worse is named plainly, followed
by the known ways a rewrite costs more.

A guard test keeps the two languages one document: same sections, same code
samples, every local link resolving.

Testing

  • go vet ./... clean
  • go test ./... -count=1 all green
  • go test ./... -race all green — and the mutex proven load-bearing by
    removing it and watching the detector fire
  • Both example modules vet and test green
  • The bug was reproduced first, in three separate failure modes

No dependencies

  • Production code still imports stdlib only (sync added, stdlib)
  • eino lives in an example module and nowhere else

Format & API compatibility

  • No field added, changed or removed; EngineVersion 2.2.2 → 2.2.3
  • CHANGELOG.md updated
  • Go API unchanged

Checklist

  • Godoc added/updated for the documented properties
  • No secrets / tokens in the diff

🤖 Generated with Claude Code

inhuman added 8 commits August 4, 2026 19:05
… own folder

The examples showed the FORMAT and never the seam. What an embedder actually
needs to see is the other half: what you hand the engine, what it hands back,
and which of the two descriptions you are supposed to run.

  examples/skills/          the skill files, where they were
  examples/simple-llm-app/  the engine embedded in ~200 lines, net/http only
  examples/eino-llm-app/    the same, with the model reached through eino

Both are SEPARATE modules, and that is the point rather than tidiness. The
engine's promise is that embedding it adds no dependencies; an example pulling
in a framework would break exactly that promise unless it is its own module.
The engine's go.mod has never heard of eino, and `go list -deps ./...` says so.

Two guards keep it that way, both in imports_test.go: the self-containment walk
now skips nested modules, and a new test fails if a directory with Go code in it
loses its go.mod. Proven by deleting one — eino is reported as a dependency of
the engine on the next run, which at that moment is what it is.

CI gained a step that vets and tests each example module, because `go test ./...`
at the top never descends into them. An example that does not build teaches the
format wrong, and it would rot silently.

Each application is verified end to end against a stub — an httptest server for
the simple one, eino's own interface for the other — so "it works" is something
a reader can check rather than something a README claims. The eino test also
checks that a step's `model:` and `sampling:` arrive as options: a field the
skill sets and the executor drops is decoration, and nothing tells the author.

Both go.mod files carry `replace => ../..` so CI checks the examples against the
engine as it is now rather than the last release. The READMEs say to delete that
line when copying the example out.
The README explained HOW the format works to somebody already inside. A reader
coming from an article decides in a minute whether to run `go get`, and asks
three things the file never answered.

**What is this instead of.** A section naming the neighbours, because the
differences are architectural and checkable in the code rather than a matter of
taste: not a process orchestrator — the engine owns no state between turns, has
no storage and does not survive a restart; not an agent framework — there the
graph is the application developer's code, here the steps are the skill author's
data, portable between hosts, which is why the format is versioned and why
Migrate exists; zero dependencies as a consequence of being embedded, not a
pose; and when NOT to use it, said again next to the comparison.

**What can be relied on.** A Status section: the library is v0.5.x and the Go
API may still move, while the FORMAT is what is stable at 2.2.2 — a number that
lived in version.go and nowhere a reader would look. Where it runs: the engine
executes the whole skill catalogue of a working assistant, which is the
strongest argument for it and was missing entirely. And the caveat nobody wants
to discover on their own: skills written for format 1.x do not load until
Migrate is run, schedules included.

**How to start.** go get, badges, and a link to examples/ on the first screen —
two applications that embed the engine are the best answer to "how do I use
this" and were invisible from the top. Coverage is stated because a reader
choosing a dependency looks: 89.3% engine, 95.1% linter, both measured, not
quoted.

**The measurements are qualified.** The three before/after pairs now say what is
NOT established about them — runs per scenario, spread, whether anything besides
the form of the description changed — and the counterexample moved up beside
them: a skill that cost 43k tokens as steps against 36k as prose, because
knowledge is expensive in a step WITH tools. Three wins and no losses read as
advertising; one named loss is what makes the other three credible.

Re-running those measurements properly is still open — it needs a model and a
catalogue, not an edit.

A guard test keeps the two READMEs one document in two languages: same number of
sections, same number of code samples, and every local link resolving. Proven by
deleting a section from one of them.
A reader arriving from an article wants to see the claim, not read it. So the
quickstart is built around an A/B they run themselves: one skill carrying BOTH
descriptions, the same request, one word changed on the command line.

  go run . -skill ../skills/menu.yaml -mode playbook -input "…"   227 tokens
  go run . -skill ../skills/menu.yaml -mode workflow -input "…"   102 tokens

Three things had to exist for that to be real rather than a diagram.

`examples/skills/menu.yaml` gained a `playbook` half and `mode: workflow`. No
example showed `mode` before, which is odd for a field whose whole purpose is
keeping both descriptions while a prompt is being moved into steps — the
comparison this quickstart walks through is exactly what it is for.

`examples/simple-llm-app` gained a `-mode` flag that overrides the skill's
declared mode, and counts what a turn cost: generations, prompt and completion
tokens, printed after the trace. "Steps are cheaper" is a claim until something
puts a number next to it.

The test stub now charges by prompt length the way a real endpoint does, so the
A/B is checkable offline — and there is a test asserting the direction: the
steps half must reach the model with fewer prompt tokens, because the dictionary
of synonyms never gets there at all. The minimal skill the quickstart tells a
reader to write has its own test too: it is the first thing they will run.

The quickstart is honest about what its numbers are. They come from a stub, so
they show the shape rather than a bill; with a real endpoint the prose half also
spends extra generations deciding to call a tool and reading the result back, so
the gap shown is understated; the accuracy figure is somebody else's measurement
and the README says what is not established about it; and the counterexample is
there — a skill that cost 43k tokens as steps against 36k as prose.

Both READMEs now point at the quickstart first and `examples/` second, and the
translation guard covers the new pair as well: same sections, same code samples,
every local link resolving.
The README carried three pairs of numbers with no methodology behind them and a
paragraph admitting as much. There is now a real measurement, and it is both
stronger and more honest than what it replaces.

The event log of a working installation, five weeks, every turn where a skill
matched, questions asked by people rather than by the author of the skill. The
metric is LLM generations per turn, orchestrator and subagents together.

  23 skills with at least 5 turns on each side, 5 280 turns
  20 significantly cheaper, 1 significantly more expensive, 2 unchanged
  effect from -18 to -0.5 generations per turn; largest 38 -> 20, typical 7 -> 3

Recomputed from the raw rows rather than quoted: Mann-Whitney with a tie
correction, a stated inclusion threshold, and the aggregate checked against the
summary that came with the data. The threshold is why the counts here differ
slightly from that summary — it is written down.

Two things are stated because leaving them out would be the expensive kind of
silence.

The comparison is OBSERVATIONAL. The periods are split by a date rather than
randomised, and other things changed in those same days: the engine was being
edited alongside the skills. So it shows the catalogue got cheaper across that
boundary, not that nothing else contributed. Named in the README in those words.

And the loss. One skill went the other way — median 6 -> 10 generations,
p<0.001. Twenty wins and no losses read as advertising; one measured loss is
what makes the other twenty worth reading. What caused it the measurement does
not say, and the text says that too: it counts generations, not reasons.

The old counterexample (43k tokens against 36k) is gone with the rest of the
unbacked numbers, replaced by this measured one.

No skill names travelled with the data: the log belongs to a private
installation, so what is published is the aggregate. Both quickstarts point at
the measurement instead of restating figures nobody could check.
The paragraph about the skill that got more expensive was written as a
credibility move — "one measured loss is what makes the other twenty worth
reading". That is advertising with extra steps, and it buries the useful part.

It now says what happened and what follows from it: one skill went from a median
of 6 generations per turn to 10, the measurement does not say why, steps are not
automatically cheaper, and the format does not replace checking.

Then the part that was missing — the known ways a rewrite costs MORE:

- an asset inside a step that has tools, riding into every generation of the
  loop rather than just the first;
- splitting a turn that had nothing to split, so the second prompt adds a
  generation without removing work from the first;
- a decision that is genuinely open — wording, judging, reading intent — which a
  condition cannot replace, and pretending otherwise only moves the model call
  somewhere less visible;
- one or two steps and no branching, where prose was already cheaper.

Ending on the thing a reader actually needs: measure each skill on its own after
rewriting it. The engine gives the trace to measure with and a linter for the
quiet defects; it does not promise a rewrite pays.

The same tone crept into both quickstarts ("one counterexample, on purpose") and
is gone from there too.
23 Mann-Whitney tests at p<0.05 is 23 chances for a fluke, and a reader who
knows that will say so. Computed from the same raw rows: with Holm-Bonferroni
16 of the 23 stay significant — 15 cheaper and the one that got more expensive.

The skills that drop out are mostly those with the fewest turns, and the
conclusion does not move: 20 cheaper without correction, 15 with it, against 1
more expensive either way.

Worth its own sentence: the loss survives the correction too, so it is not an
artifact of testing many skills at once. A counterexample that vanished under
correction would have been worth less than no counterexample at all.
A branch of a `parallel` step did not inherit six of the flow's fields: the
assets, their resolver, their cache and context, working memory, and the
application's vocabulary. The sub-state was assembled by naming fields, and
those six were not named.

Nothing failed loudly, which is why it survived. An unknown asset expands to an
empty string by contract — that contract exists so a marker never reaches a
model — so `{{asset:x}}` inside a branch became "" and the tool call that needed
it lost a required argument. The error named the argument, not the substitution.
Working memory and the vocabulary went the same way: a step in a branch read a
preview instead of the whole value, and `one_of` lost its tie-breaker.

It stayed hidden because in a live catalogue of 29 skills not one `call` step
with an asset had ever sat inside a `parallel` branch. The path never ran.

The branch state is now FORKED from the flow's, with the few branch-local fields
reset right after: its own copy of the variables (branches must not see each
other's work, or the result would depend on who finished first), and an empty
trace, skip list and answer until the join. A list of fields has to be extended
by whoever adds a field to the engine, and that person is not thinking about
`parallel`; forking inverts the default, and what does NOT reach a branch is
visible in one place.

The asset cache is shared with the branches rather than cloned into them, so an
asset three branches need is fetched once — which makes it concurrent state, so
it has a lock, held across the resolve rather than around the map alone. The
lock is a pointer: a forked state copies the struct, and a mutex copied by value
guards nothing. Removing the lock and running the tests under -race shows the
race it prevents, and CI now runs the race detector.

Engine fix, not a format change: EngineVersion 2.2.2 -> 2.2.3.
…guess

Both came out of the parallel fix — one from the race detector it added, one
from reading the merge at the join. Neither changes behaviour.

**The step callbacks are concurrent.** OnStep and OnStepStart fire from the
goroutine that ran the step, so inside a `parallel` they fire from several at
once, and an embedder appending to a slice without a lock has a data race in
production. The engine does not serialise them on purpose: a lock there would
hold up a branch for as long as somebody else's telemetry write takes, and that
is not the engine's call to make. Now said on Deps and in both READMEs, and the
test that found it is written the way an embedder has to write it — with the
lock in plain sight.

Worth noting how it surfaced: the race detector went into CI with the previous
commit, and the first thing it caught was my own test. Without it this would
have reached embedders as an occasional wrong answer under load.

**Outcome.Steps stops at a parallel, Outcome.Skipped does not.** A branch runs
in a forked state, and only its variables and its skips are merged back. Nothing
is lost — branch steps reach OnStep as they happen, which is where per-step
telemetry comes from — but the two fields disagree, and the asymmetry is
invisible until somebody checks one and assumes the other.

Left as it is rather than "fixed": merging branch traces into Outcome.Steps
would add entries for every embedder reading that field, which is a behaviour
change and not a bug fix. A test now pins the property and says in its comment
what to do if it ever starts failing.
@inhuman
inhuman merged commit 2e49a26 into main Aug 4, 2026
1 check passed
@inhuman
inhuman deleted the examples-docs-parallel branch August 4, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant