Examples, quickstart, the live measurement — and a parallel branch that lost its assets - #8
Merged
Merged
Conversation
… own folder The examples showed the FORMAT and never the seam. What an embedder actually needs to see is the other half: what you hand the engine, what it hands back, and which of the two descriptions you are supposed to run. examples/skills/ the skill files, where they were examples/simple-llm-app/ the engine embedded in ~200 lines, net/http only examples/eino-llm-app/ the same, with the model reached through eino Both are SEPARATE modules, and that is the point rather than tidiness. The engine's promise is that embedding it adds no dependencies; an example pulling in a framework would break exactly that promise unless it is its own module. The engine's go.mod has never heard of eino, and `go list -deps ./...` says so. Two guards keep it that way, both in imports_test.go: the self-containment walk now skips nested modules, and a new test fails if a directory with Go code in it loses its go.mod. Proven by deleting one — eino is reported as a dependency of the engine on the next run, which at that moment is what it is. CI gained a step that vets and tests each example module, because `go test ./...` at the top never descends into them. An example that does not build teaches the format wrong, and it would rot silently. Each application is verified end to end against a stub — an httptest server for the simple one, eino's own interface for the other — so "it works" is something a reader can check rather than something a README claims. The eino test also checks that a step's `model:` and `sampling:` arrive as options: a field the skill sets and the executor drops is decoration, and nothing tells the author. Both go.mod files carry `replace => ../..` so CI checks the examples against the engine as it is now rather than the last release. The READMEs say to delete that line when copying the example out.
The README explained HOW the format works to somebody already inside. A reader coming from an article decides in a minute whether to run `go get`, and asks three things the file never answered. **What is this instead of.** A section naming the neighbours, because the differences are architectural and checkable in the code rather than a matter of taste: not a process orchestrator — the engine owns no state between turns, has no storage and does not survive a restart; not an agent framework — there the graph is the application developer's code, here the steps are the skill author's data, portable between hosts, which is why the format is versioned and why Migrate exists; zero dependencies as a consequence of being embedded, not a pose; and when NOT to use it, said again next to the comparison. **What can be relied on.** A Status section: the library is v0.5.x and the Go API may still move, while the FORMAT is what is stable at 2.2.2 — a number that lived in version.go and nowhere a reader would look. Where it runs: the engine executes the whole skill catalogue of a working assistant, which is the strongest argument for it and was missing entirely. And the caveat nobody wants to discover on their own: skills written for format 1.x do not load until Migrate is run, schedules included. **How to start.** go get, badges, and a link to examples/ on the first screen — two applications that embed the engine are the best answer to "how do I use this" and were invisible from the top. Coverage is stated because a reader choosing a dependency looks: 89.3% engine, 95.1% linter, both measured, not quoted. **The measurements are qualified.** The three before/after pairs now say what is NOT established about them — runs per scenario, spread, whether anything besides the form of the description changed — and the counterexample moved up beside them: a skill that cost 43k tokens as steps against 36k as prose, because knowledge is expensive in a step WITH tools. Three wins and no losses read as advertising; one named loss is what makes the other three credible. Re-running those measurements properly is still open — it needs a model and a catalogue, not an edit. A guard test keeps the two READMEs one document in two languages: same number of sections, same number of code samples, and every local link resolving. Proven by deleting a section from one of them.
A reader arriving from an article wants to see the claim, not read it. So the quickstart is built around an A/B they run themselves: one skill carrying BOTH descriptions, the same request, one word changed on the command line. go run . -skill ../skills/menu.yaml -mode playbook -input "…" 227 tokens go run . -skill ../skills/menu.yaml -mode workflow -input "…" 102 tokens Three things had to exist for that to be real rather than a diagram. `examples/skills/menu.yaml` gained a `playbook` half and `mode: workflow`. No example showed `mode` before, which is odd for a field whose whole purpose is keeping both descriptions while a prompt is being moved into steps — the comparison this quickstart walks through is exactly what it is for. `examples/simple-llm-app` gained a `-mode` flag that overrides the skill's declared mode, and counts what a turn cost: generations, prompt and completion tokens, printed after the trace. "Steps are cheaper" is a claim until something puts a number next to it. The test stub now charges by prompt length the way a real endpoint does, so the A/B is checkable offline — and there is a test asserting the direction: the steps half must reach the model with fewer prompt tokens, because the dictionary of synonyms never gets there at all. The minimal skill the quickstart tells a reader to write has its own test too: it is the first thing they will run. The quickstart is honest about what its numbers are. They come from a stub, so they show the shape rather than a bill; with a real endpoint the prose half also spends extra generations deciding to call a tool and reading the result back, so the gap shown is understated; the accuracy figure is somebody else's measurement and the README says what is not established about it; and the counterexample is there — a skill that cost 43k tokens as steps against 36k as prose. Both READMEs now point at the quickstart first and `examples/` second, and the translation guard covers the new pair as well: same sections, same code samples, every local link resolving.
The README carried three pairs of numbers with no methodology behind them and a paragraph admitting as much. There is now a real measurement, and it is both stronger and more honest than what it replaces. The event log of a working installation, five weeks, every turn where a skill matched, questions asked by people rather than by the author of the skill. The metric is LLM generations per turn, orchestrator and subagents together. 23 skills with at least 5 turns on each side, 5 280 turns 20 significantly cheaper, 1 significantly more expensive, 2 unchanged effect from -18 to -0.5 generations per turn; largest 38 -> 20, typical 7 -> 3 Recomputed from the raw rows rather than quoted: Mann-Whitney with a tie correction, a stated inclusion threshold, and the aggregate checked against the summary that came with the data. The threshold is why the counts here differ slightly from that summary — it is written down. Two things are stated because leaving them out would be the expensive kind of silence. The comparison is OBSERVATIONAL. The periods are split by a date rather than randomised, and other things changed in those same days: the engine was being edited alongside the skills. So it shows the catalogue got cheaper across that boundary, not that nothing else contributed. Named in the README in those words. And the loss. One skill went the other way — median 6 -> 10 generations, p<0.001. Twenty wins and no losses read as advertising; one measured loss is what makes the other twenty worth reading. What caused it the measurement does not say, and the text says that too: it counts generations, not reasons. The old counterexample (43k tokens against 36k) is gone with the rest of the unbacked numbers, replaced by this measured one. No skill names travelled with the data: the log belongs to a private installation, so what is published is the aggregate. Both quickstarts point at the measurement instead of restating figures nobody could check.
The paragraph about the skill that got more expensive was written as a
credibility move — "one measured loss is what makes the other twenty worth
reading". That is advertising with extra steps, and it buries the useful part.
It now says what happened and what follows from it: one skill went from a median
of 6 generations per turn to 10, the measurement does not say why, steps are not
automatically cheaper, and the format does not replace checking.
Then the part that was missing — the known ways a rewrite costs MORE:
- an asset inside a step that has tools, riding into every generation of the
loop rather than just the first;
- splitting a turn that had nothing to split, so the second prompt adds a
generation without removing work from the first;
- a decision that is genuinely open — wording, judging, reading intent — which a
condition cannot replace, and pretending otherwise only moves the model call
somewhere less visible;
- one or two steps and no branching, where prose was already cheaper.
Ending on the thing a reader actually needs: measure each skill on its own after
rewriting it. The engine gives the trace to measure with and a linter for the
quiet defects; it does not promise a rewrite pays.
The same tone crept into both quickstarts ("one counterexample, on purpose") and
is gone from there too.
23 Mann-Whitney tests at p<0.05 is 23 chances for a fluke, and a reader who knows that will say so. Computed from the same raw rows: with Holm-Bonferroni 16 of the 23 stay significant — 15 cheaper and the one that got more expensive. The skills that drop out are mostly those with the fewest turns, and the conclusion does not move: 20 cheaper without correction, 15 with it, against 1 more expensive either way. Worth its own sentence: the loss survives the correction too, so it is not an artifact of testing many skills at once. A counterexample that vanished under correction would have been worth less than no counterexample at all.
A branch of a `parallel` step did not inherit six of the flow's fields: the
assets, their resolver, their cache and context, working memory, and the
application's vocabulary. The sub-state was assembled by naming fields, and
those six were not named.
Nothing failed loudly, which is why it survived. An unknown asset expands to an
empty string by contract — that contract exists so a marker never reaches a
model — so `{{asset:x}}` inside a branch became "" and the tool call that needed
it lost a required argument. The error named the argument, not the substitution.
Working memory and the vocabulary went the same way: a step in a branch read a
preview instead of the whole value, and `one_of` lost its tie-breaker.
It stayed hidden because in a live catalogue of 29 skills not one `call` step
with an asset had ever sat inside a `parallel` branch. The path never ran.
The branch state is now FORKED from the flow's, with the few branch-local fields
reset right after: its own copy of the variables (branches must not see each
other's work, or the result would depend on who finished first), and an empty
trace, skip list and answer until the join. A list of fields has to be extended
by whoever adds a field to the engine, and that person is not thinking about
`parallel`; forking inverts the default, and what does NOT reach a branch is
visible in one place.
The asset cache is shared with the branches rather than cloned into them, so an
asset three branches need is fetched once — which makes it concurrent state, so
it has a lock, held across the resolve rather than around the map alone. The
lock is a pointer: a forked state copies the struct, and a mutex copied by value
guards nothing. Removing the lock and running the tests under -race shows the
race it prevents, and CI now runs the race detector.
Engine fix, not a format change: EngineVersion 2.2.2 -> 2.2.3.
…guess Both came out of the parallel fix — one from the race detector it added, one from reading the merge at the join. Neither changes behaviour. **The step callbacks are concurrent.** OnStep and OnStepStart fire from the goroutine that ran the step, so inside a `parallel` they fire from several at once, and an embedder appending to a slice without a lock has a data race in production. The engine does not serialise them on purpose: a lock there would hold up a branch for as long as somebody else's telemetry write takes, and that is not the engine's call to make. Now said on Deps and in both READMEs, and the test that found it is written the way an embedder has to write it — with the lock in plain sight. Worth noting how it surfaced: the race detector went into CI with the previous commit, and the first thing it caught was my own test. Without it this would have reached embedders as an occasional wrong answer under load. **Outcome.Steps stops at a parallel, Outcome.Skipped does not.** A branch runs in a forked state, and only its variables and its skips are merged back. Nothing is lost — branch steps reach OnStep as they happen, which is where per-step telemetry comes from — but the two fields disagree, and the asymmetry is invisible until somebody checks one and assumes the other. Left as it is rather than "fixed": merging branch traces into Outcome.Steps would add entries for every embedder reading that field, which is a behaviour change and not a bug fix. A test now pins the property and says in its comment what to do if it ever starts failing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two example applications, a quickstart, a README written for a reader who
arrives cold, the live measurement that replaces three unbacked numbers — and
one real bug found along the way.
Format 2.2.2 → 2.2.3 (an engine fix; no field added, changed or removed).
The bug: a
parallelbranch lost six of the flow's fieldsA branch's state was assembled by naming fields, and six were not named: the
assets, their resolver, cache and context, working memory, and the application's
vocabulary.
Nothing failed loudly, which is why it survived. An unknown asset expands to an
empty string by contract — that contract exists so a marker never reaches a
model — so
{{asset:x}}inside a branch became""and the tool call thatneeded it lost a required argument. The error named the argument, not the
substitution.
Reproduced all three failure modes separately before fixing; they break
differently and all silently:
one_ofloses its tie-breaker and stores nothingIt stayed hidden because in a live catalogue of 29 skills, not one
callstepwith an asset had ever sat inside a
parallelbranch.Fixed by forking the parent state rather than rebuilding it, with the few
branch-local fields reset right after. A list has to be extended by whoever adds
a field to the engine, and that person is not thinking about
parallel; forkinginverts the default, and what does NOT reach a branch is visible in one place.
The asset cache is shared with branches rather than cloned, so an asset three
branches need is fetched once — which makes it concurrent state, so it has a
lock, held across the resolve rather than around the map alone, and the mutex is
a pointer because a forked state copies the struct.
CI now runs
-race. That was the widest hole here:parallelis goroutinesover shared state, and such bugs give a wrong answer under load rather than a
crash in the quiet. The first thing the detector caught was a test of my own —
see below.
Two properties documented, not changed
The step callbacks are concurrent.
OnStepandOnStepStartfire from thegoroutine that ran the step, so inside a
parallelthey fire from several atonce. An embedder appending to a slice without a lock has a data race in
production. Not serialised inside the engine on purpose: a lock there would hold
up a branch for as long as somebody else's telemetry write takes.
Outcome.Stepsstops at aparallel,Outcome.Skippeddoes not. Branchsteps reach
OnStepas they happen, so nothing is lost — but the two fieldsdisagree, and that is invisible until somebody checks one and assumes the other.
Left as it is rather than "fixed": merging branch traces in would add entries for
every embedder reading that field, which is a behaviour change, not a bug fix. A
test pins the property and says what to do if it ever starts failing.
Examples
examples/now holds the format AND how to embed it:Both are separate modules, which is what keeps eino out of the engine's
go.mod—go list -deps ./...confirms it. Two guards hold that: theself-containment walk skips nested modules, and a new test fails if a directory
with Go code loses its
go.mod. Proven by deleting one — eino is reported as adependency of the engine on the next run.
CI vets and tests each example module separately, because
go test ./...at thetop never descends into them.
Docs
QUICKSTART.md — fifteen minutes ending in an A/B the reader
runs themselves: one skill carrying both descriptions, same request, one flag
changed. 227 tokens as prose against 102 as steps, and on a request naming
nothing, prose answers while steps exit honestly. Steps 1–5 need no model:
the test stub charges by prompt length, and a test asserts the direction.
README gained what a cold reader looks for and never found:
go get,badges, a Status section (library
v0.5.x, format2.2.3, where it runs, andthe caveat that 1.x skills do not load until
Migrateruns), and a "What this isnot" section — not a process orchestrator, not an agent framework, zero
dependencies as a consequence of being embedded, and when NOT to use it.
The measurement is real now. The three unbacked pairs of numbers are gone,
replaced by the event log of a working installation recomputed from the raw
rows: 23 skills, 5 280 turns, 20 significantly cheaper, 1 more expensive, 2
unchanged; with Holm–Bonferroni 16 of 23 survive. Stated as observational,
not an experiment — the periods are split by a date, not randomised, and other
things changed in those days. The skill that got worse is named plainly, followed
by the known ways a rewrite costs more.
A guard test keeps the two languages one document: same sections, same code
samples, every local link resolving.
Testing
go vet ./...cleango test ./... -count=1all greengo test ./... -raceall green — and the mutex proven load-bearing byremoving it and watching the detector fire
No dependencies
syncadded, stdlib)Format & API compatibility
EngineVersion2.2.2 → 2.2.3CHANGELOG.mdupdatedChecklist
🤖 Generated with Claude Code