Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/adr/0012-model-call-journaling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# 0012. Model-call journaling through an application-layer decorator

- Status: Accepted
- Date: 2026-11-08

## Context

`docs/spec/domain.md` section 8.10 rule 2 lists "model call, tool call, channel send" as effects that must be written to the journal before they run, and invariant 5 restates the same rule as "the journal is written before any side effect". The agent loop in section 9 calls `d.llm.complete(...)` on every step, and neither the call site nor the snippet writes a journal record. The gap was tracked as Q13 in `docs/spec/open-questions.md`.

The loop cannot journal the call itself in the obvious way, and neither can the provider. `LlmPort.complete` as sketched in section 8.5 takes `messages`, `tools`, and `budget` only: it carries no `RunId`, so a record cannot be addressed to a run from below the loop. The provider adapters live in `providers/` packages, which have no access to the journal and must stay unaware of it. Something between the loop and the port has to hold the `RunId` and the `JournalPort`.

Transcript persistence belongs to the same effort, but it needs a run-lifecycle owner that does not exist yet in Step 7.

## Decision

Model-call journaling lives in a per-run decorator, `JournalingLlm`, that wraps `LlmPort` and lives in `runtime/core/lib/src/application/agent/`. The decorator holds the `RunId`. Before it delegates to the provider it writes a `modelStarted` record and flushes it to disk; when the provider returns it records `succeeded` through the existing `seq`-addressed `finished(...)`; when the call throws it records `failed` there with a bounded error description and rethrows the original error unchanged.

The section 9 loop and the provider adapter stay journal-free. The loop keeps calling `d.llm` as shown, and the adapter knows nothing about the journal or about runs. Model-call records carry no prompt and no reply *fields*: only the run, the sequence number, the status, the time, and the optional error. The failure `error` string holds the runtime type plus the message, truncated to 200 characters with a `…` mark; it is not encrypted, so a provider that echoes the prompt can leak prompt text into that one field.

A thrown call is recorded `failed`, not `unknownOutcome`. A provider that throws has returned control to the caller, so the outcome is known to be a failure; `unknownOutcome` is reserved for a call that may still be running (Q14).

`AgentDepsFactory.forRun(RunId)` is the intended production path that hands out agent dependencies, and it is the only path that decorates the provider, so every `AgentDeps` built through it carries the journaled port. The rule is not structurally enforced: `AgentDeps` has a public primary constructor, and resolving `LlmPort` from the container (`getIt<LlmPort>()`) still returns the raw provider, so code that chooses either route bypasses the decorator. Making that impossible (private constructor, `@visibleForTesting`, or a non-default Hive `instanceName`) is deferred. Tests that need a bare or fake port construct the dependencies directly.

Transcript persistence will also live in this decorator, but that half is deferred to plan Step 8.

Q13 is resolved by this ADR.

## Alternatives

- **Journal inside the section 9 loop.** Rejected: it couples the loop to `JournalPort`, adds pressure on invariant 1 by putting storage concerns in the loop's step, and every future caller of the loop, or of `LlmPort` directly, would have to remember the record itself. A decorator puts the rule in one place that cannot be forgotten.
- **Journal inside the provider adapter.** Rejected: the adapters are `providers/` packages with no access to the journal or to `RunId`, and giving them one would invert the dependency direction and leak run bookkeeping into a public package.
- **Reuse the existing tool `Invocation` record with a `tool: 'model'` pseudo-tool.** Rejected: it conflates tool semantics with model calls. `Invocation` carries `tool`, `Risk`, and feeds the approval and policy paths, so a model call would look like a policy-checked tool, and the risk and denial fields would be meaningless.
- **A symmetric `modelStarted`/`modelFinished` pair.** Rejected: it writes two records per call, while the `seq`-addressed `finished(...)` already covers the outcome of a started record. One start record plus one outcome update keeps the journal smaller and the replay logic the same as for tools.
- **Transcript persistence in Step 7.** Rejected: there is no run-lifecycle owner for a transcript stream yet, and guessing one here would pre-empt plan Step 8.

## Consequences

- `JournalPort` grows two methods: `modelStarted`, which writes and flushes the start record and returns its `seq`, and `modelCallsForRun`, which reads the model-call records of a run. The section 8.5 snippet is a sketch and is not exhaustive; it shows the shape, not the full port.
- `HiveJournal` needs a kind discriminator so that model-call records and tool `Invocation` records can share a store and still be read back separately.
- No model-call timeout exists yet. The decorator therefore cannot distinguish a timeout from any other failure, and a hung provider blocks the step; timeouts are Q11/Q14 territory and are not solved here.
- Transcript persistence and multi-turn history are still pending in plan Step 8; until then a run's messages live only in the loop's local list.
- Known gap: journaling is bypassable. `AgentDeps` keeps its public primary constructor and `LlmPort` stays resolvable from the container, so only code that goes through `AgentDepsFactory.forRun` is guaranteed a decorated port. Enforcing this is deferred (see Decision).
- A failing success write after the model call leaves the record `started`, which recovery would read as an unknown outcome. This is by design: rewriting it `failed` would claim a failure the provider never reported.
- `AgentDepsFactory` captures a single `LlmPort`, so it cannot serve the per-profile model selection of section 14 (plan Step 9); that step has to revisit the factory.
- `docs/spec/open-questions.md` Q13 is marked resolved; no other question changes.
31 changes: 20 additions & 11 deletions docs/plan/M2.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,18 @@
# Milestone M2: domain and loop

- Status: In progress — `chat` mode, the loop-to-`RunBloc` application service, and the section 2.8 metrics remain.
- Status: In progress — the loop-to-`RunBloc` application service and the section 2.8 metrics remain.
- Specification: `docs/spec/domain.md` (sections 8–9), `docs/spec/storage.md` (section 15), `docs/spec/roadmap.md` (M2), `docs/spec/dart-language.md` (section 7)
- ADR: — (no decision changed; the discrepancies found during implementation are recorded as Q7–Q14 in `docs/spec/open-questions.md`)
- ADR: [0012](../adr/0012-model-call-journaling.md) (model-call journaling through an application-layer decorator); the other discrepancies found during implementation are recorded as Q7–Q14 in `docs/spec/open-questions.md`

## Acceptance criteria

- [x] `Run.addTaint` adds a label idempotently; the `taints` getter is a read-only view, the constructor copies its input, and the run never removes a label (`runtime/core/test/domain/run_test.dart`).
- [x] `HiveJournal` implements `JournalPort`: the `seq` grows across calls and reopen, and `forRun` filters by run (`runtime/core/test/infrastructure/hive_journal_test.dart`).
- [x] `runAgent` follows the fixed section 9 order: malformed and unavailable checks, policy, approval, journal `started` before the tool call, timeout with a per-invocation child token, taint on untrusted output, non-throwing truncation, fresh tool snapshot per step, cancellation; 24 tests (`runtime/core/test/application/agent_loop_test.dart`).
- [x] `truncateToolOutput` measures UTF-8 bytes, cuts on a rune boundary, and JSON-encodes non-strings (`runtime/core/test/application/tool_output_test.dart`); `ToolSet` matches on the LLM name (`runtime/core/test/application/tool_set_test.dart`).
- [x] Fail-closed agent defaults are registered in `CoreModule`; `registerJournal({required String directory})` registers the journal after `configureCore` and rejects a second registration; `AgentDeps` resolves from the registered collaborators (`runtime/core/test/di/injection_test.dart`).
- [x] Fail-closed agent defaults are registered in `CoreModule`; `registerJournal({required String directory})` registers the journal after `configureCore` and rejects a second registration; `AgentDepsFactory` resolves and `forRun` returns decorated `AgentDeps` (`runtime/core/test/di/injection_test.dart`).
- [x] `dart run melos run ci` is green (format, analyze, arch check, tests) and `dart run tools/docs_check/bin/docs_check.dart` exits 0.
- [ ] `chat` mode runs end-to-end against a fake `LlmPort` (roadmap M2).
- [x] `chat` mode runs end-to-end against a fake `LlmPort` (roadmap M2); 4 tests in `runtime/core/test/application/chat_end_to_end_test.dart`.
- [ ] The application service binds the loop to `RunBloc`, projects journal state, and publishes `AgentEvent`s through `StatePublisher` (sections 8.10 and 8.11).
- [ ] The section 2.8 target metrics are measured (not measurable before the daemon exists; postponed to M4).

Expand All @@ -24,8 +24,8 @@
- [x] Step 4 (added): section 9 loop — `runAgent` in `runtime/core/lib/src/application/agent/agent_loop.dart`; 24 tests in `runtime/core/test/application/agent_loop_test.dart` with the fakes in `runtime/core/test/application/agent_fakes.dart`.
- [x] Step 5 (added): fail-closed defaults in `runtime/core/lib/src/application/agent/agent_defaults.dart` (`DefaultAgentConfig`, `DenyAllApprovals`, `PromptContextBuilder`, `EmptyToolRegistry`), `AgentDeps` binding in `runtime/core/lib/src/di/core_module.dart`, and `registerJournal({required String directory})` in `runtime/core/lib/src/di/injection.dart`; tests in `runtime/core/test/di/injection_test.dart`.
- [x] Step 6 (added): rewrite this plan in the `_template.md` form and record the M2 open questions (Q7–Q14) in `docs/spec/open-questions.md`.
- [ ] Step 7 (added): `chat` mode end-to-end with a fake `LlmPort`; a test drives a conversation through the loop and checks the final reply and the journal. Gated by Q13: the model-call journaling decision must land with this step, not be postponed again.
- [ ] Step 8 (added): application service that binds `runAgent` to `RunBloc`; it projects run state from the journal and publishes `AgentEvent`s through `StatePublisher` (sections 8.10, 8.11). Gated by Q13: the service journals model calls and transcript persistence.
- [x] Step 7 (added): `chat` mode end-to-end with a fake `LlmPort`; a test drives a conversation through the loop and checks the final reply and the journal. Gated by Q13: the model-call journaling decision must land with this step, not be postponed again — it did, as [ADR-0012](../adr/0012-model-call-journaling.md).
- [ ] Step 8 (added): application service that binds `runAgent` to `RunBloc`; it projects run state from the journal and publishes `AgentEvent`s through `StatePublisher` (sections 8.10, 8.11). Q13 now gates only transcript persistence there; model-call journaling is already handled by `JournalingLlm` ([ADR-0012](../adr/0012-model-call-journaling.md)).
- [ ] Step 9 (added): section 14 profile and risk configuration plus an interactive approval channel; replaces `DefaultAgentConfig` and `DenyAllApprovals`. Profile rules must never lower a declared risk.
- [ ] Step 10 (added): section 13 untrusted wrapping — 128-bit random boundary per run, `untrusted_` variables, the injection test with a `write` tool, and the tool-output boundary wrapping deferred from the loop (Q8). Until then `PromptContextBuilder` passes the prompt through unwrapped and tool output reaches the model as the truncated plain value.
- [ ] Step 11 (added): section 2.8 target metrics (cold start, idle memory, event latency p99, tool-call overhead). This needs the daemon and is therefore done in M4.
Expand Down Expand Up @@ -53,20 +53,22 @@
- [ ] Q10: journal directory resolution versus a synchronous `configureCore` — see `../spec/open-questions.md`.
- [ ] Q11: enforcement of the budget beyond `maxSteps` — see `../spec/open-questions.md`.
- [ ] Q12: journaling an approval refusal — see `../spec/open-questions.md`.
- [ ] Q13 (gates Steps 7–8): model-call journaling and transcript persistence (`JournalPort` or application-service decorator) — see `../spec/open-questions.md`.
- [x] Q13 (gates Steps 7–8): model-call journaling and transcript persistence (`JournalPort` or application-service decorator) — resolved by [ADR-0012](../adr/0012-model-call-journaling.md); see `../spec/open-questions.md`.
- [ ] Q14: `unknownOutcome` versus `failed` for a timed-out, uncancellable tool — see `../spec/open-questions.md`.

## Outcome

The domain and loop groundwork landed on `feat/m2-run-aggregate` (draft PR #41): the `Run` aggregate with a read-only `taints` view and `addTaint`, the Hive journal and its registration hook, the `application/agent/` seam types, the section 9 `runAgent` loop with per-invocation child tokens and non-throwing truncation, fail-closed defaults and their DI bindings.

Postponed within M2: `chat` mode end-to-end, the application service that binds the loop to `RunBloc` and `StatePublisher`, section 14 configuration and the interactive approval channel, and section 13 untrusted prompt and tool-output wrapping. The section 2.8 metrics are postponed to M4 because there is no daemon to measure yet.
Step 7 then landed on top of it: `chat` mode end-to-end with model-call journaling. `JournalingLlm` writes a `modelStarted` record and flushes it before every provider call ([ADR-0012](../adr/0012-model-call-journaling.md)); `AgentDepsFactory.forRun` hands out the decorated dependencies. Covered by `runtime/core/test/application/chat_end_to_end_test.dart`, `runtime/core/test/application/journaling_llm_test.dart`, and the extended `runtime/core/test/infrastructure/hive_journal_test.dart` and `runtime/core/test/di/injection_test.dart`.

Decisions the specification does not settle are recorded as Q7–Q14 in [open-questions.md](../spec/open-questions.md); `docs/spec/domain.md` gained only a pointer to Q7, Q8, and Q14, and no decision was changed. Q12 records the observed gap that an approval refusal is not journalled; Q13 records that the loop does not journal model calls, which section 8.10 rule 2 demands (gates plan Steps 7–8); Q14 records that a timed-out call is kept as `failed`.
Postponed within M2: the application service that binds the loop to `RunBloc` and `StatePublisher` (including transcript persistence), section 14 configuration and the interactive approval channel, and section 13 untrusted prompt and tool-output wrapping. The section 2.8 metrics are postponed to M4 because there is no daemon to measure yet.

### Milestone history (M0, M1, M2 step 1)
Decisions the specification does not settle are recorded as Q7–Q14 in [open-questions.md](../spec/open-questions.md); `docs/spec/domain.md` gained pointers to Q7, Q8, and Q14, and ADR-0012 implementation notes in sections 8.5 and 9. Q12 records the observed gap that an approval refusal is not journalled; Q13 records that the loop does not journal model calls, which section 8.10 rule 2 demands, and is resolved in Step 7 by [ADR-0012](../adr/0012-model-call-journaling.md); Q14 records that a timed-out call is kept as `failed`.

Recorded before the step 6 rewrite; kept as written.
### Milestone history (M0, M1, M2 steps 1 and 7)

The entries up to M2 step 1 were recorded before the step 6 rewrite; kept as written.

#### M0 — Done

Expand All @@ -84,3 +86,10 @@ Recorded before the step 6 rewrite; kept as written.
- Run aggregate added: `enum RunMode { chat, task, resident }` and `final class Run` in `runtime/core/lib/src/domain/run.dart`, exported from `package:alteri_one_core/alteri_one_core.dart`.
- Hive-backed `JournalPort` implementation added at `runtime/core/lib/src/infrastructure/journal/hive_journal.dart`.
- Unit tests added: `runtime/core/test/domain/run_test.dart` and `runtime/core/test/infrastructure/hive_journal_test.dart`.

#### M2 step 7 — Done

- `chat` mode runs end to end through `runAgent` against a fake `LlmPort`: 4 tests in `runtime/core/test/application/chat_end_to_end_test.dart` cover a reply-only run, a tool round-trip with the journal order asserted, `maxSteps` exhaustion, and a failing model-call journal that stops the run before the provider.
- Model-call journaling (ADR-0012): `ModelCall` in `runtime/core/lib/src/domain/model_call.dart`; `JournalPort.modelStarted` and `JournalPort.modelCallsForRun` in `runtime/core/lib/src/domain/ports/journal_port.dart`; a `kind` discriminator (`tool` or `model`) over the shared `steps` box and `seq` counter in `runtime/core/lib/src/infrastructure/journal/hive_journal.dart`; `JournalingLlm` in `runtime/core/lib/src/application/agent/journaling_llm.dart`; `AgentDepsFactory.forRun(RunId)` in `runtime/core/lib/src/application/agent/agent_deps.dart`, which replaces the `AgentDeps` DI binding in `runtime/core/lib/src/di/core_module.dart` (`injection.config.dart` regenerated). Both new types are exported from `package:alteri_one_core`.
- Tests added or extended: `runtime/core/test/application/chat_end_to_end_test.dart`, `runtime/core/test/application/journaling_llm_test.dart`, `runtime/core/test/infrastructure/hive_journal_test.dart`, `runtime/core/test/di/injection_test.dart`.
- ADR-0012 accepted, which resolves Q13; `docs/spec/domain.md` (sections 8.5 and 9), `docs/spec/storage.md` (section 15) and `docs/spec/open-questions.md` (Q13) updated.
4 changes: 4 additions & 0 deletions docs/spec/domain.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,8 @@ abstract interface class ToolPort {

`RunId`, `MemoryId`, `Message`, `Turn`, `ToolCall`, and `CancelToken` are domain types. `ToolSpec` and `Risk` are shared with the contracts.

Implementation note: this sketch lists the ports and their central methods, not every method of each port. In particular, model calls use a start method plus the existing `seq`-addressed `finished(...)`: the application-layer `JournalingLlm` decorator wraps `LlmPort`, holds the `RunId`, calls `JournalPort.modelStarted` before the call, closes that record through `finished`, and reads the records back with `JournalPort.modelCallsForRun`. The section 9 loop does not journal model calls itself (ADR-0012, Q13).

### 8.6 Memory

- `MemoryEntry` is a short fact, not a transcript. Fields: `id`, `text`, `source` (`user` or `agent`), `createdAt`, `runId`, `tags`, `untrusted`.
Expand Down Expand Up @@ -263,6 +265,8 @@ Mechanics:

The loop lives in the application layer. It works only through `LlmPort`, `ToolPort`, and `PolicyGate`. Tool calls are intercepted before execution: genkit does not run tools by itself.

Implementation note: the loop does not journal the model call itself. `d.llm` arrives already wrapped in the application-layer `JournalingLlm` decorator, which writes the `modelStarted` record and flushes it before delegating to the provider, then records `succeeded` or `failed` after the call returns or throws, and rethrows the original error (ADR-0012, Q13).

```dart
Future<RunResult> run(Run run, AgentDeps d) async {
final messages = await d.context.build(run);
Expand Down
1 change: 1 addition & 0 deletions docs/spec/forbidden-glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ Do not use anything listed here, even if you find it in old examples, articles,
| Profile | a named agent configuration: model, prompt, tools, budgets, rules |
| Tool | an action the model can call |
| Invocation | one journal record of a tool call, with its status |
| Model call | one journal record of a call to the model provider, with its status; it carries no prompt and no reply text |
| Risk | the effect level of a tool: `read`, `write`, `external`, `destructive` |
| PolicyGate | the single decision point for whether a tool runs |
| Taint label | a marker that the run read an untrusted source |
Expand Down
Loading
Loading