diff --git a/docs/adr/0012-model-call-journaling.md b/docs/adr/0012-model-call-journaling.md new file mode 100644 index 0000000..a1c7943 --- /dev/null +++ b/docs/adr/0012-model-call-journaling.md @@ -0,0 +1,45 @@ +# 0012. Model-call journaling through an application-layer decorator + +- Status: Accepted +- Date: 2026-11-08 + +## Context + +`docs/spec/domain.md` section 8.10 rule 2 lists "model call, tool call, channel send" as effects that must be written to the journal before they run, and invariant 5 restates the same rule as "the journal is written before any side effect". The agent loop in section 9 calls `d.llm.complete(...)` on every step, and neither the call site nor the snippet writes a journal record. The gap was tracked as Q13 in `docs/spec/open-questions.md`. + +The loop cannot journal the call itself in the obvious way, and neither can the provider. `LlmPort.complete` as sketched in section 8.5 takes `messages`, `tools`, and `budget` only: it carries no `RunId`, so a record cannot be addressed to a run from below the loop. The provider adapters live in `providers/` packages, which have no access to the journal and must stay unaware of it. Something between the loop and the port has to hold the `RunId` and the `JournalPort`. + +Transcript persistence belongs to the same effort, but it needs a run-lifecycle owner that does not exist yet in Step 7. + +## Decision + +Model-call journaling lives in a per-run decorator, `JournalingLlm`, that wraps `LlmPort` and lives in `runtime/core/lib/src/application/agent/`. The decorator holds the `RunId`. Before it delegates to the provider it writes a `modelStarted` record and flushes it to disk; when the provider returns it records `succeeded` through the existing `seq`-addressed `finished(...)`; when the call throws it records `failed` there with a bounded error description and rethrows the original error unchanged. + +The section 9 loop and the provider adapter stay journal-free. The loop keeps calling `d.llm` as shown, and the adapter knows nothing about the journal or about runs. Model-call records carry no prompt and no reply *fields*: only the run, the sequence number, the status, the time, and the optional error. The failure `error` string holds the runtime type plus the message, truncated to 200 characters with a `…` mark; it is not encrypted, so a provider that echoes the prompt can leak prompt text into that one field. + +A thrown call is recorded `failed`, not `unknownOutcome`. A provider that throws has returned control to the caller, so the outcome is known to be a failure; `unknownOutcome` is reserved for a call that may still be running (Q14). + +`AgentDepsFactory.forRun(RunId)` is the intended production path that hands out agent dependencies, and it is the only path that decorates the provider, so every `AgentDeps` built through it carries the journaled port. The rule is not structurally enforced: `AgentDeps` has a public primary constructor, and resolving `LlmPort` from the container (`getIt()`) still returns the raw provider, so code that chooses either route bypasses the decorator. Making that impossible (private constructor, `@visibleForTesting`, or a non-default Hive `instanceName`) is deferred. Tests that need a bare or fake port construct the dependencies directly. + +Transcript persistence will also live in this decorator, but that half is deferred to plan Step 8. + +Q13 is resolved by this ADR. + +## Alternatives + +- **Journal inside the section 9 loop.** Rejected: it couples the loop to `JournalPort`, adds pressure on invariant 1 by putting storage concerns in the loop's step, and every future caller of the loop, or of `LlmPort` directly, would have to remember the record itself. A decorator puts the rule in one place that cannot be forgotten. +- **Journal inside the provider adapter.** Rejected: the adapters are `providers/` packages with no access to the journal or to `RunId`, and giving them one would invert the dependency direction and leak run bookkeeping into a public package. +- **Reuse the existing tool `Invocation` record with a `tool: 'model'` pseudo-tool.** Rejected: it conflates tool semantics with model calls. `Invocation` carries `tool`, `Risk`, and feeds the approval and policy paths, so a model call would look like a policy-checked tool, and the risk and denial fields would be meaningless. +- **A symmetric `modelStarted`/`modelFinished` pair.** Rejected: it writes two records per call, while the `seq`-addressed `finished(...)` already covers the outcome of a started record. One start record plus one outcome update keeps the journal smaller and the replay logic the same as for tools. +- **Transcript persistence in Step 7.** Rejected: there is no run-lifecycle owner for a transcript stream yet, and guessing one here would pre-empt plan Step 8. + +## Consequences + +- `JournalPort` grows two methods: `modelStarted`, which writes and flushes the start record and returns its `seq`, and `modelCallsForRun`, which reads the model-call records of a run. The section 8.5 snippet is a sketch and is not exhaustive; it shows the shape, not the full port. +- `HiveJournal` needs a kind discriminator so that model-call records and tool `Invocation` records can share a store and still be read back separately. +- No model-call timeout exists yet. The decorator therefore cannot distinguish a timeout from any other failure, and a hung provider blocks the step; timeouts are Q11/Q14 territory and are not solved here. +- Transcript persistence and multi-turn history are still pending in plan Step 8; until then a run's messages live only in the loop's local list. +- Known gap: journaling is bypassable. `AgentDeps` keeps its public primary constructor and `LlmPort` stays resolvable from the container, so only code that goes through `AgentDepsFactory.forRun` is guaranteed a decorated port. Enforcing this is deferred (see Decision). +- A failing success write after the model call leaves the record `started`, which recovery would read as an unknown outcome. This is by design: rewriting it `failed` would claim a failure the provider never reported. +- `AgentDepsFactory` captures a single `LlmPort`, so it cannot serve the per-profile model selection of section 14 (plan Step 9); that step has to revisit the factory. +- `docs/spec/open-questions.md` Q13 is marked resolved; no other question changes. diff --git a/docs/plan/M2.md b/docs/plan/M2.md index 43c9a04..9fb1712 100644 --- a/docs/plan/M2.md +++ b/docs/plan/M2.md @@ -1,8 +1,8 @@ # Milestone M2: domain and loop -- Status: In progress — `chat` mode, the loop-to-`RunBloc` application service, and the section 2.8 metrics remain. +- Status: In progress — the loop-to-`RunBloc` application service and the section 2.8 metrics remain. - Specification: `docs/spec/domain.md` (sections 8–9), `docs/spec/storage.md` (section 15), `docs/spec/roadmap.md` (M2), `docs/spec/dart-language.md` (section 7) -- ADR: — (no decision changed; the discrepancies found during implementation are recorded as Q7–Q14 in `docs/spec/open-questions.md`) +- ADR: [0012](../adr/0012-model-call-journaling.md) (model-call journaling through an application-layer decorator); the other discrepancies found during implementation are recorded as Q7–Q14 in `docs/spec/open-questions.md` ## Acceptance criteria @@ -10,9 +10,9 @@ - [x] `HiveJournal` implements `JournalPort`: the `seq` grows across calls and reopen, and `forRun` filters by run (`runtime/core/test/infrastructure/hive_journal_test.dart`). - [x] `runAgent` follows the fixed section 9 order: malformed and unavailable checks, policy, approval, journal `started` before the tool call, timeout with a per-invocation child token, taint on untrusted output, non-throwing truncation, fresh tool snapshot per step, cancellation; 24 tests (`runtime/core/test/application/agent_loop_test.dart`). - [x] `truncateToolOutput` measures UTF-8 bytes, cuts on a rune boundary, and JSON-encodes non-strings (`runtime/core/test/application/tool_output_test.dart`); `ToolSet` matches on the LLM name (`runtime/core/test/application/tool_set_test.dart`). -- [x] Fail-closed agent defaults are registered in `CoreModule`; `registerJournal({required String directory})` registers the journal after `configureCore` and rejects a second registration; `AgentDeps` resolves from the registered collaborators (`runtime/core/test/di/injection_test.dart`). +- [x] Fail-closed agent defaults are registered in `CoreModule`; `registerJournal({required String directory})` registers the journal after `configureCore` and rejects a second registration; `AgentDepsFactory` resolves and `forRun` returns decorated `AgentDeps` (`runtime/core/test/di/injection_test.dart`). - [x] `dart run melos run ci` is green (format, analyze, arch check, tests) and `dart run tools/docs_check/bin/docs_check.dart` exits 0. -- [ ] `chat` mode runs end-to-end against a fake `LlmPort` (roadmap M2). +- [x] `chat` mode runs end-to-end against a fake `LlmPort` (roadmap M2); 4 tests in `runtime/core/test/application/chat_end_to_end_test.dart`. - [ ] The application service binds the loop to `RunBloc`, projects journal state, and publishes `AgentEvent`s through `StatePublisher` (sections 8.10 and 8.11). - [ ] The section 2.8 target metrics are measured (not measurable before the daemon exists; postponed to M4). @@ -24,8 +24,8 @@ - [x] Step 4 (added): section 9 loop — `runAgent` in `runtime/core/lib/src/application/agent/agent_loop.dart`; 24 tests in `runtime/core/test/application/agent_loop_test.dart` with the fakes in `runtime/core/test/application/agent_fakes.dart`. - [x] Step 5 (added): fail-closed defaults in `runtime/core/lib/src/application/agent/agent_defaults.dart` (`DefaultAgentConfig`, `DenyAllApprovals`, `PromptContextBuilder`, `EmptyToolRegistry`), `AgentDeps` binding in `runtime/core/lib/src/di/core_module.dart`, and `registerJournal({required String directory})` in `runtime/core/lib/src/di/injection.dart`; tests in `runtime/core/test/di/injection_test.dart`. - [x] Step 6 (added): rewrite this plan in the `_template.md` form and record the M2 open questions (Q7–Q14) in `docs/spec/open-questions.md`. -- [ ] Step 7 (added): `chat` mode end-to-end with a fake `LlmPort`; a test drives a conversation through the loop and checks the final reply and the journal. Gated by Q13: the model-call journaling decision must land with this step, not be postponed again. -- [ ] Step 8 (added): application service that binds `runAgent` to `RunBloc`; it projects run state from the journal and publishes `AgentEvent`s through `StatePublisher` (sections 8.10, 8.11). Gated by Q13: the service journals model calls and transcript persistence. +- [x] Step 7 (added): `chat` mode end-to-end with a fake `LlmPort`; a test drives a conversation through the loop and checks the final reply and the journal. Gated by Q13: the model-call journaling decision must land with this step, not be postponed again — it did, as [ADR-0012](../adr/0012-model-call-journaling.md). +- [ ] Step 8 (added): application service that binds `runAgent` to `RunBloc`; it projects run state from the journal and publishes `AgentEvent`s through `StatePublisher` (sections 8.10, 8.11). Q13 now gates only transcript persistence there; model-call journaling is already handled by `JournalingLlm` ([ADR-0012](../adr/0012-model-call-journaling.md)). - [ ] Step 9 (added): section 14 profile and risk configuration plus an interactive approval channel; replaces `DefaultAgentConfig` and `DenyAllApprovals`. Profile rules must never lower a declared risk. - [ ] Step 10 (added): section 13 untrusted wrapping — 128-bit random boundary per run, `untrusted_` variables, the injection test with a `write` tool, and the tool-output boundary wrapping deferred from the loop (Q8). Until then `PromptContextBuilder` passes the prompt through unwrapped and tool output reaches the model as the truncated plain value. - [ ] Step 11 (added): section 2.8 target metrics (cold start, idle memory, event latency p99, tool-call overhead). This needs the daemon and is therefore done in M4. @@ -53,20 +53,22 @@ - [ ] Q10: journal directory resolution versus a synchronous `configureCore` — see `../spec/open-questions.md`. - [ ] Q11: enforcement of the budget beyond `maxSteps` — see `../spec/open-questions.md`. - [ ] Q12: journaling an approval refusal — see `../spec/open-questions.md`. -- [ ] Q13 (gates Steps 7–8): model-call journaling and transcript persistence (`JournalPort` or application-service decorator) — see `../spec/open-questions.md`. +- [x] Q13 (gates Steps 7–8): model-call journaling and transcript persistence (`JournalPort` or application-service decorator) — resolved by [ADR-0012](../adr/0012-model-call-journaling.md); see `../spec/open-questions.md`. - [ ] Q14: `unknownOutcome` versus `failed` for a timed-out, uncancellable tool — see `../spec/open-questions.md`. ## Outcome The domain and loop groundwork landed on `feat/m2-run-aggregate` (draft PR #41): the `Run` aggregate with a read-only `taints` view and `addTaint`, the Hive journal and its registration hook, the `application/agent/` seam types, the section 9 `runAgent` loop with per-invocation child tokens and non-throwing truncation, fail-closed defaults and their DI bindings. -Postponed within M2: `chat` mode end-to-end, the application service that binds the loop to `RunBloc` and `StatePublisher`, section 14 configuration and the interactive approval channel, and section 13 untrusted prompt and tool-output wrapping. The section 2.8 metrics are postponed to M4 because there is no daemon to measure yet. +Step 7 then landed on top of it: `chat` mode end-to-end with model-call journaling. `JournalingLlm` writes a `modelStarted` record and flushes it before every provider call ([ADR-0012](../adr/0012-model-call-journaling.md)); `AgentDepsFactory.forRun` hands out the decorated dependencies. Covered by `runtime/core/test/application/chat_end_to_end_test.dart`, `runtime/core/test/application/journaling_llm_test.dart`, and the extended `runtime/core/test/infrastructure/hive_journal_test.dart` and `runtime/core/test/di/injection_test.dart`. -Decisions the specification does not settle are recorded as Q7–Q14 in [open-questions.md](../spec/open-questions.md); `docs/spec/domain.md` gained only a pointer to Q7, Q8, and Q14, and no decision was changed. Q12 records the observed gap that an approval refusal is not journalled; Q13 records that the loop does not journal model calls, which section 8.10 rule 2 demands (gates plan Steps 7–8); Q14 records that a timed-out call is kept as `failed`. +Postponed within M2: the application service that binds the loop to `RunBloc` and `StatePublisher` (including transcript persistence), section 14 configuration and the interactive approval channel, and section 13 untrusted prompt and tool-output wrapping. The section 2.8 metrics are postponed to M4 because there is no daemon to measure yet. -### Milestone history (M0, M1, M2 step 1) +Decisions the specification does not settle are recorded as Q7–Q14 in [open-questions.md](../spec/open-questions.md); `docs/spec/domain.md` gained pointers to Q7, Q8, and Q14, and ADR-0012 implementation notes in sections 8.5 and 9. Q12 records the observed gap that an approval refusal is not journalled; Q13 records that the loop does not journal model calls, which section 8.10 rule 2 demands, and is resolved in Step 7 by [ADR-0012](../adr/0012-model-call-journaling.md); Q14 records that a timed-out call is kept as `failed`. -Recorded before the step 6 rewrite; kept as written. +### Milestone history (M0, M1, M2 steps 1 and 7) + +The entries up to M2 step 1 were recorded before the step 6 rewrite; kept as written. #### M0 — Done @@ -84,3 +86,10 @@ Recorded before the step 6 rewrite; kept as written. - Run aggregate added: `enum RunMode { chat, task, resident }` and `final class Run` in `runtime/core/lib/src/domain/run.dart`, exported from `package:alteri_one_core/alteri_one_core.dart`. - Hive-backed `JournalPort` implementation added at `runtime/core/lib/src/infrastructure/journal/hive_journal.dart`. - Unit tests added: `runtime/core/test/domain/run_test.dart` and `runtime/core/test/infrastructure/hive_journal_test.dart`. + +#### M2 step 7 — Done + +- `chat` mode runs end to end through `runAgent` against a fake `LlmPort`: 4 tests in `runtime/core/test/application/chat_end_to_end_test.dart` cover a reply-only run, a tool round-trip with the journal order asserted, `maxSteps` exhaustion, and a failing model-call journal that stops the run before the provider. +- Model-call journaling (ADR-0012): `ModelCall` in `runtime/core/lib/src/domain/model_call.dart`; `JournalPort.modelStarted` and `JournalPort.modelCallsForRun` in `runtime/core/lib/src/domain/ports/journal_port.dart`; a `kind` discriminator (`tool` or `model`) over the shared `steps` box and `seq` counter in `runtime/core/lib/src/infrastructure/journal/hive_journal.dart`; `JournalingLlm` in `runtime/core/lib/src/application/agent/journaling_llm.dart`; `AgentDepsFactory.forRun(RunId)` in `runtime/core/lib/src/application/agent/agent_deps.dart`, which replaces the `AgentDeps` DI binding in `runtime/core/lib/src/di/core_module.dart` (`injection.config.dart` regenerated). Both new types are exported from `package:alteri_one_core`. +- Tests added or extended: `runtime/core/test/application/chat_end_to_end_test.dart`, `runtime/core/test/application/journaling_llm_test.dart`, `runtime/core/test/infrastructure/hive_journal_test.dart`, `runtime/core/test/di/injection_test.dart`. +- ADR-0012 accepted, which resolves Q13; `docs/spec/domain.md` (sections 8.5 and 9), `docs/spec/storage.md` (section 15) and `docs/spec/open-questions.md` (Q13) updated. diff --git a/docs/spec/domain.md b/docs/spec/domain.md index 9a751e2..b83fb20 100644 --- a/docs/spec/domain.md +++ b/docs/spec/domain.md @@ -194,6 +194,8 @@ abstract interface class ToolPort { `RunId`, `MemoryId`, `Message`, `Turn`, `ToolCall`, and `CancelToken` are domain types. `ToolSpec` and `Risk` are shared with the contracts. +Implementation note: this sketch lists the ports and their central methods, not every method of each port. In particular, model calls use a start method plus the existing `seq`-addressed `finished(...)`: the application-layer `JournalingLlm` decorator wraps `LlmPort`, holds the `RunId`, calls `JournalPort.modelStarted` before the call, closes that record through `finished`, and reads the records back with `JournalPort.modelCallsForRun`. The section 9 loop does not journal model calls itself (ADR-0012, Q13). + ### 8.6 Memory - `MemoryEntry` is a short fact, not a transcript. Fields: `id`, `text`, `source` (`user` or `agent`), `createdAt`, `runId`, `tags`, `untrusted`. @@ -263,6 +265,8 @@ Mechanics: The loop lives in the application layer. It works only through `LlmPort`, `ToolPort`, and `PolicyGate`. Tool calls are intercepted before execution: genkit does not run tools by itself. +Implementation note: the loop does not journal the model call itself. `d.llm` arrives already wrapped in the application-layer `JournalingLlm` decorator, which writes the `modelStarted` record and flushes it before delegating to the provider, then records `succeeded` or `failed` after the call returns or throws, and rethrows the original error (ADR-0012, Q13). + ```dart Future run(Run run, AgentDeps d) async { final messages = await d.context.build(run); diff --git a/docs/spec/forbidden-glossary.md b/docs/spec/forbidden-glossary.md index 752cd16..38f652d 100644 --- a/docs/spec/forbidden-glossary.md +++ b/docs/spec/forbidden-glossary.md @@ -52,6 +52,7 @@ Do not use anything listed here, even if you find it in old examples, articles, | Profile | a named agent configuration: model, prompt, tools, budgets, rules | | Tool | an action the model can call | | Invocation | one journal record of a tool call, with its status | +| Model call | one journal record of a call to the model provider, with its status; it carries no prompt and no reply text | | Risk | the effect level of a tool: `read`, `write`, `external`, `destructive` | | PolicyGate | the single decision point for whether a tool runs | | Taint label | a marker that the run read an untrusted source | diff --git a/docs/spec/open-questions.md b/docs/spec/open-questions.md index 87ac97c..01d92cb 100644 --- a/docs/spec/open-questions.md +++ b/docs/spec/open-questions.md @@ -20,10 +20,10 @@ Added during the M0 scaffold (2026-10-17); Q7–Q14 added during M2 (2026-11-08) | Q10 | `configureCore` is synchronous and the journal needs a directory; section 18.3 path resolution is M4. Confirm who resolves the path. | `registerJournal(directory:)` after `configureCore`; the caller owns path and lifetime, no home-directory default | | Q11 | Section 9 enforces only `maxSteps`, while `Budget` also has `maxTokens`, `maxDuration`, and `maxOutputTokens`; `Turn` carries no usage yet, so the loop cannot enforce them. | enforce `maxSteps` only; revisit when the provider reports usage (M3) | | Q12 | An approval refusal is not journalled: section 9 writes `denied` only for policy denials, so `not_approved` leaves no audit record although it stops a side effect. Should a refusal be recorded? | keep the spec order, no extra record in M2; add an approval outcome record when the interactive channel lands (M4/M9) | -| Q13 | Section 8.10 rule 2 lists "model call, tool call, channel send" as effects that must be journalled before they run, but the section 9 loop calls `d.llm.complete(...)` without a journal record and the section 9 snippet omits one. Is model-call journaling and transcript persistence the `JournalPort`'s job, or a decorator in the application service? | out of the loop for M2; decided together with transcript persistence in plan Step 8 | +| Q13 | ~~Section 8.10 rule 2 lists "model call, tool call, channel send" as effects that must be journalled before they run, but the section 9 loop calls `d.llm.complete(...)` without a journal record and the section 9 snippet omits one. Is model-call journaling and transcript persistence the `JournalPort`'s job, or a decorator in the application service?~~ Resolved 2026-11-08: journaling lives in a per-run `JournalingLlm` decorator in the application layer, which writes `modelStarted` before the call and records the outcome after; the loop and the provider stay journal-free. Transcript persistence stays in plan Step 8. See [ADR-0012](../adr/0012-model-call-journaling.md). | `JournalingLlm` decorator around `LlmPort`; the loop and the provider stay journal-free | | Q14 | A timed-out tool cannot be cancelled by `Future.timeout`. Must a call that may still complete be recorded as `unknownOutcome` instead of `failed`? Section 9 writes `failed`. | keep `failed` (section 9); cancel a per-invocation child token and leave the run token alone | -Previously none were open. Decisions are recorded in ADR-0005, ADR-0006, and ADR-0009. +Previously none were open. Decisions are recorded in ADR-0005, ADR-0006, ADR-0009, and ADR-0012 (Q13). ## 25.2 Technical checks [VERIFY] diff --git a/docs/spec/storage.md b/docs/spec/storage.md index 69fa8a7..401c561 100644 --- a/docs/spec/storage.md +++ b/docs/spec/storage.md @@ -14,6 +14,7 @@ - A normal `Box` keeps all values in memory. The journal and memory grow, so `steps` and `memory` are `LazyBox`es. - Values are JSON strings, not type adapters. The format is controlled by the project. Each object carries a version field `v`. +- `steps` holds both tool-invocation and model-call records. A `kind` field (`tool` or `model`) discriminates them; the format version `v` stays `1`. A record without `kind` predates the discriminator and is read as a tool invocation. Both kinds share the same `seq` counter. - `seq` is a shared monotonic counter for SSE. Incrementing it and writing the step happen sequentially inside one isolate. ### 15.2 Write semantics [VERIFY, spike T7] @@ -21,6 +22,7 @@ - Hive documentation: the future of a write completes when the change is written to the backend. If the write fails, the change is reverted. `flush()` writes pending changes to disk. - Whether `flush` performs an fsync is not documented. It is checked by a `kill -9` test in spike T7. - Journal protocol: `await put(started)` and `flush()` before the tool call; `await put(finished)` and `flush()` after it. This guarantees that `started` is on disk before the side effect runs. +- Model calls follow the same protocol: `modelStarted` writes and flushes a `kind: model` record before the provider call, and the `seq`-addressed `finished` update records the outcome. A model-call record carries neither prompt nor reply text. - There are no transactions across boxes. The guarantees are the monotonic `seq` and recovery from states (below). ### 15.3 Recovery after a crash @@ -30,6 +32,8 @@ On startup, the daemon: 2. For every `started` step without `finished`: if `idempotent: true`, the call may be repeated. Otherwise the status is `unknownOutcome`, and the agent asks the owner through a channel. Repeating a sent email or a payment silently is forbidden. 3. The lock file is checked before any box is opened. +Reading a run back needs both record kinds: `forRun` returns tool invocations only, so a run projection merges `forRun` and `modelCallsForRun` in `seq` order to get the full timeline (section 15.1). A `model` record left `started` without a matching `finished` is not safe to retry, because the model call is not idempotent; its recovery outcome is open question Q14 and is not decided here. + Test: `kill -9` in the middle of a tool call, restart, and verify the state transitions in the journal. ### 15.4 Encryption diff --git a/runtime/core/lib/alteri_one_core.dart b/runtime/core/lib/alteri_one_core.dart index 5db55d1..75a0d01 100644 --- a/runtime/core/lib/alteri_one_core.dart +++ b/runtime/core/lib/alteri_one_core.dart @@ -17,6 +17,7 @@ export 'src/application/agent/agent_deps.dart'; export 'src/application/agent/agent_loop.dart'; export 'src/application/agent/approval_service.dart'; export 'src/application/agent/context_builder.dart'; +export 'src/application/agent/journaling_llm.dart'; export 'src/application/agent/run_result.dart'; export 'src/application/agent/tool_set.dart'; export 'src/application/approvals_bloc.dart'; @@ -35,6 +36,7 @@ export 'src/domain/ids.dart'; export 'src/domain/invocation.dart'; export 'src/domain/memory.dart'; export 'src/domain/messages.dart'; +export 'src/domain/model_call.dart'; export 'src/domain/policy.dart'; export 'src/domain/ports/journal_port.dart'; export 'src/domain/ports/llm_port.dart'; diff --git a/runtime/core/lib/src/application/agent/agent_deps.dart b/runtime/core/lib/src/application/agent/agent_deps.dart index 8cdee98..02528b6 100644 --- a/runtime/core/lib/src/application/agent/agent_deps.dart +++ b/runtime/core/lib/src/application/agent/agent_deps.dart @@ -1,12 +1,14 @@ // Copyright 2026 The AlteriOne Authors. // SPDX-License-Identifier: Apache-2.0 +import '../../domain/ids.dart'; import '../../domain/policy.dart'; import '../../domain/ports/journal_port.dart'; import '../../domain/ports/llm_port.dart'; import 'agent_config.dart'; import 'approval_service.dart'; import 'context_builder.dart'; +import 'journaling_llm.dart'; import 'tool_set.dart'; /// The collaborators of the agent loop (docs/spec/domain.md, section 9). @@ -35,3 +37,62 @@ final class AgentDeps( /// Starting conversation. final ContextBuilder context, ); + +/// Hands out [AgentDeps] for one run +/// (docs/adr/0012-model-call-journaling.md). +/// +/// [LlmPort.complete] carries no [RunId], so the journaling decorator cannot be +/// applied per call. The factory holds the raw provider and the collaborators +/// in the same order as [AgentDeps], and wraps the provider in [JournalingLlm] +/// for each run in [forRun]. It is the intended production path; constructing +/// [AgentDeps] directly or resolving [LlmPort] from the container still +/// bypasses the decorator, and enforcing that is deferred. +/// +/// The journal is shared, not copied: the decorator and the tool path use the +/// same instance, so both share one `seq` counter. +final class AgentDepsFactory { + /// Creates the factory from the collaborators of [AgentDeps], in the same + /// order. + new( + this._llm, + this._registry, + this._journal, + this._policy, + this._approvals, + this._config, + this._context, + ); + + /// The undecorated model provider. + final LlmPort _llm; + + /// Source of the tool snapshot for each step. + final ToolRegistry _registry; + + /// Journal written before every call. + final JournalPort _journal; + + /// The only component that decides whether a tool runs. + final PolicyGate _policy; + + /// Owner decisions for [Decision.ask]. + final ApprovalService _approvals; + + /// Per-profile risk overrides. + final AgentConfig _config; + + /// Starting conversation. + final ContextBuilder _context; + + /// The collaborators of one run, with the model provider decorated by + /// [JournalingLlm] for [runId]. + AgentDeps forRun(RunId runId) => AgentDeps( + JournalingLlm(_llm, _journal, runId), + _registry, + _journal, + _policy, + _approvals, + _config, + _context, + ); +} diff --git a/runtime/core/lib/src/application/agent/journaling_llm.dart b/runtime/core/lib/src/application/agent/journaling_llm.dart new file mode 100644 index 0000000..b9bd6b5 --- /dev/null +++ b/runtime/core/lib/src/application/agent/journaling_llm.dart @@ -0,0 +1,102 @@ +// Copyright 2026 The AlteriOne Authors. +// SPDX-License-Identifier: Apache-2.0 + +import 'package:alteri_one_contracts/alteri_one_contracts.dart'; + +import '../../domain/budget.dart'; +import '../../domain/ids.dart'; +import '../../domain/invocation.dart'; +import '../../domain/messages.dart'; +import '../../domain/ports/journal_port.dart'; +import '../../domain/ports/llm_port.dart'; + +/// An [LlmPort] that journals every model call of one run before it runs +/// (docs/adr/0012-model-call-journaling.md). +/// +/// A call first writes a `modelStarted` record and flushes it, then delegates +/// to the wrapped provider. A returned turn is recorded `succeeded`, a thrown +/// call `failed` with a bounded error description, and the original error is +/// rethrown unchanged. Prompt and reply text never reach a journal *field*: +/// its records carry only the run, the sequence number, the status, the time, +/// and the optional error. That error string is not encrypted and providers +/// often echo the prompt, so it is cut to [_maxErrorChars] characters +/// (docs/adr/0012-model-call-journaling.md). +/// +/// There is no timeout: a provider that never returns hangs the run. Timeouts +/// are open question Q11/Q14 territory and are not solved here. +final class JournalingLlm implements LlmPort { + /// Wraps [inner] for [runId], journaling through [journal]. + new(this._inner, this._journal, this._runId); + + /// The provider whose calls are journaled. + final LlmPort _inner; + + /// The journal that receives the start and outcome records. + final JournalPort _journal; + + /// The run the calls belong to. + final RunId _runId; + + /// The longest provider-error description written to the journal. + /// + /// Provider errors often echo the prompt and the journal stores plaintext, + /// so the failure record is bounded; the cut is marked with `…` + /// (docs/adr/0012-model-call-journaling.md). + static const int _maxErrorChars = 200; + + /// A bounded, marked description of [error] for the failure record. + /// + /// The description starts with the runtime type, which keeps bare errors + /// distinguishable, and keeps at most [_maxErrorChars] characters of the + /// message. Only this journal string is cut; [error] itself is rethrown + /// unchanged. + static String _describeError(Object error) { + final description = '${error.runtimeType}: $error'; + if (description.length <= _maxErrorChars) { + return description; + } + final kept = String.fromCharCodes(description.runes.take(_maxErrorChars)); + return '$kept…'; + } + + @override + Future complete({ + required List messages, + required List tools, + required Budget budget, + }) async { + // Invariant 5: the record is written (and flushed) before the side effect. + // If the write throws, the provider must not run, so the error propagates. + final seq = await _journal.modelStarted(_runId); + + final Turn turn; + try { + turn = await _inner.complete( + messages: messages, + tools: tools, + budget: budget, + ); + } catch (err, st) { + // A provider that throws has returned control, so the outcome is known + // to be a failure rather than `unknownOutcome` (ADR-0012). A failing + // outcome write must not mask the model error: swallow it and rethrow + // the original below. + try { + await _journal.finished( + _runId, + seq, + InvocationStatus.failed, + _describeError(err), + ); + } catch (_) { + // Keep the original model error. + } + Error.throwWithStackTrace(err, st); + } + + // Outside the catch: a failing success write must leave the record + // `started` rather than rewrite it as `failed`, and its error propagates. + await _journal.finished(_runId, seq, InvocationStatus.succeeded, null); + return turn; + } +} diff --git a/runtime/core/lib/src/di/core_module.dart b/runtime/core/lib/src/di/core_module.dart index 3b95882..28ad146 100644 --- a/runtime/core/lib/src/di/core_module.dart +++ b/runtime/core/lib/src/di/core_module.dart @@ -41,12 +41,13 @@ abstract class CoreModule { @lazySingleton ToolRegistry get toolRegistry => const EmptyToolRegistry(); - /// The collaborators of the agent loop (docs/spec/domain.md, section 9). + /// The factory that hands out the agent loop's collaborators per run + /// (docs/adr/0012-model-call-journaling.md). /// /// Lazy on purpose: resolving this without a registered `LlmPort` (providers /// arrive in M3) fails at resolution time, not at startup. @lazySingleton - AgentDeps agentDeps( + AgentDepsFactory agentDepsFactory( LlmPort llm, ToolRegistry registry, JournalPort journal, @@ -54,5 +55,13 @@ abstract class CoreModule { ApprovalService approvals, AgentConfig config, ContextBuilder context, - ) => AgentDeps(llm, registry, journal, policy, approvals, config, context); + ) => AgentDepsFactory( + llm, + registry, + journal, + policy, + approvals, + config, + context, + ); } diff --git a/runtime/core/lib/src/di/injection.config.dart b/runtime/core/lib/src/di/injection.config.dart index 3a9901e..0cebef9 100644 --- a/runtime/core/lib/src/di/injection.config.dart +++ b/runtime/core/lib/src/di/injection.config.dart @@ -50,8 +50,8 @@ extension GetItInjectableX on _i174.GetIt { gh.lazySingleton<_i194.ApprovalService>(() => coreModule.approvalService); gh.lazySingleton<_i186.ContextBuilder>(() => coreModule.contextBuilder); gh.lazySingleton<_i454.ToolRegistry>(() => coreModule.toolRegistry); - gh.lazySingleton<_i982.AgentDeps>( - () => coreModule.agentDeps( + gh.lazySingleton<_i982.AgentDepsFactory>( + () => coreModule.agentDepsFactory( gh<_i500.LlmPort>(), gh<_i454.ToolRegistry>(), gh<_i673.JournalPort>(), diff --git a/runtime/core/lib/src/domain/invocation.dart b/runtime/core/lib/src/domain/invocation.dart index f572138..f8ad178 100644 --- a/runtime/core/lib/src/domain/invocation.dart +++ b/runtime/core/lib/src/domain/invocation.dart @@ -5,18 +5,19 @@ import 'package:alteri_one_contracts/alteri_one_contracts.dart'; import 'ids.dart'; -/// Outcome of a journaled tool invocation (docs/spec/domain.md, 8.4). +/// Outcome of any journaled effect, tool invocation or model call +/// (docs/spec/domain.md, section 8.4). enum InvocationStatus { /// The call was written to the journal and has started. started, - /// The tool returned normally. + /// The call returned normally. succeeded, - /// The tool threw. + /// The call threw. failed, - /// The policy denied the call before it started. + /// The policy denied the tool call before it started. denied, /// The process died during the call; recovered from the journal. diff --git a/runtime/core/lib/src/domain/model_call.dart b/runtime/core/lib/src/domain/model_call.dart new file mode 100644 index 0000000..4448365 --- /dev/null +++ b/runtime/core/lib/src/domain/model_call.dart @@ -0,0 +1,31 @@ +// Copyright 2026 The AlteriOne Authors. +// SPDX-License-Identifier: Apache-2.0 + +import 'ids.dart'; +import 'invocation.dart'; + +/// One journaled model call (docs/spec/domain.md, section 8.10). +/// +/// The model call shares the `seq` counter and the `steps` box with tool +/// invocations; a `kind` discriminator separates the two kinds of record +/// (docs/spec/storage.md, section 15.1). Prompt and reply text are +/// deliberately not stored here: the transcript owns them (section 15.6). +final class ModelCall( + /// Journal sequence number, shared with tool invocations. + final int seq, + + /// The run that made the call. + final RunId runId, + + /// Outcome. Shares [InvocationStatus] with tool records, but a model call can + /// only be `started`, `succeeded`, `failed`, or `unknownOutcome`; `denied` + /// never occurs, because the policy gate decides whether a tool runs, not + /// whether a model call does. + final InvocationStatus status, + + /// When the call started. + final DateTime at, + + /// Error text for [InvocationStatus.failed]; `null` otherwise. + final String? error, +); diff --git a/runtime/core/lib/src/domain/ports/journal_port.dart b/runtime/core/lib/src/domain/ports/journal_port.dart index a5f2689..ce6dc80 100644 --- a/runtime/core/lib/src/domain/ports/journal_port.dart +++ b/runtime/core/lib/src/domain/ports/journal_port.dart @@ -5,6 +5,7 @@ import 'package:alteri_one_contracts/alteri_one_contracts.dart'; import '../ids.dart'; import '../invocation.dart'; +import '../model_call.dart'; /// Port of the run journal (docs/spec/domain.md, section 8.5). /// @@ -15,10 +16,15 @@ abstract interface class JournalPort { /// flushed to disk before this returns. Future started(RunId runId, String tool, Risk risk); + /// Records the start of a model call and returns its `seq`. The record is + /// flushed to disk before this returns. + Future modelStarted(RunId runId); + /// Records that the policy denied a call before it started. Future denied(RunId runId, String tool, Risk risk, String reason); - /// Records the outcome of a started call. + /// Records the outcome of a started record, whether a tool invocation or a + /// model call. Future finished( RunId runId, int seq, @@ -26,6 +32,9 @@ abstract interface class JournalPort { String? error, ); - /// Returns all invocations of a run, in `seq` order. + /// Returns all tool invocations of a run, in `seq` order. Future> forRun(RunId runId); + + /// Returns all model calls of a run, in `seq` order. + Future> modelCallsForRun(RunId runId); } diff --git a/runtime/core/lib/src/infrastructure/journal/hive_journal.dart b/runtime/core/lib/src/infrastructure/journal/hive_journal.dart index 75f3725..9558834 100644 --- a/runtime/core/lib/src/infrastructure/journal/hive_journal.dart +++ b/runtime/core/lib/src/infrastructure/journal/hive_journal.dart @@ -8,16 +8,27 @@ import 'package:hive_ce/hive.dart'; import '../../domain/ids.dart'; import '../../domain/invocation.dart'; +import '../../domain/model_call.dart'; import '../../domain/ports/journal_port.dart'; /// Hive-backed implementation of [JournalPort]. +/// +/// One `steps` box holds both tool-invocation and model-call records, told +/// apart by a `kind` discriminator and ordered by the shared `seq` counter +/// (docs/spec/storage.md, section 15.1). final class HiveJournal implements JournalPort { new _(this._meta, this._steps); + /// Kind discriminator for tool-invocation records. + static const String _toolKind = 'tool'; + + /// Kind discriminator for model-call records. + static const String _modelKind = 'model'; + /// Metadata box (sequence counter). final Box _meta; - /// Lazy box holding journaled invocations. + /// Lazy box holding the journaled tool and model records. final LazyBox _steps; /// Opens (or creates) a journal stored under [directory]. @@ -55,6 +66,7 @@ final class HiveJournal implements JournalPort { final now = DateTime.now(); final record = { 'v': 1, + 'kind': _toolKind, 'seq': seq, 'runId': runId.value, 'tool': tool, @@ -68,6 +80,24 @@ final class HiveJournal implements JournalPort { return seq; } + @override + Future modelStarted(RunId runId) async { + final seq = await _nextSeq(); + final now = DateTime.now(); + final record = { + 'v': 1, + 'kind': _modelKind, + 'seq': seq, + 'runId': runId.value, + 'status': InvocationStatus.started.name, + 'at': now.toIso8601String(), + 'error': null, + }; + await _steps.put(_key(runId, seq), jsonEncode(record)); + await _steps.flush(); + return seq; + } + @override Future denied( RunId runId, @@ -79,6 +109,7 @@ final class HiveJournal implements JournalPort { final now = DateTime.now(); final record = { 'v': 1, + 'kind': _toolKind, 'seq': seq, 'runId': runId.value, 'tool': tool, @@ -110,10 +141,18 @@ final class HiveJournal implements JournalPort { await _steps.flush(); } - @override - Future> forRun(RunId runId) async { + /// The kind of [json]. A record written before the discriminator existed has + /// no kind and is a tool invocation (docs/spec/storage.md, 15.1). + String _kindOf(Map json) => + json['kind'] as String? ?? _toolKind; + + /// Decoded records of [runId] whose kind is [kind], in `seq` order. + Future>> _recordsForRun( + RunId runId, + String kind, + ) async { final prefix = '${runId.value}:'; - final result = []; + final result = >[]; for (final key in _steps.keys) { final keyStr = key as String; if (!keyStr.startsWith(prefix)) { @@ -124,7 +163,20 @@ final class HiveJournal implements JournalPort { continue; } final json = jsonDecode(raw) as Map; - result.add( + if (_kindOf(json) != kind) { + continue; + } + result.add(json); + } + result.sort((a, b) => (a['seq'] as int).compareTo(b['seq'] as int)); + return result; + } + + @override + Future> forRun(RunId runId) async { + final records = await _recordsForRun(runId, _toolKind); + return [ + for (final json in records) Invocation( json['seq'] as int, RunId(json['runId'] as String), @@ -134,9 +186,21 @@ final class HiveJournal implements JournalPort { DateTime.parse(json['at'] as String), json['error'] as String?, ), - ); - } - result.sort((a, b) => a.seq.compareTo(b.seq)); - return result; + ]; + } + + @override + Future> modelCallsForRun(RunId runId) async { + final records = await _recordsForRun(runId, _modelKind); + return [ + for (final json in records) + ModelCall( + json['seq'] as int, + RunId(json['runId'] as String), + InvocationStatus.values.byName(json['status'] as String), + DateTime.parse(json['at'] as String), + json['error'] as String?, + ), + ]; } } diff --git a/runtime/core/test/application/agent_fakes.dart b/runtime/core/test/application/agent_fakes.dart index fad9ca4..eb33f7c 100644 --- a/runtime/core/test/application/agent_fakes.dart +++ b/runtime/core/test/application/agent_fakes.dart @@ -39,6 +39,16 @@ final class FakeJournal implements JournalPort { /// tool never runs. bool failOnStarted = false; + /// Number of [modelStarted] calls. + int modelStarts = 0; + + /// One line per [modelStarted] call, in call order. + final List modelRecords = []; + + /// When set, [modelStarted] throws instead of recording, so a test can prove + /// the model call never runs. + bool failOnModelStarted = false; + /// The risk passed to each [denied] call, in order. final List deniedRisks = []; @@ -55,6 +65,18 @@ final class FakeJournal implements JournalPort { return Future.value(_nextSeq++); } + @override + Future modelStarted(RunId runId) { + if (failOnModelStarted) { + return Future.error(StateError('journal unavailable')); + } + modelStarts++; + records.add('modelStarted'); + modelRecords.add('modelStarted'); + log?.add('journal.modelStarted'); + return Future.value(_nextSeq++); + } + @override Future denied(RunId runId, String tool, Risk risk, String reason) { deniedRisks.add(risk); @@ -79,6 +101,10 @@ final class FakeJournal implements JournalPort { @override Future> forRun(RunId runId) => Future.value(const []); + + @override + Future> modelCallsForRun(RunId runId) => + Future.value(const []); } /// A [LlmPort] driven by [onComplete] and recording every request. diff --git a/runtime/core/test/application/chat_end_to_end_test.dart b/runtime/core/test/application/chat_end_to_end_test.dart new file mode 100644 index 0000000..216a541 --- /dev/null +++ b/runtime/core/test/application/chat_end_to_end_test.dart @@ -0,0 +1,301 @@ +// Copyright 2026 The AlteriOne Authors. +// SPDX-License-Identifier: Apache-2.0 + +/// One `chat` run through the production composition +/// (docs/spec/domain.md, section 9; docs/adr/0012-model-call-journaling.md). +/// +/// The run goes context → journaled model call → tool → reply → journal with +/// the real [PromptContextBuilder], [PolicyGate], [DefaultAgentConfig], +/// [DenyAllApprovals], and [HiveJournal]. Only the provider, the tool, and the +/// registry are faked; the journal sits on a real temp directory. +library; + +import 'dart:io'; + +import 'package:alteri_one_contracts/alteri_one_contracts.dart'; +import 'package:alteri_one_core/alteri_one_core.dart'; +import 'package:test/test.dart'; + +import 'agent_fakes.dart'; + +/// A [JournalPort] that writes through [inner] and appends one line per write +/// to [log], so a test can assert the order the journal is written in. +final class _RecordingJournal implements JournalPort { + /// Wraps [inner], logging to [log]. + new(this._inner, this._log); + + /// The real journal. + final JournalPort _inner; + + /// Ordered log shared with the model and tool fakes. + final EventLog _log; + + @override + Future started(RunId runId, String tool, Risk risk) { + _log.add('journal.started $tool'); + return _inner.started(runId, tool, risk); + } + + @override + Future modelStarted(RunId runId) { + _log.add('journal.modelStarted'); + return _inner.modelStarted(runId); + } + + @override + Future denied(RunId runId, String tool, Risk risk, String reason) { + _log.add('journal.denied $tool'); + return _inner.denied(runId, tool, risk, reason); + } + + @override + Future finished( + RunId runId, + int seq, + InvocationStatus status, + String? error, + ) { + _log.add('journal.finished ${status.name}'); + return _inner.finished(runId, seq, status, error); + } + + @override + Future> forRun(RunId runId) => _inner.forRun(runId); + + @override + Future> modelCallsForRun(RunId runId) => + _inner.modelCallsForRun(runId); +} + +/// A [JournalPort] that delegates to [inner] but fails [modelStarted], so a +/// test can prove the model never runs when the journal write fails. +final class _FailOnModelStarted implements JournalPort { + /// Wraps [inner]. + new(this._inner); + + /// The journal that receives every other write. + final JournalPort _inner; + + @override + Future modelStarted(RunId runId) => + Future.error(StateError('journal unavailable')); + + @override + Future started(RunId runId, String tool, Risk risk) => + _inner.started(runId, tool, risk); + + @override + Future denied(RunId runId, String tool, Risk risk, String reason) => + _inner.denied(runId, tool, risk, reason); + + @override + Future finished( + RunId runId, + int seq, + InvocationStatus status, + String? error, + ) => _inner.finished(runId, seq, status, error); + + @override + Future> forRun(RunId runId) => _inner.forRun(runId); + + @override + Future> modelCallsForRun(RunId runId) => + _inner.modelCallsForRun(runId); +} + +/// A real `chat` run with test defaults. +Run _chatRun({ + String prompt = 'hello', + int maxSteps = 4, + Set tools = const {}, +}) => Run( + id: RunId('run-chat'), + profile: 'chat', + mode: RunMode.chat, + prompt: prompt, + budget: Budget(maxSteps, 60_000, const Duration(seconds: 30), 2_048), + capabilities: RunCapabilities(tools, const {}, const [], MemoryRights.none), + cancel: CancelToken(), +); + +/// The production composition, with the profile rule that allows read tools. +AgentDepsFactory _factory({ + required LlmPort llm, + required ToolRegistry registry, + required JournalPort journal, +}) => AgentDepsFactory( + llm, + registry, + journal, + PolicyGate(const {Risk.read: Decision.allow}, false), + const DenyAllApprovals(), + const DefaultAgentConfig(), + const PromptContextBuilder(), +); + +/// A turn in which the model only requests [calls]. +Turn _toolTurn(List calls) => + Turn('', Message.assistant('', calls), calls); + +/// A turn with a final reply and no tool calls. +Turn _finalTurn(String text) => Turn(text, Message.assistant(text), const []); + +/// A well-formed call of the tool [name]. +ToolCall _call(String id, String name) => + ToolCall.valid(id: id, name: name, input: const {'q': 'x'}); + +void main() { + group('chat run end to end', () { + late Directory dir; + late HiveJournal hive; + late EventLog events; + late _RecordingJournal journal; + + setUp(() async { + dir = await Directory.systemTemp.createTemp('chat_end_to_end_'); + hive = await HiveJournal.open(dir.path); + events = EventLog(); + journal = _RecordingJournal(hive, events); + }); + + tearDown(() async { + await hive.close(); + await dir.delete(recursive: true); + }); + + test('returns the model reply and journals the model call', () async { + final run = _chatRun(prompt: 'what is up'); + final llm = FakeLlm((call, messages) => _finalTurn('all good')); + + final result = await runAgent( + run, + _factory( + llm: llm, + registry: FakeRegistry((_) => ToolSet(const [])), + journal: journal, + ).forRun(run.id), + ); + + expect( + result, + isA().having((r) => r.text, 'text', 'all good'), + ); + // The production context builder starts the run from its prompt alone. + expect(llm.requests.first, hasLength(1)); + expect(llm.requests.first.single.role, MessageRole.user); + expect(llm.requests.first.single.text, 'what is up'); + + final modelCalls = await journal.modelCallsForRun(run.id); + expect(modelCalls, hasLength(1)); + expect(modelCalls.single.status, InvocationStatus.succeeded); + expect(await journal.forRun(run.id), isEmpty); + }); + + test('journals a tool round-trip in order', () async { + final run = _chatRun(tools: {'ext__echo'}); + final tool = FakeTool(fakeSpec('echo'), output: 'done', log: events); + final llm = FakeLlm((call, messages) { + if (call == 0) { + return _toolTurn([_call('c1', 'ext__echo')]); + } + return _finalTurn('finished'); + }, events); + + final result = await runAgent( + run, + _factory( + llm: llm, + registry: FakeRegistry((_) => ToolSet([tool])), + journal: journal, + ).forRun(run.id), + ); + + expect(result, isA()); + expect(events.entries, [ + 'journal.modelStarted', + 'llm.complete 0', + 'journal.finished succeeded', + 'journal.started ext__echo', + 'tool.call ext__echo', + 'journal.finished succeeded', + 'journal.modelStarted', + 'llm.complete 1', + 'journal.finished succeeded', + ]); + + expect(tool.calls, 1); + final invocations = await journal.forRun(run.id); + expect(invocations, hasLength(1)); + expect(invocations.single.status, InvocationStatus.succeeded); + + // The second request carries the assistant turn and the tool result. + final secondRequest = llm.requests[1]; + expect(secondRequest.last.role, MessageRole.tool); + expect(secondRequest.last.toolResults, hasLength(1)); + expect(secondRequest.last.toolResults.single.value, 'done'); + expect(secondRequest.last.toolResults.single.isOk, isTrue); + }); + + test( + 'returns budgetExceeded after maxSteps and journals what ran', + () async { + final run = _chatRun(maxSteps: 1, tools: {'ext__echo'}); + final tool = FakeTool(fakeSpec('echo'), output: 'done'); + final llm = FakeLlm( + (call, messages) => _toolTurn([_call('c1', 'ext__echo')]), + ); + + final result = await runAgent( + run, + _factory( + llm: llm, + registry: FakeRegistry((_) => ToolSet([tool])), + journal: journal, + ).forRun(run.id), + ); + + expect(result, isA()); + + final modelCalls = await journal.modelCallsForRun(run.id); + expect(modelCalls, hasLength(1)); + expect(modelCalls.single.status, InvocationStatus.succeeded); + + final invocations = await journal.forRun(run.id); + expect(invocations, hasLength(1)); + expect(invocations.single.tool, 'ext__echo'); + expect(invocations.single.status, InvocationStatus.succeeded); + }, + ); + + test('a failing model-call journal stops the run before the model ' + 'runs', () async { + final run = _chatRun(tools: {'ext__echo'}); + final llm = FakeLlm((call, messages) => _finalTurn('unreachable')); + final failing = _FailOnModelStarted(journal); + + await expectLater( + runAgent( + run, + _factory( + llm: llm, + registry: FakeRegistry((_) => ToolSet(const [])), + journal: failing, + ).forRun(run.id), + ), + throwsA( + isA().having( + (e) => e.message, + 'message', + 'journal unavailable', + ), + ), + ); + + expect(llm.requests, isEmpty); + expect(events.entries, isEmpty); + expect(await failing.modelCallsForRun(run.id), isEmpty); + expect(await failing.forRun(run.id), isEmpty); + }); + }); +} diff --git a/runtime/core/test/application/journaling_llm_test.dart b/runtime/core/test/application/journaling_llm_test.dart new file mode 100644 index 0000000..30dbeb1 --- /dev/null +++ b/runtime/core/test/application/journaling_llm_test.dart @@ -0,0 +1,227 @@ +// Copyright 2026 The AlteriOne Authors. +// SPDX-License-Identifier: Apache-2.0 + +import 'package:alteri_one_contracts/alteri_one_contracts.dart'; +import 'package:alteri_one_core/alteri_one_core.dart'; +import 'package:test/test.dart'; + +import 'agent_fakes.dart'; + +/// A [JournalPort] that delegates to [inner] but always fails [finished], so a +/// test can prove the decorator neither masks nor rewrites the outcome. +final class _FinishFailsJournal implements JournalPort { + /// Wraps [inner]. + new(this._inner); + + /// The journal that records everything but the outcome. + final FakeJournal _inner; + + @override + Future started(RunId runId, String tool, Risk risk) => + _inner.started(runId, tool, risk); + + @override + Future modelStarted(RunId runId) => _inner.modelStarted(runId); + + @override + Future denied(RunId runId, String tool, Risk risk, String reason) => + _inner.denied(runId, tool, risk, reason); + + @override + Future finished( + RunId runId, + int seq, + InvocationStatus status, + String? error, + ) => Future.error(StateError('journal unavailable')); + + @override + Future> forRun(RunId runId) => _inner.forRun(runId); + + @override + Future> modelCallsForRun(RunId runId) => + _inner.modelCallsForRun(runId); +} + +/// A turn with a final reply and no tool calls. +Turn _finalTurn(String text) => Turn(text, Message.assistant(text), const []); + +void main() { + group('JournalingLlm', () { + test( + 'journals the start, then calls the provider, then records success', + () async { + final events = EventLog(); + final journal = FakeJournal(events); + final turn = _finalTurn('hi there'); + final llm = FakeLlm((call, messages) => turn, events); + final decorator = JournalingLlm(llm, journal, RunId('run-1')); + + final result = await decorator.complete( + messages: [Message.user('hello')], + tools: const [], + budget: fakeRun().budget, + ); + + expect( + result, + same(turn), + reason: 'the provider turn is passed through', + ); + expect(events.entries, [ + 'journal.modelStarted', + 'llm.complete 0', + 'journal.finished succeeded', + ]); + expect(journal.records, ['modelStarted', 'finished succeeded 1']); + expect(llm.requests, hasLength(1)); + }, + ); + + test('records a failure and rethrows the same error object', () async { + final journal = FakeJournal(); + final error = StateError('provider down'); + final llm = FakeLlm((call, messages) => throw error); + final decorator = JournalingLlm(llm, journal, RunId('run-1')); + + await expectLater( + decorator.complete( + messages: [Message.user('hello')], + tools: const [], + budget: fakeRun().budget, + ), + throwsA(same(error)), + ); + + expect(journal.records, [ + 'modelStarted', + 'finished failed 1 StateError: Bad state: provider down', + ]); + expect(llm.requests, hasLength(1)); + }); + + test('bounds and marks a long provider error in the journal', () async { + final journal = FakeJournal(); + final error = StateError('SECRET-PROMPT ${'x' * 400}'); + final llm = FakeLlm((call, messages) => throw error); + final decorator = JournalingLlm(llm, journal, RunId('run-1')); + + await expectLater( + decorator.complete( + messages: [Message.user('SECRET-PROMPT')], + tools: const [], + budget: fakeRun().budget, + ), + throwsA(same(error)), + ); + + final record = journal.records.singleWhere( + (entry) => entry.startsWith('finished'), + ); + final stored = record.substring('finished failed 1 '.length); + expect(stored, endsWith('…'), reason: 'a cut message is marked'); + expect( + stored.length, + lessThanOrEqualTo(201), + reason: 'the journaled error stays bounded at 200 chars plus the mark', + ); + }); + + test('rethrows the same long error object after bounding it', () async { + final journal = FakeJournal(); + final error = StateError('SECRET-PROMPT ${'x' * 400}'); + final llm = FakeLlm((call, messages) => throw error); + final decorator = JournalingLlm(llm, journal, RunId('run-1')); + + await expectLater( + decorator.complete( + messages: [Message.user('SECRET-PROMPT')], + tools: const [], + budget: fakeRun().budget, + ), + throwsA(same(error)), + ); + + expect(llm.requests, hasLength(1)); + }); + + test('does not call the provider when the start record fails', () async { + final journal = FakeJournal()..failOnModelStarted = true; + final llm = FakeLlm( + (call, messages) => fail('the provider must not run'), + ); + final decorator = JournalingLlm(llm, journal, RunId('run-1')); + + await expectLater( + decorator.complete( + messages: [Message.user('hello')], + tools: const [], + budget: fakeRun().budget, + ), + throwsA(isA()), + ); + + expect(llm.requests, isEmpty); + expect(journal.records, isEmpty); + }); + + test( + 'propagates a success record failure without rewriting the outcome', + () async { + final inner = FakeJournal(); + final journal = _FinishFailsJournal(inner); + final turn = _finalTurn('hi there'); + final llm = FakeLlm((call, messages) => turn); + final decorator = JournalingLlm(llm, journal, RunId('run-1')); + + await expectLater( + decorator.complete( + messages: [Message.user('hello')], + tools: const [], + budget: fakeRun().budget, + ), + throwsA(isA()), + ); + + expect(inner.records, ['modelStarted']); + expect(llm.requests, hasLength(1)); + }, + ); + + test('keeps the model error when the failure record also fails', () async { + final inner = FakeJournal(); + final journal = _FinishFailsJournal(inner); + final error = StateError('provider down'); + final llm = FakeLlm((call, messages) => throw error); + final decorator = JournalingLlm(llm, journal, RunId('run-1')); + + await expectLater( + decorator.complete( + messages: [Message.user('hello')], + tools: const [], + budget: fakeRun().budget, + ), + throwsA(same(error)), + ); + + expect(inner.records, ['modelStarted']); + }); + + test('journals only metadata, never prompt or reply text', () async { + final journal = FakeJournal(); + final llm = FakeLlm((call, messages) => _finalTurn('SECRET-REPLY')); + final decorator = JournalingLlm(llm, journal, RunId('run-1')); + + await decorator.complete( + messages: [Message.user('SECRET-PROMPT')], + tools: const [], + budget: fakeRun().budget, + ); + + expect(journal.records, ['modelStarted', 'finished succeeded 1']); + final joined = journal.records.join('|'); + expect(joined, isNot(contains('SECRET-PROMPT'))); + expect(joined, isNot(contains('SECRET-REPLY'))); + }); + }); +} diff --git a/runtime/core/test/di/injection_test.dart b/runtime/core/test/di/injection_test.dart index d30fd07..0f5b57b 100644 --- a/runtime/core/test/di/injection_test.dart +++ b/runtime/core/test/di/injection_test.dart @@ -149,8 +149,8 @@ void main() { }); }); - group('AgentDeps', () { - test('resolves from the registered collaborators', () { + group('AgentDepsFactory', () { + test('resolves from the container and decorates the provider', () { configureCore(environment: 'test'); final llm = FakeLlm( (call, messages) => Turn('done', Message.assistant('done'), const []), @@ -160,21 +160,59 @@ void main() { ..registerSingleton(llm) ..registerSingleton(journal); - final deps = getIt(); - expect(identical(deps.llm, llm), isTrue); - expect(identical(deps.registry, getIt()), isTrue); + final deps = getIt().forRun(RunId('run-a')); + + // The production path never hands out the bare provider. + expect(deps.llm, isA()); + expect(identical(deps.llm, llm), isFalse); expect(identical(deps.journal, journal), isTrue); + expect(identical(deps.registry, getIt()), isTrue); expect(identical(deps.policy, getIt()), isTrue); expect(identical(deps.approvals, getIt()), isTrue); expect(identical(deps.config, getIt()), isTrue); expect(identical(deps.context, getIt()), isTrue); }); + test('decorates per run and journals under the run id', () async { + configureCore(environment: 'test'); + tempDir = await Directory.systemTemp.createTemp('injection_deps_'); + await registerJournal(directory: tempDir!.path); + getIt.registerSingleton( + FakeLlm( + (call, messages) => Turn('done', Message.assistant('done'), const []), + ), + ); + + final factory = getIt(); + final first = factory.forRun(RunId('run-a')); + final second = factory.forRun(RunId('run-b')); + + // Separate decorators, one shared journal. + expect(identical(first.llm, second.llm), isFalse); + expect(identical(first.journal, second.journal), isTrue); + + final budget = Budget(4, 60_000, const Duration(seconds: 30), 2_048); + await first.llm.complete( + messages: const [], + tools: const [], + budget: budget, + ); + await second.llm.complete( + messages: const [], + tools: const [], + budget: budget, + ); + + final journal = getIt(); + expect(await journal.modelCallsForRun(RunId('run-a')), hasLength(1)); + expect(await journal.modelCallsForRun(RunId('run-b')), hasLength(1)); + }); + test('fails only when resolved without an LlmPort', () { configureCore(environment: 'test'); getIt.registerSingleton(FakeJournal()); - AgentDeps resolve() => getIt(); + AgentDepsFactory resolve() => getIt(); // Providers land in M3: the binding is lazy, so startup stays green and // only resolution fails. diff --git a/runtime/core/test/infrastructure/hive_journal_test.dart b/runtime/core/test/infrastructure/hive_journal_test.dart index ae8a96b..b413aa8 100644 --- a/runtime/core/test/infrastructure/hive_journal_test.dart +++ b/runtime/core/test/infrastructure/hive_journal_test.dart @@ -1,10 +1,12 @@ // Copyright 2026 The AlteriOne Authors. // SPDX-License-Identifier: Apache-2.0 +import 'dart:convert'; import 'dart:io'; import 'package:alteri_one_contracts/alteri_one_contracts.dart'; import 'package:alteri_one_core/alteri_one_core.dart'; +import 'package:hive_ce/hive.dart'; import 'package:test/test.dart'; void main() { @@ -99,5 +101,98 @@ void main() { throwsA(isA()), ); }); + + test( + 'modelStarted and finished record one successful model call', + () async { + final run = RunId('run-model'); + final seq = await journal.modelStarted(run); + final startedAt = (await journal.modelCallsForRun(run)).single.at; + await journal.finished(run, seq, InvocationStatus.succeeded, null); + + final calls = await journal.modelCallsForRun(run); + expect(calls, hasLength(1)); + expect(calls.single.seq, seq); + expect(calls.single.runId, run); + expect(calls.single.status, InvocationStatus.succeeded); + expect(calls.single.at, startedAt); + expect(calls.single.error, isNull); + }, + ); + + test( + 'model and tool records share the seq counter and filter apart', + () async { + final run = RunId('run-mixed'); + final model1 = await journal.modelStarted(run); + final tool = await journal.started(run, 'ext__post', Risk.read); + final model2 = await journal.modelStarted(run); + expect(model1, 1); + expect(tool, 2); + expect(model2, 3); + + final tools = await journal.forRun(run); + expect(tools.map((i) => i.seq), [tool]); + expect(tools.single.tool, 'ext__post'); + + final models = await journal.modelCallsForRun(run); + expect(models.map((m) => m.seq), [model1, model2]); + }, + ); + + test('forRun ignores model records', () async { + final run = RunId('run-model-only'); + await journal.modelStarted(run); + + expect(await journal.forRun(run), isEmpty); + }); + + test('reads a record without kind as a tool invocation', () async { + final run = RunId('run-legacy'); + await journal.close(); + Hive.init(dir.path); + final steps = await Hive.openLazyBox('steps'); + await steps.put( + 'run-legacy:000000000001', + jsonEncode({ + 'v': 1, + 'seq': 1, + 'runId': 'run-legacy', + 'tool': 'ext__post', + 'risk': Risk.write.name, + 'status': InvocationStatus.started.name, + 'at': DateTime.now().toIso8601String(), + 'error': null, + }), + ); + await steps.flush(); + await steps.close(); + journal = await HiveJournal.open(dir.path); + + final inv = (await journal.forRun(run)).single; + expect(inv.tool, 'ext__post'); + expect(inv.seq, 1); + expect(await journal.modelCallsForRun(run), isEmpty); + }); + + test('model records survive close and reopen', () async { + final run = RunId('run-model-reopen'); + final seq = await journal.modelStarted(run); + await journal.finished( + run, + seq, + InvocationStatus.failed, + 'provider error', + ); + await journal.close(); + journal = await HiveJournal.open(dir.path); + + final call = (await journal.modelCallsForRun(run)).single; + expect(call.seq, seq); + expect(call.status, InvocationStatus.failed); + expect(call.error, 'provider error'); + final next = await journal.modelStarted(run); + expect(next, seq + 1); + }); }); }