feat: structured LLM responses for pre-flight and the writing guide

.github/workflows/evals.yaml

@@ -0,0 +1,113 @@
+name: evals
+
+# Live evals of pre-flight, the writing guide and summarize through Opper, one
+# path per model at its production setting, reporting each case's result and
+# the cost (20261101-010604-d-tac-769, 20261102-101250-d-tac-s98). The job
+# holds OPPER_API_KEY, so it runs only on manual dispatch or when a pull
+# request from this repository gets the `eval` label: never on push, never for
+# a fork.
+on:
+  workflow_dispatch:
+    inputs:
+      runs:
+        description: Runs per pass-rate case (empty keeps each case's own count)
+        default: ""
+  pull_request:
+    types: [labeled]
+
+permissions:
+  contents: read
+
+concurrency:
+  group: evals-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  evals:
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      (github.event.label.name == 'eval' &&
+       github.event.pull_request.head.repo.full_name == github.repository)
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          # Opper's OpenAI-compatible route takes the full chat completions path.
+          # GLM is pinned to a route with structured output; the pooled id also
+          # reaches providers that list none.
+          - name: glm-5.3-flash
+            provider: openai
+            model: greenpt/glm-5.3-flash
+            params: reasoning_effort=low
+            endpoint: https://api.opper.ai/v3/compat/chat/completions
+          - name: gpt-5.6-luna
+            provider: openai
+            model: openai/gpt-5.6-luna
+            params: ""
+            endpoint: https://api.opper.ai/v3/compat/chat/completions
+          - name: claude-haiku-5-5
+            provider: anthropic
+            model: anthropic/claude-haiku-5-5
+            params: ""
+            endpoint: https://api.opper.ai/v3/compat
+    steps:
+      - uses: actions/checkout@v5
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+      - name: Write the eval config
+        env:
+          OPPER_API_KEY: ${{ secrets.OPPER_API_KEY }}
+          ENDPOINT: ${{ matrix.endpoint }}
+        run: |
+          test -n "$OPPER_API_KEY" || { echo "OPPER_API_KEY is not set"; exit 1; }
+          cat > "$RUNNER_TEMP/eval-config.yaml" <<EOF
+          llm:
+            endpoint: $ENDPOINT
+            timeout: 4m
+            api_keys:
+              openai: "$OPPER_API_KEY"
+              anthropic: "$OPPER_API_KEY"
+          EOF
+      - name: Run the evals
+        env:
+          SDD_EVAL_CONFIG: ${{ runner.temp }}/eval-config.yaml
+          SDD_EVAL_PROVIDER: ${{ matrix.provider }}
+          SDD_EVAL_MODEL: ${{ matrix.model }}
+          SDD_EVAL_PARAMS: ${{ matrix.params }}
+          SDD_EVAL_RUNS: ${{ inputs.runs }}
+          # One judge for every candidate, so all are scored alike.
+          SDD_EVAL_JUDGE_PROVIDER: openai
+          SDD_EVAL_JUDGE_MODEL: greenpt/glm-5.3-flash
+          SDD_EVAL_JUDGE_PARAMS: reasoning_effort=high
+          SDD_EVAL_JUDGE_ENDPOINT: https://api.opper.ai/v3/compat/chat/completions
+          SDD_EVAL_RESPONSES_DIR: ${{ github.workspace }}/eval-out
+          SDD_EVAL_STATS_DIR: ${{ github.workspace }}/eval-out
+        # The sweeps assert nothing, so they stay out.
+        run: |
+          mkdir -p eval-out
+          go test -tags=eval -json -run '^(TestPreflightEval|TestWritingGuideEval|TestSummarizeEval|TestSummarizeJudgeCalibration)' -parallel 4 -timeout 80m ./internal/llmops/ \
+            | tee eval-out/test.jsonl | jq -rj 'select(.Action == "output") | .Output' | tee eval-out/eval.log
+        shell: bash -o pipefail {0}
+      - name: Report results and cost
+        if: always()
+        run: |
+          test -f eval-out/test.jsonl || { echo "no test results written"; exit 0; }
+          {
+            echo "## Cases"
+            echo
+            echo "| Test | Result | Seconds |"
+            echo "|---|---|---|"
+            jq -rs '[.[] | select(.Test != null and (.Action == "pass" or .Action == "fail" or .Action == "skip"))] | sort_by(.Test)[] | "| \(.Test) | \(.Action) | \(.Elapsed) |"' eval-out/test.jsonl
+            echo
+            if test -f eval-out/cost-summary.md; then cat eval-out/cost-summary.md; else echo "no cost summary written"; fi
+          } >> "$GITHUB_STEP_SUMMARY"
+      - name: Keep the raw responses
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: evals-${{ matrix.name }}
+          path: eval-out/
+          if-no-files-found: warn

.sdd/graph/2026/11/01-002900-s-tac-3q0.md

@@ -0,0 +1,30 @@
+---
+type: signal
+layer: tactical
+kind: gap
+refs:
+    - id: 20261031-141038-d-tac-p8l
+      kind: surfaced-by
+      desc: its format evals surfaced the deviation; the plan counts Ollama among the providers that support structured output
+    - id: 20260901-093840-s-tac-ahc
+      kind: grounded-in
+      desc: the recommendation of glm-5.3-flash on Ollama Cloud for all three purposes, the expected side
+    - id: 20261004-135748-d-tac-ndh
+      kind: grounded-in
+      desc: the rule that an endpoint ignoring the schema fails loudly, and its revisit clause
+participants:
+    - Christopher
+confidence: high
+topics:
+    - llm/providers
+    - reliability/pre-flight
+summary: Ollama Cloud ignores a request's response schema for `glm-5.3-flash:cloud`, returning prose on every route probed, so pre-flight and the writing guide fail to parse on the configuration the provider evaluation (20260901-093840-s-tac-ahc) recommends; Ollama tracks it as ollama/ollama#12362. The structured-output plan's format evals (20261031-141038-d-tac-p8l) surfaced the deviation, the native-enforcement requirement (20261004-135748-d-tac-ndh) stays in force, and through Opper on an EU route the same model returned parseable responses in every run.
+---
+
+Ollama Cloud ignores a request's response schema, so pre-flight and the writing guide, which now send a schema instead of describing JSON in the prompt, fail to parse on `glm-5.3-flash:cloud`, the configuration the provider evaluation recommends.
+
+**Actual.** On 2026-11-01 `https://ollama.com` returned prose for `glm-5.3-flash:cloud` on `/api/generate` and `/api/chat` with `format` and on `/v1/chat/completions` with a strict `response_format`, whatever the thinking setting; a local Ollama daemon returned schema-valid JSON for the same request. On this model the format evals of the structured-output plan (20261031-141038-d-tac-p8l) returned mostly prose, where nearly every response parsed before the change ([probe record](./01-002900-s-tac-3q0/probe-record.md)). Ollama tracks the behavior as [ollama/ollama#12362](https://github.com/ollama/ollama/issues/12362), "JSON reply schema is ignored by Cloud model", open since 2025-09-21.
+
+**Expected.** The plan counts Ollama among the providers that support structured output, and the provider evaluation (20260901-093840-s-tac-ahc) recommends `glm-5.3-flash` at `think=low` on Ollama Cloud for all three purposes.
+
+Christopher kept the native-enforcement requirement (20261004-135748-d-tac-ndh) rather than revisiting it. Remedy for this model: through Opper on an EU route (`greenpt/glm-5.3-flash`, `reasoning_effort: low`) the same evals returned parseable responses in every run on 2026-11-01.

.sdd/graph/2026/11/01-002900-s-tac-3q0/probe-record.md

@@ -0,0 +1,26 @@
+# Ollama Cloud schema probe, 2026-11-01
+
+Request body, sent to `https://ollama.com` with the account's bearer key (cloud) and to `http://localhost:11434` (local daemon):
+
+- schema: `{"type":"object","properties":{"greeting":{"type":"string"},"language":{"type":"string","enum":["de","en"]}},"required":["greeting","language"],"additionalProperties":false}`
+- prompt: "Write two sentences about the weather in Hamburg, in German, as plain prose." (a prompt that pulls away from JSON, so only enforcement yields JSON)
+- `stream: false`
+
+| Endpoint | Route | Schema field | Thinking | Model | Result |
+|---|---|---|---|---|---|
+| ollama.com | `/api/generate` | `format` | unset | glm-5.3-flash:cloud | prose |
+| ollama.com | `/api/generate` | `format` | `low` | glm-5.3-flash:cloud | prose |
+| ollama.com | `/api/generate` | `format` | `false` | glm-5.3-flash:cloud | prose (leaked reasoning) |
+| ollama.com | `/api/chat` (system + user) | `format` | unset | glm-5.3-flash:cloud | prose |
+| ollama.com | `/api/chat` (system + user) | `format` | `low` | glm-5.3-flash:cloud | prose |
+| ollama.com | `/v1/chat/completions` | strict `response_format` | unset | glm-5.3-flash:cloud | prose |
+| localhost | `/api/generate` | `format` | unset | gemma4:26b | schema-valid JSON |
+| localhost | `/api/generate` | `format` | `low` | gemma4:26b | schema-valid JSON |
+| localhost | `/api/generate` | `format` | `false` | gemma4:26b | schema-valid JSON |
+
+Format evals (`TestWritingGuideEval_Format`, `TestPreflightEval_Format`, 3 runs per specimen and operation, 60 runs) on `ollama glm-5.3-flash:cloud think=low`:
+
+- before the structured-output change (prompts ask for JSON, no schema sent): 58 of 60 parseable; both failures were a second JSON object after leaked reasoning
+- after it (schema sent as `format`, no JSON instructions): 5 of 60 parseable; the rest were Markdown prose, some opening with "Findings: []"
+
+For comparison after the change: claude-sonnet-4-6 (Anthropic API) 60 of 60, gpt-5-mini (OpenAI API) 60 of 60, claude CLI sonnet 60 of 60.

.sdd/graph/2026/11/01-010604-d-tac-769.md

@@ -0,0 +1,29 @@
+---
+type: decision
+layer: tactical
+kind: directive
+refs:
+    - id: 20261031-141038-d-tac-p8l
+      kind: refines
+      desc: sets the models its GitHub Actions workflow runs and keeps it to model coverage
+    - id: 20261031-174216-d-tac-p83
+      kind: related
+      desc: the eval workflow is the one sdd eval later takes over
+participants:
+    - Christopher
+confidence: high
+intent: pending
+topics:
+    - dev/ci
+    - llm/evaluation
+    - reliability/testing
+summary: 'The structured-output plan''s GitHub Actions eval workflow runs three models through Opper, one path per model at production settings: `glm-5.3-flash` at `reasoning_effort: low`, and `openai/gpt-5.6-luna` and `anthropic/claude-haiku-5-5` at their default `medium` effort, reporting schema validity and the cost per model and setting; Claude Sonnet 4.6 is left out for poor cost-performance. It refines the plan (20261031-141038-d-tac-p8l) with the models it left open, pins GLM to an Opper route with structured output, defers lower-effort comparisons to the judge of `sdd eval`, which takes the workflow over (20261031-174216-d-tac-p83), and sets aside a provider smoke workflow, since checks through Opper say little about the vendors'' own APIs and the matrix already runs both adapters.'
+---
+
+The structured-output plan's GitHub Actions workflow evaluates three models through Opper, one path per model, each at the setting it would run with in production: `glm-5.3-flash` at `reasoning_effort: low` and `openai/gpt-5.6-luna` at its default effort (`medium`) on the OpenAI-compatible route, and `anthropic/claude-haiku-5-5` at its default effort (`medium`) on the Messages route. Each run reports schema validity and the cost per model and setting. This refines the plan (20261031-141038-d-tac-p8l), which named Opper's two routes but no models; the workflow is the one `sdd eval` later takes over (20261031-174216-d-tac-p83).
+
+All models run through Opper, so evaluation needs one key and one bill. GLM is pinned to an Opper route that supports structured output, because the pooled `glm-5.3-flash` also routes to providers that list none, and a format failure there would not be SDD's. Claude Sonnet 4.6 is left out, performing too poorly for its cost (Christopher). Data residency plays no role, since the eval material is anonymized or public.
+
+Lower efforts are not compared yet: effort levels do not match across model families, and whether quality holds at a cheaper setting needs the judge `sdd eval` brings. Each combination is recorded by its variant, so cost and latency already separate per setting. The cost is the one Opper reports with every response.
+
+A separate provider smoke workflow, one job per adapter, was set aside. Through Opper it checks the adapters against a translation layer that can be more tolerant than the vendors' own APIs: claude-haiku-5-5 accepted a temperature through Opper that Anthropic documents as rejected for that model. Covering the vendors and Ollama would take more keys and a runner with a local model. And the matrix already runs both adapters, so a broken one shows as every run of its model failing. Request shapes stay covered by the deterministic wire tests; live checks against the vendors' APIs and Ollama stay local runs.

.sdd/graph/2026/11/01-110510-s-tac-buz.md

@@ -0,0 +1,31 @@
+---
+type: signal
+layer: tactical
+kind: done
+refs:
+    - id: 20261031-141038-d-tac-p8l
+      kind: addresses
+      desc: 'delivers its criteria except the merge and closing #20 and #21, so it stays open'
+    - id: 20261101-010604-d-tac-769
+      kind: addresses
+      desc: carries out its model matrix and cost report
+    - id: 20261101-002900-s-tac-3q0
+      kind: surfaces
+      desc: the live evals found Ollama Cloud ignoring response schemas
+participants:
+    - Christopher
+confidence: high
+topics:
+    - llm/providers
+    - reliability/testing
+    - dev/ci
+summary: 'Structured output for pre-flight and the writing guide is built on branch `structured-llm-responses` and open as unmerged PR #26: both checks send a JSON schema each adapter enforces natively and the prompts no longer teach JSON; after the change the format evals returned valid JSON on the Anthropic API, the OpenAI API, the Claude CLI and three models through Opper, and a pre-flight calibration run scored unchanged. It delivers the plan''s criteria (20261031-141038-d-tac-p8l) except the merge and closing #20/#21, carries out the model-matrix and cost-report directive (20261101-010604-d-tac-769), and surfaced that Ollama Cloud ignores response schemas (20261101-002900-s-tac-3q0).'
+---
+
+Structured output for pre-flight and the writing guide is built on branch `structured-llm-responses` and open as PR #26, unmerged: both checks send a JSON schema that each adapter enforces natively, and the prompts no longer teach JSON. This covers the structured-output plan (20261031-141038-d-tac-p8l) except its closing steps, the merge and closing GitHub #20 and #21, so the plan stays open.
+
+**Delivered.** In sdd: `4fb9af75` format evals over a hostile-character example set in data files, with #21's cases anonymized; `55186d3f` `ResponseSchema` on `pkg/llm.Request` and the adapters (Claude CLI `--json-schema`; strict `response_format`, `output_config.format` and `format` through the gollm fork; `llm.endpoint` for the anthropic provider); `62cec3f9` the schemas as ordered JSON files, prompts without JSON construction rules, quotes that only locate their passage; `dca0a76b` capture's instruction to locate the passage from a quote; `0421a997` and `d332ed4b` the format-evals workflow; `b12440f6` and `58607295` the cost Opper reports, in usage and eval results. In networkteam/gollm `generate-with-usage`, pinned at `698361b`: `14f7165` the configurable Anthropic base URL, `c460d23` the system prompt sent once and per call, `30e7a78` OpenAI parameters without model-name matching, `25ba06d` the schema path on the regular request path, `698361b` the reported cost.
+
+**Checked.** Before the change the format evals failed on the Anthropic API and on Ollama Cloud. After it every run returned valid JSON on the Anthropic API, the OpenAI API, the Claude CLI and `greenpt/glm-5.3-flash` through Opper, and local Ollama returned no invalid response. The workflow's first run on PR #26 returned valid JSON in every run for glm-5.3-flash, gpt-5.6-luna and claude-haiku-5-5, and reported the cost per model. A paired run of the pre-flight calibration suite on glm through Opper scored the same before and after.
+
+**Along the way.** Reading the fork turned up two defects: it sent the system prompt twice on OpenAI and Anthropic, and overlapping calls could send each other's system prompt. Both were reproduced by failing tests and fixed on all request paths (Christopher). The Claude CLI keeps its single stdin payload, so it has no prompt split to keep. Ollama Cloud ignores response schemas (20261101-002900-s-tac-3q0). The workflow's models and cost report were settled in 20261101-010604-d-tac-769, which this work carries out.

.sdd/graph/2026/11/01-112131-s-tac-nk8.md

@@ -0,0 +1,38 @@
+---
+type: signal
+layer: tactical
+kind: gap
+refs:
+    - id: 20261101-110510-s-tac-buz
+      kind: surfaced-by
+      desc: holds the actual side
+    - id: 20261031-141038-d-tac-p8l
+      kind: related
+      desc: asks for validity only and leaves content checks to sdd eval
+    - id: 20261101-010604-d-tac-769
+      kind: related
+      desc: sets the models; recorded the narrow reading
+    - id: 20261031-174216-d-tac-p83
+      kind: grounded-in
+      desc: replaces these tests, so the fix stays simple
+    - id: 20260919-144857-s-prc-a1x
+      kind: related
+      desc: the narrowing pattern this run's part fits
+participants:
+    - Christopher
+confidence: high
+topics:
+    - llm/evaluation
+    - reliability/testing
+    - dev/ci
+    - reliability/discipline
+summary: The live evals of the structured-output run only assert that responses parse against the schema, and the GitHub Actions workflow runs nothing else, so whether findings and quotes are right goes unchecked. Christopher wants a few specimens with problem characters in the pre-flight and writing-guide eval tests, asserting expected findings and passage-locating quotes, and the workflow to run the pre-flight, writing-guide and summarize eval tests for the models of 20261101-010604-d-tac-769, kept simple since `sdd eval` replaces these tests (20261031-174216-d-tac-p83). The plan (20261031-141038-d-tac-p8l) asked for validity only and left content judgment to `sdd eval`; the run built to it, and answering Christopher's objection with the format evals again fits the narrowing pattern (20260919-144857-s-prc-a1x). The miss surfaced at the done's playback (20261101-110510-s-tac-buz).
+---
+
+The structured-output run's live evals only check that responses parse, and its GitHub Actions workflow runs nothing else; Christopher wants a few specimens in the existing eval tests that check content, run in the workflow for each decided model.
+
+**Actual.** The format evals (`4fb9af75`) make about 60 calls per model and assert only that responses parse against the schema; the workflow (`d332ed4b`) runs only them. With the schema enforced by the provider, that repeats a guarantee and says nothing about whether findings and quotes are right.
+
+**Expected** (Christopher, 2026-11-01). A few specimens with problem characters in the pre-flight and writing-guide eval tests, asserting the expected findings and quotes that locate their passage; the workflow runs the pre-flight, writing-guide and summarize eval tests for the models of 20261101-010604-d-tac-769. Kept simple, since `sdd eval` replaces these tests (20261031-174216-d-tac-p83).
+
+**How it was missed.** The plan (20261031-141038-d-tac-p8l) asks for validity only and leaves "judgment tests, whether findings hold and quotes point to the right passage" to `sdd eval`, a split the planning agent proposed and Christopher confirmed. The run built to the plan. When Christopher objected that he expected the eval tests to run per model, the run answered with the format evals again and recorded 20261101-010604-d-tac-769 that way; this part fits the narrowing pattern of 20260919-144857-s-prc-a1x. The miss surfaced at the done's playback (20261101-110510-s-tac-buz).