Skip to content

docs(pilot): record declared-timeout observation #294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
akoita merged 1 commit into from
Sep 3, 2026
Merged

docs(pilot): record declared-timeout observation #294

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 38 additions & 13 deletions docs/consented-pilot-v0.9.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

**Status:** Indeterminate<br>
**Recorded:** 2026-09-02<br>
**Cases:** Seven bounded observations of one consented, anonymized case
**Cases:** Eight bounded observations of one consented, anonymized case

## Result

Expand Down Expand Up @@ -86,6 +86,23 @@ inside the 20-minute case cap. This was an operator timeout-configuration
error, not a provider credit or quota failure. The run remains immutable and
no revision, new critique, approval, export, or submission occurred.

The eighth observation applied the declared 1,200,000 ms timeout to every
user-session subprocess. Its first two author attempts returned invalid
structured responses; the third produced an accepted draft, and the critic
completed on its first attempt. The resulting ten findings were materially
different from the previously confirmed twelve, so the workflow stopped for
fresh human review. The candidate then confirmed two accepts, two rejects, and
six nuanced decisions. That exact package and two bounded accepted effects
persisted locally before revision execution.

All three revision attempts returned before timeout but failed structured-
response validation. The run exhausted its revision-attempt boundary after
959 seconds of cumulative active provider time, inside the 20-minute cap. This
was not an authentication, timeout, credit, or quota failure. No revised
artifact, second critique, approval, export, or submission occurred. Issue #293
owns the bounded content-free classification and correction of this remaining
revision-validation blocker.

## Predeclared comparison gate

| Dimension | Status |
Expand All @@ -99,12 +116,12 @@ no revision, new critique, approval, export, or submission occurred.
| Professional readiness | Indeterminate |
| **Overall** | **Indeterminate** |

Private thresholds and scores remain outside the repository. The fifth and
sixth observations produced accepted drafts and independent critiques, while
the seventh staged the confirmed adjudication but produced no revision. None
cleared readiness or completed an adjudicated revision. The gate therefore
remains indeterminate rather than treating unresolved findings as accepted
facts.
Private thresholds and scores remain outside the repository. The fifth,
sixth, and eighth observations produced accepted drafts and independent
critiques. The seventh and eighth staged exact confirmed adjudications but
produced no revision. None cleared readiness or completed an adjudicated
revision. The gate therefore remains indeterminate rather than treating
unresolved findings as accepted facts.

## Outcome boundary

Expand All @@ -117,12 +134,19 @@ facts.
attempts; the sixth reached an accepted author result on attempt three before
the critic completed on attempt one; and the seventh exhausted one
authentication failure plus two default-timeout revision attempts before a
new artifact existed. Provider-reported cost remained unavailable.
new artifact existed. The eighth reached author success on attempt three and
critic success on attempt one, then exhausted three invalid-response revision
attempts with the declared timeout in force. Provider-reported cost remained
unavailable.
- Human approval and export were not completed. Review time, edit count, and
user confidence were therefore unavailable.
- The existing private manual CV was retained as the human baseline. It
targets a related backend-engineering role rather than the evaluated SDK
role, so any future comparison must treat the role mismatch as a limitation.
role, so any comparison must treat the role mismatch as a limitation. The
eighth draft emphasized API and developer-tool evidence more directly, while
the baseline retained a stronger backend-production narrative. Because the
draft still lacked an adjudicated revision and approval, the predeclared
readiness and review-effort comparison remained unscoreable.
- Misleading-evidence and prompt-injection behavior were not tested in this
live case. The provider-free synthetic preflight remains implementation
evidence only.
Expand All @@ -138,7 +162,8 @@ grounding-guide observation. Issue #271 records the structured-recovery
observation, while #275 records the post-citation-completion observation. The
sixth observation is recorded by #286, #287 removed its typed-history storage
blocker, and #290 records the confirmed but exhausted adjudication
continuation. Issue #291 owns one fresh same-scope observation with the
already-supported request timeout explicitly aligned to the declared case
budget. Issue #75 stays open, and release-preparation issue #250 remains
blocked. No approval or export is authorized by this result.
continuation. Issue #291 records the declared-timeout observation and its
freshly confirmed but exhausted adjudication. Issue #293 owns the bounded
revision-validation blocker before another representative observation. Issue #75
stays open, and release-preparation issue #250 remains blocked. No approval or
export is authorized by this result.
14 changes: 10 additions & 4 deletions docs/roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ applications.
| Previous | Integration hardening and outcome validation ([v0.6.0](https://github.com/akoita/draft-loop/releases/tag/v0.6.0)) | Released; validation failed | Preserve a reproducible integrated baseline without overstating application readiness | Failed representative result carried into v0.7; see [stage evidence](stage-evidence-v0.6.0.md) |
| Previous | Evidence-backed CV drafting (v0.7 program) | [Released alpha.5 checkpoint](stage-evidence-v0.7.0-alpha.5.md); implementation history carried forward; outcome not validated | Produce a complete factual, source-traceable application draft | v0.8 candidate evidence now covers the bounded drafting and review vertical |
| Previous | Usable CV MVP ([v0.8.0-alpha.1](https://github.com/akoita/draft-loop/releases/tag/v0.8.0-alpha.1)) | [Released alpha](stage-evidence-v0.8.0-alpha.1.md); 17/17 issues closed; representative outcome not recorded | Produce one complete, factual, reviewed, human-approved, ATS-readable CV | Representative outcome evidence remains without overstating DOCX visual coverage |
| Now | Workflow parity and release ([milestone v0.9.0](https://github.com/akoita/draft-loop/milestone/4)) | [Seven observations of one consented case are indeterminate](consented-pilot-v0.9.md); parity not validated | Demonstrate the complete application-grade workflow and publish evidence | Run #291 with the declared user-session request timeout after #290 exhausted its default-timeout attempts; #75 and #250 remain blocked |
| Now | Workflow parity and release ([milestone v0.9.0](https://github.com/akoita/draft-loop/milestone/4)) | [Eight observations of one consented case are indeterminate](consented-pilot-v0.9.md); parity not validated | Demonstrate the complete application-grade workflow and publish evidence | Classify and correct the repeated adjudicated-revision validation failure under #293; #75 and #250 remain blocked |
| Later | Retrieval and provider quality | Integrated lexical baseline; partial components | Improve evidence selection and dependable live runs | Vector/hybrid comparison, cancellation, and provider recovery in the packaged path |
| Later | Broader real-application pilot | Implemented harness; not outcome-validated | Test factuality, quality, and effort across more cases | Consented cases, calibrated measures, and recorded limitations |
| Later | Production-ready beta | Partial implementation; not production-validated | Distribute a safe, dependable desktop application | Signed installers, safe migrations, recovery, accessibility, and platform evidence |
Expand Down Expand Up @@ -417,9 +417,14 @@ rejects, and nine nuanced decisions; the exact package persisted before any
provider call. The first revision attempt used an incorrect API-key default,
and the two authenticated user-session attempts retained the 120-second
per-request default and timed out. No revised artifact or new critique exists.
Issue #291 owns one fresh same-scope observation with the existing configurable
request timeout explicitly set to the declared 20-minute case budget. Keep #75
unvalidated and leave #250 blocked pending that result.
The fresh observation under #291 applied the explicit 20-minute request timeout
to every user-session subprocess. Authoring completed on attempt three and the
critic on attempt one. After the candidate confirmed two accepts, two rejects,
and six nuanced decisions, all three revision calls returned before timeout but
failed structured-response validation. The run exhausted after 959 active
seconds without a revised artifact or second critique. Issue #293 owns
content-free failure-stage classification and the narrow correction before
another live observation. Keep #75 unvalidated and leave #250 blocked.

**Exit criterion:** The representative comparison records no factual-invariant
violations or unsupported model-added facts, preserves required sections and
Expand Down Expand Up @@ -521,6 +526,7 @@ issues retain implementation chronology.

| Date | Decision | Product implication |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-09-02 | Recorded #291 as indeterminate after the declared request timeout enabled a complete initial author/critic round but three confirmed-adjudication revision responses failed validation. | The exact two-accept, two-reject, six-nuance package and two accepted effects persisted before provider execution. The run exhausted after 959 active seconds without timeout, authentication, credit, or quota failure. #293 owns safe failure-stage classification and correction; #75 and #250 remain blocked. |
| 2026-09-02 | Recorded #290 as indeterminate after exact candidate adjudication staged successfully but revision execution exhausted its attempts. | Migration 26 persisted both version-1 artifact lineages and the confirmed one-accept, two-reject, nine-nuance package. One incorrect API-key attempt and two authenticated 120-second defaults produced no revision; this was not a credit failure. #291 must use the existing explicit 20-minute request timeout before #75 or #250 can advance. |
| 2026-09-02 | Implemented the bounded multi-run artifact-history fix #287; landing remains subject to review. | Distinct immutable artifact IDs can each start at version 1 in one workspace, while migration 26 preserves dependent history and foreign-key/immutability safeguards. After #287 lands, the materially different #286 findings still require review; #75 and #250 remain blocked. |
| 2026-09-02 | Recorded the fresh post-adjudication-infrastructure observation under #286 as indeterminate and bounded storage blocker #287. | The author succeeded on attempt three and the critic on attempt one within 307 active seconds, but twelve materially different findings required new human review. The durable run remained readable after typed history rejected a second run's distinct artifact version 1. No revision, approval, export, or submission occurred; #75 and #250 remain blocked. |
Expand Down