diff --git a/docs/consented-pilot-v0.9.md b/docs/consented-pilot-v0.9.md index bab0b27..96f06ea 100644 --- a/docs/consented-pilot-v0.9.md +++ b/docs/consented-pilot-v0.9.md @@ -2,7 +2,7 @@ **Status:** Indeterminate
**Recorded:** 2026-09-02
-**Cases:** Seven bounded observations of one consented, anonymized case +**Cases:** Eight bounded observations of one consented, anonymized case ## Result @@ -86,6 +86,23 @@ inside the 20-minute case cap. This was an operator timeout-configuration error, not a provider credit or quota failure. The run remains immutable and no revision, new critique, approval, export, or submission occurred. +The eighth observation applied the declared 1,200,000 ms timeout to every +user-session subprocess. Its first two author attempts returned invalid +structured responses; the third produced an accepted draft, and the critic +completed on its first attempt. The resulting ten findings were materially +different from the previously confirmed twelve, so the workflow stopped for +fresh human review. The candidate then confirmed two accepts, two rejects, and +six nuanced decisions. That exact package and two bounded accepted effects +persisted locally before revision execution. + +All three revision attempts returned before timeout but failed structured- +response validation. The run exhausted its revision-attempt boundary after +959 seconds of cumulative active provider time, inside the 20-minute cap. This +was not an authentication, timeout, credit, or quota failure. No revised +artifact, second critique, approval, export, or submission occurred. Issue #293 +owns the bounded content-free classification and correction of this remaining +revision-validation blocker. + ## Predeclared comparison gate | Dimension | Status | @@ -99,12 +116,12 @@ no revision, new critique, approval, export, or submission occurred. | Professional readiness | Indeterminate | | **Overall** | **Indeterminate** | -Private thresholds and scores remain outside the repository. The fifth and -sixth observations produced accepted drafts and independent critiques, while -the seventh staged the confirmed adjudication but produced no revision. None -cleared readiness or completed an adjudicated revision. The gate therefore -remains indeterminate rather than treating unresolved findings as accepted -facts. +Private thresholds and scores remain outside the repository. The fifth, +sixth, and eighth observations produced accepted drafts and independent +critiques. The seventh and eighth staged exact confirmed adjudications but +produced no revision. None cleared readiness or completed an adjudicated +revision. The gate therefore remains indeterminate rather than treating +unresolved findings as accepted facts. ## Outcome boundary @@ -117,12 +134,19 @@ facts. attempts; the sixth reached an accepted author result on attempt three before the critic completed on attempt one; and the seventh exhausted one authentication failure plus two default-timeout revision attempts before a - new artifact existed. Provider-reported cost remained unavailable. + new artifact existed. The eighth reached author success on attempt three and + critic success on attempt one, then exhausted three invalid-response revision + attempts with the declared timeout in force. Provider-reported cost remained + unavailable. - Human approval and export were not completed. Review time, edit count, and user confidence were therefore unavailable. - The existing private manual CV was retained as the human baseline. It targets a related backend-engineering role rather than the evaluated SDK - role, so any future comparison must treat the role mismatch as a limitation. + role, so any comparison must treat the role mismatch as a limitation. The + eighth draft emphasized API and developer-tool evidence more directly, while + the baseline retained a stronger backend-production narrative. Because the + draft still lacked an adjudicated revision and approval, the predeclared + readiness and review-effort comparison remained unscoreable. - Misleading-evidence and prompt-injection behavior were not tested in this live case. The provider-free synthetic preflight remains implementation evidence only. @@ -138,7 +162,8 @@ grounding-guide observation. Issue #271 records the structured-recovery observation, while #275 records the post-citation-completion observation. The sixth observation is recorded by #286, #287 removed its typed-history storage blocker, and #290 records the confirmed but exhausted adjudication -continuation. Issue #291 owns one fresh same-scope observation with the -already-supported request timeout explicitly aligned to the declared case -budget. Issue #75 stays open, and release-preparation issue #250 remains -blocked. No approval or export is authorized by this result. +continuation. Issue #291 records the declared-timeout observation and its +freshly confirmed but exhausted adjudication. Issue #293 owns the bounded +revision-validation blocker before another representative observation. Issue #75 +stays open, and release-preparation issue #250 remains blocked. No approval or +export is authorized by this result. diff --git a/docs/roadmap.md b/docs/roadmap.md index bf24a64..b6ffa43 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -177,7 +177,7 @@ applications. | Previous | Integration hardening and outcome validation ([v0.6.0](https://github.com/akoita/draft-loop/releases/tag/v0.6.0)) | Released; validation failed | Preserve a reproducible integrated baseline without overstating application readiness | Failed representative result carried into v0.7; see [stage evidence](stage-evidence-v0.6.0.md) | | Previous | Evidence-backed CV drafting (v0.7 program) | [Released alpha.5 checkpoint](stage-evidence-v0.7.0-alpha.5.md); implementation history carried forward; outcome not validated | Produce a complete factual, source-traceable application draft | v0.8 candidate evidence now covers the bounded drafting and review vertical | | Previous | Usable CV MVP ([v0.8.0-alpha.1](https://github.com/akoita/draft-loop/releases/tag/v0.8.0-alpha.1)) | [Released alpha](stage-evidence-v0.8.0-alpha.1.md); 17/17 issues closed; representative outcome not recorded | Produce one complete, factual, reviewed, human-approved, ATS-readable CV | Representative outcome evidence remains without overstating DOCX visual coverage | -| Now | Workflow parity and release ([milestone v0.9.0](https://github.com/akoita/draft-loop/milestone/4)) | [Seven observations of one consented case are indeterminate](consented-pilot-v0.9.md); parity not validated | Demonstrate the complete application-grade workflow and publish evidence | Run #291 with the declared user-session request timeout after #290 exhausted its default-timeout attempts; #75 and #250 remain blocked | +| Now | Workflow parity and release ([milestone v0.9.0](https://github.com/akoita/draft-loop/milestone/4)) | [Eight observations of one consented case are indeterminate](consented-pilot-v0.9.md); parity not validated | Demonstrate the complete application-grade workflow and publish evidence | Classify and correct the repeated adjudicated-revision validation failure under #293; #75 and #250 remain blocked | | Later | Retrieval and provider quality | Integrated lexical baseline; partial components | Improve evidence selection and dependable live runs | Vector/hybrid comparison, cancellation, and provider recovery in the packaged path | | Later | Broader real-application pilot | Implemented harness; not outcome-validated | Test factuality, quality, and effort across more cases | Consented cases, calibrated measures, and recorded limitations | | Later | Production-ready beta | Partial implementation; not production-validated | Distribute a safe, dependable desktop application | Signed installers, safe migrations, recovery, accessibility, and platform evidence | @@ -417,9 +417,14 @@ rejects, and nine nuanced decisions; the exact package persisted before any provider call. The first revision attempt used an incorrect API-key default, and the two authenticated user-session attempts retained the 120-second per-request default and timed out. No revised artifact or new critique exists. -Issue #291 owns one fresh same-scope observation with the existing configurable -request timeout explicitly set to the declared 20-minute case budget. Keep #75 -unvalidated and leave #250 blocked pending that result. +The fresh observation under #291 applied the explicit 20-minute request timeout +to every user-session subprocess. Authoring completed on attempt three and the +critic on attempt one. After the candidate confirmed two accepts, two rejects, +and six nuanced decisions, all three revision calls returned before timeout but +failed structured-response validation. The run exhausted after 959 active +seconds without a revised artifact or second critique. Issue #293 owns +content-free failure-stage classification and the narrow correction before +another live observation. Keep #75 unvalidated and leave #250 blocked. **Exit criterion:** The representative comparison records no factual-invariant violations or unsupported model-added facts, preserves required sections and @@ -521,6 +526,7 @@ issues retain implementation chronology. | Date | Decision | Product implication | | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-09-02 | Recorded #291 as indeterminate after the declared request timeout enabled a complete initial author/critic round but three confirmed-adjudication revision responses failed validation. | The exact two-accept, two-reject, six-nuance package and two accepted effects persisted before provider execution. The run exhausted after 959 active seconds without timeout, authentication, credit, or quota failure. #293 owns safe failure-stage classification and correction; #75 and #250 remain blocked. | | 2026-09-02 | Recorded #290 as indeterminate after exact candidate adjudication staged successfully but revision execution exhausted its attempts. | Migration 26 persisted both version-1 artifact lineages and the confirmed one-accept, two-reject, nine-nuance package. One incorrect API-key attempt and two authenticated 120-second defaults produced no revision; this was not a credit failure. #291 must use the existing explicit 20-minute request timeout before #75 or #250 can advance. | | 2026-09-02 | Implemented the bounded multi-run artifact-history fix #287; landing remains subject to review. | Distinct immutable artifact IDs can each start at version 1 in one workspace, while migration 26 preserves dependent history and foreign-key/immutability safeguards. After #287 lands, the materially different #286 findings still require review; #75 and #250 remain blocked. | | 2026-09-02 | Recorded the fresh post-adjudication-infrastructure observation under #286 as indeterminate and bounded storage blocker #287. | The author succeeded on attempt three and the critic on attempt one within 307 active seconds, but twelve materially different findings required new human review. The durable run remained readable after typed history rejected a second run's distinct artifact version 1. No revision, approval, export, or submission occurred; #75 and #250 remain blocked. |