Skip to content

eval(pilot): demonstrate parity with the proven manual CV workflow #75

Description

@akoita

Parent: #4

Outcome

Demonstrate that DraftLoop can reproduce the value of the proven manual author-critic workflow with less repetitive user effort.

Scope

  • Use a consented private human-authored application CV as the local baseline.
  • Keep raw candidate material, job content, provider responses, and reference CV private.
  • Commit only sanitized structural fixtures and aggregate measurements.
  • Define the private thresholds for complete, professional, application-ready, and comparable before evaluating the run.
  • Compare first draft, critic report, revised draft, and final human-approved output.
  • Measure factual invariants, unsupported additions, required-section completeness, chronology, relevant-achievement recall, critical-requirement coverage, style violations, ATS/visual readiness, manual edits, review time, rounds, provider cost, and confidence.

Acceptance criteria

  • Zero factual invariant violations and unsupported model-added facts.
  • Required sections and chronology are preserved unless explicitly omitted.
  • Relevant-achievement recall and requirement coverage meet the agreed private baseline threshold.
  • Final output is judged usable for a real application after bounded human review.
  • Results and limitations are recorded without publishing personal data.
  • One consented case is reported as representative evidence, not general product validation; broader confidence requires later pilot cases.
  • Adversarial and misleading-source cases remain part of the evidence.
  • pnpm validate passes.

Dependencies

  • v0.7 evidence-backed drafting milestone
  • v0.8 independent review and readiness milestone

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions