Build a regulation improvement feedback loop

[ineedjet]

Summary

Design a feedback loop that improves agent regulations through reviewable text changes rather than model fine-tuning.

Proposed loop

1. Store task input, output artifact, process result, and human feedback.
2. Let a person rate or classify representative runs.
3. Ask an agent to compare the examples with the current regulations and propose focused changes.
4. Open a pull request against the repository that owns those regulations.
5. Apply the merged version to subsequent runs and retain evaluation history.

Design principles

• Every regulation change is inspectable, reversible, and attributable.
• Evaluation examples are durable and can detect regressions.
• The improving agent cannot merge its own regulation changes.
• Consumer data and secrets remain outside StarCast.
• The mechanism should build on the same human-review protocol used for ordinary agent work.

Acceptance criteria

• Define the run/feedback/evaluation data model.
• Define how examples are selected and protected from sensitive-data leakage.
• Define the pull-request generation and human approval boundary.
• Include rollback and regression detection.
• Compare the approach with prompt optimization and reflection techniques, including Reflexion and DSPy, without requiring model training.

[ineedjet]

#11 now specifies the proposal-lifecycle primitives (Propose/Apply/Reject/Distill/Rework) this loop runs on top of — Distill there is this issue's loop applied to one proposal at a time: diff what an agent originally proposed against what a human edited directly, classify the lesson, and open a PR against the regulations when it's a taxonomy/policy or tooling lesson rather than an issue-specific one.

Also worth noting there: the native GitHub Copilot coding agent doesn't fit this loop — it's task-execution-oriented (assign it work, it tries to complete it), not task-improvement-oriented, and has no concept of posting a structured proposal or participating in this lifecycle. This needs purpose-built logic, though it can run as an ephemeral GitHub Actions job rather than a persistent hosted runtime.

[ineedjet]

Split Distill's implementation out into its own issue, #86, now that #11 (which originally scoped it as a placeholder) is closed — #86 is the concrete mechanism this issue's loop runs on top of.