Prompt quality as a governed, measurable artifact in the agentic SDLC. GitHub Copilot–native, with adapters for Claude Code, Cursor, Gemini CLI, and any AGENTS.md-aware assistant — all driving the same deterministic engine.
Specification: tools/promptiq/specs/SPEC-COPILOT-PROMPT.md — Part I (§1–§13) defines the defect model, archetypes, detector catalogue, security model, and KPIs; Part II (§14–§22) defines the deterministic rewrite engine: no model in the transformation path, improved prompt only.
# The engine. stdout is the improved prompt and nothing else.
python tools/promptiq/promptiq_rewrite.py --text "the login is broken, fix it" --root .
# With symbol verification and diagnostics on stderr
python tools/promptiq/promptiq_rewrite.py --text "..." --symbol-index symbol_index.json --explain
# Prove determinism
python tools/promptiq/promptiq_rewrite.py --text "..." --check-determinism
# Full acceptance suite (41 checks)
python tools/promptiq/test_rewrite.pyThe engine also reads stdin (echo "fix the login" | python tools/promptiq/promptiq_rewrite.py), and --rules points at an alternative rewrite_rules.yaml. Exit 0 clean, 1 unresolved [NEEDS INPUT:] slots remain (usable as a pre-execution gate), 2 failure. Both the engine and the linter pin stdin/stdout to UTF-8 and strip a leading BOM, so output bytes are identical across OSes and code pages; on Windows PowerShell 5.1, still set the UTF-8 console encodings documented in the adapters — the shell itself mangles non-ASCII before Python sees it otherwise.
For the login is broken, fix it (abridged — full output in tools/promptiq/tests/deterministic-output-sample.txt):
# Task — Bug fix
## Goal
the login is broken.
Outcome that makes this done: [NEEDS INPUT: state the observable end state].
## Requirements
1. fix [NEEDS INPUT: name the component and the incorrect behaviour].
2. Reproduce the defect described below before changing any code.
3. The corrected behaviour is: [NEEDS INPUT: expected behaviour]. ...
## Constraints
- Stack (detected from repository manifest): Python — Python >=3.11.
- Do not reformat, rename, or reorganise code unrelated to the defect.
## Acceptance criteria
- [ ] A regression test reproduces the defect and fails before the fix.
- [ ] Verification: `pytest -q` completes without failures.
## Output contract
Unified diff limited to the files in scope, plus one regression test. ...
## Unresolved inputs
The items below could not be determined from the request or the repository.
Supply them before execution; do not let them be guessed.
- actual behaviour
- expected behaviour
- name the component and the incorrect behaviour
...Every [NEEDS INPUT:] slot is a question the request didn't answer and the repository couldn't. Fill them (or answer them in your original ask — stated facts fill their slots automatically), then hand the spec to your execution agent.
Measured on a repository with a manifest and symbol index:
| Raw prompt | Before | After |
|---|---|---|
the login is broken, fix it |
7/100 | 72/100 |
optimize SessionStore, it should be faster |
5/100 | 86/100 |
add caching to the API |
7/100 | 70/100 |
session.py throws a 500 … should return 401 instead |
15/100 | 92/100 |
Prompts naming no resolvable file cap near 70–75, because context anchoring cannot be earned without an anchor and the engine will not fabricate one. That ceiling is correct behaviour.
The engine cannot infer meaning. It improves quality on four axes that require no understanding: structure imposition (fixed template, closes PD-4), repository fact injection (language, versions, test command read from the manifest — closes much of PD-5), anchor resolution and verification (closes PD-2, detects PD-7), and ambiguity externalisation — every vague term maps to a replacement naming the measurement it is missing, and every unfillable slot becomes a visible [NEEDS INPUT: …] marker. Converting silent ambiguity into visible ambiguity is the quality mechanism. A model-based rewriter faced with "make it fast" produces a plausible latency target the developer accepts without noticing a requirement was invented; this engine cannot invent, so it surfaces the gap instead.
Converts an under-specified developer ask into a structured, context-anchored task specification before any code-mutating agent runs, and enforces authoring standards on every prompt artifact committed to the repository.
"the login is broken, fix it" → score 7/100 BLOCKED
↓ /prompt-improve
structured spec, resolved anchors, → score 92/100 STRONG
executable verification command
| Path | Purpose | Load cost |
|---|---|---|
tools/promptiq/promptiq_rewrite.py |
Deterministic rewrite engine (spec §14–§22) | CLI / agent |
tools/promptiq/rewrite_rules.yaml |
Rewrite rules table — the transformation logic, versioned as a contract | Config |
tools/promptiq/test_rewrite.py |
Spec §21 acceptance suite (determinism, quality delta) | CLI / CI |
tools/promptiq/specs/SPEC-COPILOT-PROMPT.md |
The governing specification (v3.0, consolidated) | Reference |
.github/agents/promptiq.agent.md |
Agent shell: invokes the engine, returns output unaltered | On invocation |
.github/prompts/prompt-improve.prompt.md |
/prompt-improve slash command |
On invocation |
.github/instructions/prompt-authoring.instructions.md |
Always-on authoring standard | Every matching turn |
.github/skills/prompt-rubric/ |
Archetypes, defect classes, Q1–Q7 rubric | On demand |
.github/skills/context-anchors/ |
Anchor resolution and PD-7 verification | On demand |
.github/skills/output-contracts/ |
Template and contract catalogue | On demand |
tools/promptiq/promptiq_lint.py |
Deterministic detectors and scorer | CLI / CI |
tools/promptiq/promptiq_llm_polish.py |
Deterministic gate for the optional advisory LLM polish layer (spec §23) | CLI / agent |
tools/promptiq/rubric.yaml |
Thresholds, weights, term lists | Config |
.github/workflows/prompt-quality.yml |
CI enforcement | CI |
.github/CODEOWNERS |
Ownership of prompt artifacts (§1.1 P6) | Review |
tools/promptiq/tests/run_acceptance.py |
Executable form of the spec §12 acceptance criteria | CLI / CI |
tools/promptiq/tests/test_detectors.py |
Per-detector unit tests (stdlib unittest) |
CLI / CI |
.claude/commands/prompt-improve.md + CLAUDE.md |
Claude Code adapter (/prompt-improve + transport rules) |
On invocation |
.cursor/commands/prompt-improve.md + .cursor/rules/promptiq.mdc |
Cursor adapter (/prompt-improve command + agent-requested rule) |
On invocation / matching ask |
.gemini/commands/prompt-improve.toml + GEMINI.md |
Gemini CLI adapter (/prompt-improve + transport rules) |
On invocation |
AGENTS.md |
Generic adapter for any assistant honouring the AGENTS.md standard | Ambient |
.github/skills/prompt-engineering/ is carried alongside PromptIQ but is not part of the spec: general technique reference, split into references/ so the skill body stays under the token ceiling.
The split matters: instructions are always-on and shape authoring but cannot evaluate; skills carry the rubric (~2.5k tokens) at zero cost on turns that don't need it; the rewrite engine and linter carry every check and transformation that doesn't require judgement, keeping the model out of the transformation path entirely.
Requirements: Python 3.9+. For the chat path, any supported assistant — VS Code with GitHub Copilot Chat, Claude Code, Cursor, Gemini CLI, or an AGENTS.md-aware tool (see Use it with other assistants).
Multi-team model (recommended): install the packaged tool pinned to a release and call the reusable CI workflow — never vendor copies, which fork the contract and cannot receive fixes. Full operating model: docs/GOVERNANCE.md.
pip install "promptiq @ git+https://<host>/<org>/<repo>.git@v3.1.0#subdirectory=tools/promptiq"
# console commands: promptiq-rewrite, promptiq-lint, promptiq-metrics, promptiq-index, promptiq-polish# .github/workflows/prompt-quality.yml in the consuming repo
jobs:
prompt-quality:
uses: <org>/<repo>/.github/workflows/prompt-quality.yml@v3.1.0Per-team tuning is an overlay, never an edit to the packaged base: promptiq-rewrite --rules <base> --rules team-rules.yaml, promptiq-lint --config team-rubric.yaml. Overlays deep-merge over the org contract; unknown keys fail loudly (engine) or surface as notes (linter), so a typo cannot silently change behaviour.
Single-repo evaluation: copying the tree still works —
cp -r .github/agents .github/prompts .github/instructions .github/skills <your-repo>/.github/
cp -r tools/promptiq <your-repo>/tools/
pip install pyyaml # required by the rewrite engine; optional for the linterUsing an assistant other than Copilot? Also copy its adapter — one extra command, listed in the section below.
Confirm the install: python tools/promptiq/test_rewrite.py should report 41/41 and python tools/promptiq/tests/run_acceptance.py should report 16/16.
Reload VS Code after copying the artifacts, then either type /prompt-improve in Copilot Chat and paste the rough ask, or select PromptIQ from the agent picker.
/prompt-improve takes two inputs. ask is the raw request; mode is one of:
| Mode | Tier | What you get back |
|---|---|---|
advisory |
T0 | Deterministic score card of the raw ask only — the prompt is not rewritten. |
rewrite |
T1 | Score card plus the engine's rewritten prompt. Default. You approve before anything executes. |
rewrite_llm |
T1 | As rewrite, plus an LLM-polished advisory variant in a third, clearly-labeled block — emitted only when the deterministic gate (promptiq_llm_polish.py, SPEC §23) proves every [NEEDS INPUT:] slot survived verbatim, the score did not regress, and no new error or security finding appeared; on any failure the gate falls back to the deterministic spec. Never handoff-eligible. |
rewrite_and_handoff |
T2 | As rewrite, then the agent executes the spec — but only when the linter's score card grants handoff: eligible (score ≥ 90, non-mutating archetype, tier-T2 rubric). Opt-in: the shipped rubric is autonomy.tier: T1, under which the linter never grants eligibility — raise the tier in tools/promptiq/rubric.yaml (a reviewed gate change) to enable it. Mutating work never auto-executes; T3 is prohibited. |
Two verdicts on the score card short-circuit the flow: bypass: eligible (SPEC §11 R1) means the ask is a short, anchored analysis question that is answered directly instead of refined, and handoff: gates T2 as above.
Because the rewrite is deterministic, the agent's role is transport, not authorship. Copilot does not rewrite the prompt; it invokes the engine — passing the ask via stdin from a heredoc-written temp file, never interpolated into the command line — and returns the output unmodified, together with the deterministic linter's score card:
/prompt-improve <ask> [mode]
→ agent writes the ask to a temp file outside the repo via quoted
heredoc, so shell metacharacters in the ask are never parsed
→ agent scores the raw ask (advisory stops here; a bypass-eligible
ask is answered directly)
→ agent runs the engine, then re-runs it piped into
promptiq_lint.py --mode inline --baseline <ask> --log
(deterministic, so the scored text is byte-identical) to produce
the score card, token delta, and telemetry line
→ agent emits the score card, then the improved prompt, each verbatim
in its own fenced block, then the linter's handoff verdict
→ agent authors nothing except an optional, clearly-delimited
"Model judgement" note on the dimensions the linter itself marks
as needing model judgement — never a score, never a rewrite
This is the only arrangement that preserves the determinism guarantee end to end. A model that "polishes" the engine output reintroduces variance and can silently fill a [NEEDS INPUT:] slot with a guess — which is the failure mode the whole design exists to prevent. Slots are unanswered questions for you, the developer: fill them, then hand the spec to your execution agent in a fresh turn. PromptIQ never edits files itself.
The always-on authoring standard needs no invocation: any edit to a file matching **/.github/{prompts,instructions,agents,skills}/**/*.md picks it up automatically.
The engine is a plain Python CLI with no model in the transformation path, so any coding assistant that can run a terminal command produces byte-identical output — which model invokes it is irrelevant. Adapters carrying the same transport contract (run the engine and the inline linter, emit their stdout verbatim, never fill a [NEEDS INPUT:] slot) ship for:
| Assistant | Adapter | How to invoke |
|---|---|---|
| GitHub Copilot | .github/agents/ + .github/prompts/ |
/prompt-improve or the PromptIQ agent |
| Claude Code | .claude/commands/prompt-improve.md + CLAUDE.md |
/prompt-improve <ask> |
| Cursor | .cursor/commands/prompt-improve.md + .cursor/rules/promptiq.mdc |
/prompt-improve (deterministic); the rule attaches by description, which is model-dependent — attach @promptiq explicitly when not using the command |
| Gemini CLI | .gemini/commands/prompt-improve.toml + GEMINI.md |
/prompt-improve <ask> |
| Codex & any AGENTS.md-aware tool | AGENTS.md |
Ask to improve/refine a prompt. Best-effort: AGENTS.md is a convention, not a protocol — see the conformance checklist in AGENTS.md |
Copy the adapter for your tool alongside tools/promptiq/:
cp -r .claude .cursor .gemini CLAUDE.md GEMINI.md AGENTS.md <your-repo>/# PowerShell equivalent
Copy-Item .claude,.cursor,.gemini <your-repo>\ -Recurse
Copy-Item CLAUDE.md,GEMINI.md,AGENTS.md <your-repo>\The artifact linter's authoring standard (--mode artifact) validates the Copilot artifacts; the other adapters are CI-enforced by tools/promptiq/tests/test_adapters.py in three layers: drift checks (fails the build if any adapter reintroduces command-line ask interpolation, loses the slot prohibition, or drops the emit-verbatim, engine-failure, mode, or tier-disclosure rules), schema checks (every adapter's frontmatter/TOML must parse and declare tools the rubric allows), and shell-level transport tests that execute the documented bash-heredoc and PowerShell recipes for real with non-ASCII text and shell metacharacters. tests/test_archetype_alignment.py additionally proves the engine and linter report the same archetype for every rendered spec. The determinism guarantee does not depend on the adapter — a disobedient adapter can add commentary around the output, but the improved prompt itself always comes from the engine, and --check-determinism proves it independently of any assistant.
The deterministic engine deliberately cannot tighten prose or suggest answers — that requires judgement. rewrite_llm mode adds that judgement as a separate advisory layer without weakening any guarantee:
deterministic engine ──> spec (canonical) ──> LLM polishes ──> promptiq_llm_polish.py (gate)
├─ accepted → labeled advisory variant, exit 0
└─ any doubt → deterministic spec back, exit 3
- The polisher sees engine output only, never the raw ask — the deterministic pass stays the injection barrier.
- It may tighten wording, add acceptance criteria, and put
> Suggested (unconfirmed): …notes next to[NEEDS INPUT:]slots. It may not fill, remove, or reword a slot, or touch the# Task —header. - The gate (model-free, deterministic) re-lints the candidate and rejects it on any missing slot, changed header, score regression, or new error/security finding. Rejection is silent-safe: you get the deterministic spec.
- Handoff eligibility is always computed from the deterministic output; a polished variant never auto-executes.
- Telemetry (
--log) recordsllm_used,model,accepted, reasons, and the score delta, so you can measure whether the layer earns its cost.
python tools/promptiq/promptiq_rewrite.py --root . < ask.txt > spec.txt
# ...your assistant writes candidate.txt from spec.txt...
python tools/promptiq/promptiq_llm_polish.py --root . --candidate candidate.txt \
--model-id <model> --log < spec.txt # exit 0 accepted, 3 fallback
# or drive an external CLI polisher directly (spec on its stdin):
python tools/promptiq/promptiq_llm_polish.py --root . --llm-cmd "<polish command>" < spec.txtThe engine rewrites; the linter scores. Both are deterministic.
# Rewrite a raw ask into a structured spec
python tools/promptiq/promptiq_rewrite.py --text "add caching to the API" --root .
# Rewrite with symbol verification (catches PD-7 hallucinated premises)
python tools/promptiq/promptiq_rewrite.py --text "..." --symbol-index symbol_index.json
# Score a raw ask
echo "add caching to the API" | python tools/promptiq/promptiq_lint.py --mode inline
# Score with symbol verification
python tools/promptiq/promptiq_lint.py --mode inline \
--symbol-index symbol_index.json < candidate.txt
# Score a refinement, recording token delta, lineage, and telemetry (§9)
python tools/promptiq/promptiq_lint.py --mode inline \
--baseline raw-ask.txt --log --session "$SESSION_ID" < candidate.txt
# Lint committed artifacts (CI)
python tools/promptiq/promptiq_lint.py --mode artifact .github --format json
# Build a polyglot symbol index (Python via ast; TS/JS/Go/Java/Kotlin/C#/Ruby/Rust/PHP/Scala via declaration regexes)
python tools/promptiq/build_symbol_index.py --root . --out symbol_index.json
# Aggregate telemetry across developers/teams and compute K1/K5 (SPEC §9)
python tools/promptiq/promptiq_metrics.py aggregate .promptiq/ team-logs/
# Diff precision (K3) for one refinement: files touched vs files declared
python tools/promptiq/promptiq_metrics.py k3 --spec improved-prompt.md --diff-range main..HEAD
# Residual defects after rewrite, per archetype — targets the next rules change
python tools/promptiq/promptiq_metrics.py gaps .promptiq/ team-logs/
# Verify every acceptance criterion
python tools/promptiq/test_rewrite.py # spec §21
python tools/promptiq/tests/run_acceptance.py # spec §12
python -m unittest discover -s tools/promptiq/tests -p "test_*.py"Useful linter flags: --format json for machine-readable output, --strict to treat warnings as failures, --config to point at a rubric.yaml (repeatable — org base first, team overlays after, deep-merged), --root to set the workspace root used for anchor resolution. The engine's --rules is repeatable the same way.
PromptIQ deterministic score: 7/100 [BLOCKED] archetype=A1-BUGFIX
Q1:0 Q2:0 Q3:0 Q4:0 Q5:1 Q6:0 Q7:5
anchors: 0 file, 0 symbol verified ~7 tokens
contract: UNDECLARED verification: none
defects: PD-1, PD-2, PD-3, PD-4
ERROR VG002 Q1 Objectless imperative: 'fix it'.
fix: Name the object and the desired end state.
Work the errors first — each carries a fix: line. Bands: < 55 BLOCKED, 55–74 REPAIRABLE, 75–89 PASS, ≥ 90 STRONG.
Linter exit codes: 0 clean / pass, 1 warnings or repairable band, 2 errors or blocked band. Engine exit codes: 0 no unresolved inputs, 1 unresolved [NEEDS INPUT:] slots remain, 2 no input or determinism check failed.
Without a symbol index, symbol anchors are reported unverified. Build one from your own sources:
python - <<'EOF'
import ast, json, pathlib
names = set()
for p in pathlib.Path('.').rglob('*.py'):
if any(part in {'.venv', 'node_modules', '.git'} for part in p.parts):
continue
try:
tree = ast.parse(p.read_text(encoding='utf-8', errors='replace'))
except SyntaxError:
continue
for n in ast.walk(tree):
if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
names.add(n.name)
json.dump({"symbols": sorted(names)}, open('symbol_index.json', 'w'))
EOFThe format is just {"symbols": [...]} (entries may also be {"name", "path", "line"} objects, which the engine uses to emit path:line locations), so any indexer that can emit a name list will do.
tools/promptiq/rewrite_rules.yaml holds the vagueness map, archetype signals, per-archetype requirement packs, stack probes, path-anchor extensions, test-directory candidates, and fact-extraction patterns. tools/promptiq/rubric.yaml holds the linter's thresholds, weights, and term lists. Both files are contracts: editing them changes output bytes (engine) or gate behaviour (linter) for every future run. Version them, gate them in CI, review them accordingly — never edit the rules file to make one specific prompt come out differently. Teams extend, they don't edit: pass a second --rules/--config overlay file that deep-merges over the org base (unknown keys are rejected by the engine and surfaced by the linter, so typos fail loudly). The change process for the base contracts is docs/GOVERNANCE.md §3.
.github/workflows/prompt-quality.yml runs on PRs touching prompt artifacts or adapters and on pushes to main, on an ubuntu + windows matrix: it builds a polyglot symbol index, lints .github in artifact mode (errors block; warnings are advisory until rollout Phase 3 flips --strict), runs the unit tests (detectors, config layering, adapter drift), the spec §12 acceptance criteria, the §21 rewrite-engine acceptance suite (byte-identical output across processes and hash seeds), and an explicit determinism self-check. A final job compares an engine-output fingerprint across the two OSes, converting the "byte-identical across platforms" claim into per-run evidence. A separate powershell-transport job runs the adapter-documented stdin transport natively under both Windows PowerShell 5.1 and pwsh with non-ASCII text and shell metacharacters, proving the documented recipe round-trips bytes unmangled. The workflow is callable from consuming repos via workflow_call — pin a tag rather than copying the file.
cat > .git/hooks/pre-commit <<'EOF'
#!/usr/bin/env bash
changed=$(git diff --cached --name-only --diff-filter=ACM | grep -E '\.github/(prompts|instructions|agents|skills)/.*\.md$')
[ -z "$changed" ] && exit 0
python tools/promptiq/promptiq_lint.py --mode artifact $changed || exit 1
EOF
chmod +x .git/hooks/pre-commitReproduce with python tools/promptiq/test_rewrite.py (41/41), python tools/promptiq/tests/run_acceptance.py (16/16), and the unit tests — detectors, config layering, adapter drift/schemas, shell transport, archetype alignment, metrics (75 tests).
| Check | Result |
|---|---|
| Five consecutive runs on identical input | identical SHA-256, also under PYTHONHASHSEED 0/1/12345 |
the login is broken, fix it |
7/100 → 72/100 after rewrite; every vague term slotted |
session.py throws a 500 … should return 401 |
stated expected/actual behaviour fills its slot rather than being re-asked |
Bare filename session.py |
glob-resolved to #file:src/auth/session.py; an ambiguous name is reported as ambiguous, not picked |
validateSession absent from the index |
marked NOT FOUND (PD-7), never silently dropped |
| Compound ask (refactor + endpoint + Dockerfile) | three ordered specs, step 1 of 3 … |
do not refactor anything else |
prohibition not counted as an intent |
make it better |
## Unresolved inputs section enumerated, exit code 1 |
Injection inside a fenced Relevant state block |
SE002 error, handoff blocked |
DROP TABLE in a 100/100 A4 spec at tier T2 |
SE003, handoff withheld |
| Repository's own artifacts | 0 errors, 0 warnings, exit 0 |
Refinement sits between untrusted text and a code-mutating agent, so it is an injection surface. The deterministic engine shrinks that surface structurally: no model participates in the transformation, so pasted text cannot steer a rewrite — fenced blocks are held out of clause parsing and re-emitted as quoted evidence. The linter's controls remain for scoring and CI:
- SE002 — prompt injection. Instruction-override phrasing is screened over the full text including fenced blocks. Error severity; blocks handoff.
- SE003 — destructive operations.
DROP/TRUNCATE,rm -rf, force-push, disabling auth. Warning — an A7-OPS task may legitimately name them — but the A7 pack forces a rollback path and forbids applying changes to live environments. - SE001 — secrets. Literal credentials are errors. Telemetry stores prompt hashes only, never prompt text.
The agent's remaining discretion is a single command invocation; its hard rules forbid rewriting, filling slots, or substituting output. The ask travels to the engine via stdin (quoted heredoc / single-quoted PowerShell here-string), never interpolated into the command line — an ask containing quotes, backticks, or $(...) is data on a pipe, not shell input. Adapter conformance to this rule is CI-tested. The residual risk is a model that disobeys transport rules, which is why the acceptance criterion is byte-identity with engine stdout.
The engine cannot judge whether stated criteria are the right criteria. It compensates by refusing to invent: anything not stated and not discoverable on disk becomes a visible [NEEDS INPUT:] slot. An output with unresolved slots is not a finished prompt; it is a prompt whose remaining defects are enumerated and impossible to miss. Exit code 1 exists so you can gate execution on it.
Measure K2 and K3, not K1. Median prompt score (K1) rises the moment you deploy this and is easy to game. Rework rate (K2) and diff precision (K3) are what justify the tooling existing. Baseline both for two weeks before enabling anything — spec §10 Phase 0 exists for this reason.