Skip to content

Repository files navigation

PromptIQ

Prompt quality as a governed, measurable artifact in the agentic SDLC. GitHub Copilot–native, with adapters for Claude Code, Cursor, Gemini CLI, and any AGENTS.md-aware assistant — all driving the same deterministic engine.

Specification: tools/promptiq/specs/SPEC-COPILOT-PROMPT.md — Part I (§1–§13) defines the defect model, archetypes, detector catalogue, security model, and KPIs; Part II (§14–§22) defines the deterministic rewrite engine: no model in the transformation path, improved prompt only.

Deterministic mode

# The engine. stdout is the improved prompt and nothing else.
python tools/promptiq/promptiq_rewrite.py --text "the login is broken, fix it" --root .

# With symbol verification and diagnostics on stderr
python tools/promptiq/promptiq_rewrite.py --text "..." --symbol-index symbol_index.json --explain

# Prove determinism
python tools/promptiq/promptiq_rewrite.py --text "..." --check-determinism

# Full acceptance suite (41 checks)
python tools/promptiq/test_rewrite.py

The engine also reads stdin (echo "fix the login" | python tools/promptiq/promptiq_rewrite.py), and --rules points at an alternative rewrite_rules.yaml. Exit 0 clean, 1 unresolved [NEEDS INPUT:] slots remain (usable as a pre-execution gate), 2 failure. Both the engine and the linter pin stdin/stdout to UTF-8 and strip a leading BOM, so output bytes are identical across OSes and code pages; on Windows PowerShell 5.1, still set the UTF-8 console encodings documented in the adapters — the shell itself mangles non-ASCII before Python sees it otherwise.

What the output looks like

For the login is broken, fix it (abridged — full output in tools/promptiq/tests/deterministic-output-sample.txt):

# Task — Bug fix

## Goal
the login is broken.
Outcome that makes this done: [NEEDS INPUT: state the observable end state].

## Requirements
1. fix [NEEDS INPUT: name the component and the incorrect behaviour].
2. Reproduce the defect described below before changing any code.
3. The corrected behaviour is: [NEEDS INPUT: expected behaviour]. ...

## Constraints
- Stack (detected from repository manifest): Python — Python >=3.11.
- Do not reformat, rename, or reorganise code unrelated to the defect.

## Acceptance criteria
- [ ] A regression test reproduces the defect and fails before the fix.
- [ ] Verification: `pytest -q` completes without failures.

## Output contract
Unified diff limited to the files in scope, plus one regression test. ...

## Unresolved inputs
The items below could not be determined from the request or the repository.
Supply them before execution; do not let them be guessed.
- actual behaviour
- expected behaviour
- name the component and the incorrect behaviour
...

Every [NEEDS INPUT:] slot is a question the request didn't answer and the repository couldn't. Fill them (or answer them in your original ask — stated facts fill their slots automatically), then hand the spec to your execution agent.

Measured on a repository with a manifest and symbol index:

Raw prompt Before After
the login is broken, fix it 7/100 72/100
optimize SessionStore, it should be faster 5/100 86/100
add caching to the API 7/100 70/100
session.py throws a 500 … should return 401 instead 15/100 92/100

Prompts naming no resolvable file cap near 70–75, because context anchoring cannot be earned without an anchor and the engine will not fabricate one. That ceiling is correct behaviour.

The engine cannot infer meaning. It improves quality on four axes that require no understanding: structure imposition (fixed template, closes PD-4), repository fact injection (language, versions, test command read from the manifest — closes much of PD-5), anchor resolution and verification (closes PD-2, detects PD-7), and ambiguity externalisation — every vague term maps to a replacement naming the measurement it is missing, and every unfillable slot becomes a visible [NEEDS INPUT: …] marker. Converting silent ambiguity into visible ambiguity is the quality mechanism. A model-based rewriter faced with "make it fast" produces a plausible latency target the developer accepts without noticing a requirement was invented; this engine cannot invent, so it surfaces the gap instead.

What it does

Converts an under-specified developer ask into a structured, context-anchored task specification before any code-mutating agent runs, and enforces authoring standards on every prompt artifact committed to the repository.

"the login is broken, fix it"          →   score 7/100   BLOCKED
                                            ↓ /prompt-improve
structured spec, resolved anchors,     →   score 92/100  STRONG
executable verification command

Layout

Path Purpose Load cost
tools/promptiq/promptiq_rewrite.py Deterministic rewrite engine (spec §14–§22) CLI / agent
tools/promptiq/rewrite_rules.yaml Rewrite rules table — the transformation logic, versioned as a contract Config
tools/promptiq/test_rewrite.py Spec §21 acceptance suite (determinism, quality delta) CLI / CI
tools/promptiq/specs/SPEC-COPILOT-PROMPT.md The governing specification (v3.0, consolidated) Reference
.github/agents/promptiq.agent.md Agent shell: invokes the engine, returns output unaltered On invocation
.github/prompts/prompt-improve.prompt.md /prompt-improve slash command On invocation
.github/instructions/prompt-authoring.instructions.md Always-on authoring standard Every matching turn
.github/skills/prompt-rubric/ Archetypes, defect classes, Q1–Q7 rubric On demand
.github/skills/context-anchors/ Anchor resolution and PD-7 verification On demand
.github/skills/output-contracts/ Template and contract catalogue On demand
tools/promptiq/promptiq_lint.py Deterministic detectors and scorer CLI / CI
tools/promptiq/promptiq_llm_polish.py Deterministic gate for the optional advisory LLM polish layer (spec §23) CLI / agent
tools/promptiq/rubric.yaml Thresholds, weights, term lists Config
.github/workflows/prompt-quality.yml CI enforcement CI
.github/CODEOWNERS Ownership of prompt artifacts (§1.1 P6) Review
tools/promptiq/tests/run_acceptance.py Executable form of the spec §12 acceptance criteria CLI / CI
tools/promptiq/tests/test_detectors.py Per-detector unit tests (stdlib unittest) CLI / CI
.claude/commands/prompt-improve.md + CLAUDE.md Claude Code adapter (/prompt-improve + transport rules) On invocation
.cursor/commands/prompt-improve.md + .cursor/rules/promptiq.mdc Cursor adapter (/prompt-improve command + agent-requested rule) On invocation / matching ask
.gemini/commands/prompt-improve.toml + GEMINI.md Gemini CLI adapter (/prompt-improve + transport rules) On invocation
AGENTS.md Generic adapter for any assistant honouring the AGENTS.md standard Ambient

.github/skills/prompt-engineering/ is carried alongside PromptIQ but is not part of the spec: general technique reference, split into references/ so the skill body stays under the token ceiling.

The split matters: instructions are always-on and shape authoring but cannot evaluate; skills carry the rubric (~2.5k tokens) at zero cost on turns that don't need it; the rewrite engine and linter carry every check and transformation that doesn't require judgement, keeping the model out of the transformation path entirely.

Install

Requirements: Python 3.9+. For the chat path, any supported assistant — VS Code with GitHub Copilot Chat, Claude Code, Cursor, Gemini CLI, or an AGENTS.md-aware tool (see Use it with other assistants).

Multi-team model (recommended): install the packaged tool pinned to a release and call the reusable CI workflow — never vendor copies, which fork the contract and cannot receive fixes. Full operating model: docs/GOVERNANCE.md.

pip install "promptiq @ git+https://<host>/<org>/<repo>.git@v3.1.0#subdirectory=tools/promptiq"
# console commands: promptiq-rewrite, promptiq-lint, promptiq-metrics, promptiq-index, promptiq-polish
# .github/workflows/prompt-quality.yml in the consuming repo
jobs:
  prompt-quality:
    uses: <org>/<repo>/.github/workflows/prompt-quality.yml@v3.1.0

Per-team tuning is an overlay, never an edit to the packaged base: promptiq-rewrite --rules <base> --rules team-rules.yaml, promptiq-lint --config team-rubric.yaml. Overlays deep-merge over the org contract; unknown keys fail loudly (engine) or surface as notes (linter), so a typo cannot silently change behaviour.

Single-repo evaluation: copying the tree still works —

cp -r .github/agents .github/prompts .github/instructions .github/skills <your-repo>/.github/
cp -r tools/promptiq <your-repo>/tools/
pip install pyyaml     # required by the rewrite engine; optional for the linter

Using an assistant other than Copilot? Also copy its adapter — one extra command, listed in the section below.

Confirm the install: python tools/promptiq/test_rewrite.py should report 41/41 and python tools/promptiq/tests/run_acceptance.py should report 16/16.

Use it in Copilot Chat

Reload VS Code after copying the artifacts, then either type /prompt-improve in Copilot Chat and paste the rough ask, or select PromptIQ from the agent picker.

/prompt-improve takes two inputs. ask is the raw request; mode is one of:

Mode Tier What you get back
advisory T0 Deterministic score card of the raw ask only — the prompt is not rewritten.
rewrite T1 Score card plus the engine's rewritten prompt. Default. You approve before anything executes.
rewrite_llm T1 As rewrite, plus an LLM-polished advisory variant in a third, clearly-labeled block — emitted only when the deterministic gate (promptiq_llm_polish.py, SPEC §23) proves every [NEEDS INPUT:] slot survived verbatim, the score did not regress, and no new error or security finding appeared; on any failure the gate falls back to the deterministic spec. Never handoff-eligible.
rewrite_and_handoff T2 As rewrite, then the agent executes the spec — but only when the linter's score card grants handoff: eligible (score ≥ 90, non-mutating archetype, tier-T2 rubric). Opt-in: the shipped rubric is autonomy.tier: T1, under which the linter never grants eligibility — raise the tier in tools/promptiq/rubric.yaml (a reviewed gate change) to enable it. Mutating work never auto-executes; T3 is prohibited.

Two verdicts on the score card short-circuit the flow: bypass: eligible (SPEC §11 R1) means the ask is a short, anchored analysis question that is answered directly instead of refined, and handoff: gates T2 as above.

Because the rewrite is deterministic, the agent's role is transport, not authorship. Copilot does not rewrite the prompt; it invokes the engine — passing the ask via stdin from a heredoc-written temp file, never interpolated into the command line — and returns the output unmodified, together with the deterministic linter's score card:

/prompt-improve <ask> [mode]
   → agent writes the ask to a temp file outside the repo via quoted
     heredoc, so shell metacharacters in the ask are never parsed
   → agent scores the raw ask (advisory stops here; a bypass-eligible
     ask is answered directly)
   → agent runs the engine, then re-runs it piped into
     promptiq_lint.py --mode inline --baseline <ask> --log
     (deterministic, so the scored text is byte-identical) to produce
     the score card, token delta, and telemetry line
   → agent emits the score card, then the improved prompt, each verbatim
     in its own fenced block, then the linter's handoff verdict
   → agent authors nothing except an optional, clearly-delimited
     "Model judgement" note on the dimensions the linter itself marks
     as needing model judgement — never a score, never a rewrite

This is the only arrangement that preserves the determinism guarantee end to end. A model that "polishes" the engine output reintroduces variance and can silently fill a [NEEDS INPUT:] slot with a guess — which is the failure mode the whole design exists to prevent. Slots are unanswered questions for you, the developer: fill them, then hand the spec to your execution agent in a fresh turn. PromptIQ never edits files itself.

The always-on authoring standard needs no invocation: any edit to a file matching **/.github/{prompts,instructions,agents,skills}/**/*.md picks it up automatically.

Use it with other assistants

The engine is a plain Python CLI with no model in the transformation path, so any coding assistant that can run a terminal command produces byte-identical output — which model invokes it is irrelevant. Adapters carrying the same transport contract (run the engine and the inline linter, emit their stdout verbatim, never fill a [NEEDS INPUT:] slot) ship for:

Assistant Adapter How to invoke
GitHub Copilot .github/agents/ + .github/prompts/ /prompt-improve or the PromptIQ agent
Claude Code .claude/commands/prompt-improve.md + CLAUDE.md /prompt-improve <ask>
Cursor .cursor/commands/prompt-improve.md + .cursor/rules/promptiq.mdc /prompt-improve (deterministic); the rule attaches by description, which is model-dependent — attach @promptiq explicitly when not using the command
Gemini CLI .gemini/commands/prompt-improve.toml + GEMINI.md /prompt-improve <ask>
Codex & any AGENTS.md-aware tool AGENTS.md Ask to improve/refine a prompt. Best-effort: AGENTS.md is a convention, not a protocol — see the conformance checklist in AGENTS.md

Copy the adapter for your tool alongside tools/promptiq/:

cp -r .claude .cursor .gemini CLAUDE.md GEMINI.md AGENTS.md <your-repo>/
# PowerShell equivalent
Copy-Item .claude,.cursor,.gemini <your-repo>\ -Recurse
Copy-Item CLAUDE.md,GEMINI.md,AGENTS.md <your-repo>\

The artifact linter's authoring standard (--mode artifact) validates the Copilot artifacts; the other adapters are CI-enforced by tools/promptiq/tests/test_adapters.py in three layers: drift checks (fails the build if any adapter reintroduces command-line ask interpolation, loses the slot prohibition, or drops the emit-verbatim, engine-failure, mode, or tier-disclosure rules), schema checks (every adapter's frontmatter/TOML must parse and declare tools the rubric allows), and shell-level transport tests that execute the documented bash-heredoc and PowerShell recipes for real with non-ASCII text and shell metacharacters. tests/test_archetype_alignment.py additionally proves the engine and linter report the same archetype for every rendered spec. The determinism guarantee does not depend on the adapter — a disobedient adapter can add commentary around the output, but the improved prompt itself always comes from the engine, and --check-determinism proves it independently of any assistant.

Optional LLM polish layer (SPEC §23)

The deterministic engine deliberately cannot tighten prose or suggest answers — that requires judgement. rewrite_llm mode adds that judgement as a separate advisory layer without weakening any guarantee:

deterministic engine ──> spec (canonical) ──> LLM polishes ──> promptiq_llm_polish.py (gate)
                                                                  ├─ accepted → labeled advisory variant, exit 0
                                                                  └─ any doubt → deterministic spec back, exit 3
  • The polisher sees engine output only, never the raw ask — the deterministic pass stays the injection barrier.
  • It may tighten wording, add acceptance criteria, and put > Suggested (unconfirmed): … notes next to [NEEDS INPUT:] slots. It may not fill, remove, or reword a slot, or touch the # Task — header.
  • The gate (model-free, deterministic) re-lints the candidate and rejects it on any missing slot, changed header, score regression, or new error/security finding. Rejection is silent-safe: you get the deterministic spec.
  • Handoff eligibility is always computed from the deterministic output; a polished variant never auto-executes.
  • Telemetry (--log) records llm_used, model, accepted, reasons, and the score delta, so you can measure whether the layer earns its cost.
python tools/promptiq/promptiq_rewrite.py --root . < ask.txt > spec.txt
# ...your assistant writes candidate.txt from spec.txt...
python tools/promptiq/promptiq_llm_polish.py --root . --candidate candidate.txt \
  --model-id <model> --log < spec.txt          # exit 0 accepted, 3 fallback

# or drive an external CLI polisher directly (spec on its stdin):
python tools/promptiq/promptiq_llm_polish.py --root . --llm-cmd "<polish command>" < spec.txt

Use it from the CLI

The engine rewrites; the linter scores. Both are deterministic.

# Rewrite a raw ask into a structured spec
python tools/promptiq/promptiq_rewrite.py --text "add caching to the API" --root .

# Rewrite with symbol verification (catches PD-7 hallucinated premises)
python tools/promptiq/promptiq_rewrite.py --text "..." --symbol-index symbol_index.json

# Score a raw ask
echo "add caching to the API" | python tools/promptiq/promptiq_lint.py --mode inline

# Score with symbol verification
python tools/promptiq/promptiq_lint.py --mode inline \
  --symbol-index symbol_index.json < candidate.txt

# Score a refinement, recording token delta, lineage, and telemetry (§9)
python tools/promptiq/promptiq_lint.py --mode inline \
  --baseline raw-ask.txt --log --session "$SESSION_ID" < candidate.txt

# Lint committed artifacts (CI)
python tools/promptiq/promptiq_lint.py --mode artifact .github --format json

# Build a polyglot symbol index (Python via ast; TS/JS/Go/Java/Kotlin/C#/Ruby/Rust/PHP/Scala via declaration regexes)
python tools/promptiq/build_symbol_index.py --root . --out symbol_index.json

# Aggregate telemetry across developers/teams and compute K1/K5 (SPEC §9)
python tools/promptiq/promptiq_metrics.py aggregate .promptiq/ team-logs/

# Diff precision (K3) for one refinement: files touched vs files declared
python tools/promptiq/promptiq_metrics.py k3 --spec improved-prompt.md --diff-range main..HEAD

# Residual defects after rewrite, per archetype — targets the next rules change
python tools/promptiq/promptiq_metrics.py gaps .promptiq/ team-logs/

# Verify every acceptance criterion
python tools/promptiq/test_rewrite.py                     # spec §21
python tools/promptiq/tests/run_acceptance.py             # spec §12
python -m unittest discover -s tools/promptiq/tests -p "test_*.py"

Useful linter flags: --format json for machine-readable output, --strict to treat warnings as failures, --config to point at a rubric.yaml (repeatable — org base first, team overlays after, deep-merged), --root to set the workspace root used for anchor resolution. The engine's --rules is repeatable the same way.

Reading the linter output

PromptIQ deterministic score: 7/100  [BLOCKED]  archetype=A1-BUGFIX
  Q1:0  Q2:0  Q3:0  Q4:0  Q5:1  Q6:0  Q7:5
  anchors: 0 file, 0 symbol verified   ~7 tokens
  contract: UNDECLARED   verification: none
  defects: PD-1, PD-2, PD-3, PD-4
  ERROR  VG002 Q1  Objectless imperative: 'fix it'.
          fix: Name the object and the desired end state.

Work the errors first — each carries a fix: line. Bands: < 55 BLOCKED, 55–74 REPAIRABLE, 75–89 PASS, ≥ 90 STRONG.

Linter exit codes: 0 clean / pass, 1 warnings or repairable band, 2 errors or blocked band. Engine exit codes: 0 no unresolved inputs, 1 unresolved [NEEDS INPUT:] slots remain, 2 no input or determinism check failed.

Verifying #sym: anchors

Without a symbol index, symbol anchors are reported unverified. Build one from your own sources:

python - <<'EOF'
import ast, json, pathlib
names = set()
for p in pathlib.Path('.').rglob('*.py'):
    if any(part in {'.venv', 'node_modules', '.git'} for part in p.parts):
        continue
    try:
        tree = ast.parse(p.read_text(encoding='utf-8', errors='replace'))
    except SyntaxError:
        continue
    for n in ast.walk(tree):
        if isinstance(n, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            names.add(n.name)
json.dump({"symbols": sorted(names)}, open('symbol_index.json', 'w'))
EOF

The format is just {"symbols": [...]} (entries may also be {"name", "path", "line"} objects, which the engine uses to emit path:line locations), so any indexer that can emit a name list will do.

Tuning

tools/promptiq/rewrite_rules.yaml holds the vagueness map, archetype signals, per-archetype requirement packs, stack probes, path-anchor extensions, test-directory candidates, and fact-extraction patterns. tools/promptiq/rubric.yaml holds the linter's thresholds, weights, and term lists. Both files are contracts: editing them changes output bytes (engine) or gate behaviour (linter) for every future run. Version them, gate them in CI, review them accordingly — never edit the rules file to make one specific prompt come out differently. Teams extend, they don't edit: pass a second --rules/--config overlay file that deep-merges over the org base (unknown keys are rejected by the engine and surfaced by the linter, so typos fail loudly). The change process for the base contracts is docs/GOVERNANCE.md §3.

CI

.github/workflows/prompt-quality.yml runs on PRs touching prompt artifacts or adapters and on pushes to main, on an ubuntu + windows matrix: it builds a polyglot symbol index, lints .github in artifact mode (errors block; warnings are advisory until rollout Phase 3 flips --strict), runs the unit tests (detectors, config layering, adapter drift), the spec §12 acceptance criteria, the §21 rewrite-engine acceptance suite (byte-identical output across processes and hash seeds), and an explicit determinism self-check. A final job compares an engine-output fingerprint across the two OSes, converting the "byte-identical across platforms" claim into per-run evidence. A separate powershell-transport job runs the adapter-documented stdin transport natively under both Windows PowerShell 5.1 and pwsh with non-ASCII text and shell metacharacters, proving the documented recipe round-trips bytes unmangled. The workflow is callable from consuming repos via workflow_call — pin a tag rather than copying the file.

Pre-commit hook

cat > .git/hooks/pre-commit <<'EOF'
#!/usr/bin/env bash
changed=$(git diff --cached --name-only --diff-filter=ACM | grep -E '\.github/(prompts|instructions|agents|skills)/.*\.md$')
[ -z "$changed" ] && exit 0
python tools/promptiq/promptiq_lint.py --mode artifact $changed || exit 1
EOF
chmod +x .git/hooks/pre-commit

Verified behaviour

Reproduce with python tools/promptiq/test_rewrite.py (41/41), python tools/promptiq/tests/run_acceptance.py (16/16), and the unit tests — detectors, config layering, adapter drift/schemas, shell transport, archetype alignment, metrics (75 tests).

Check Result
Five consecutive runs on identical input identical SHA-256, also under PYTHONHASHSEED 0/1/12345
the login is broken, fix it 7/100 → 72/100 after rewrite; every vague term slotted
session.py throws a 500 … should return 401 stated expected/actual behaviour fills its slot rather than being re-asked
Bare filename session.py glob-resolved to #file:src/auth/session.py; an ambiguous name is reported as ambiguous, not picked
validateSession absent from the index marked NOT FOUND (PD-7), never silently dropped
Compound ask (refactor + endpoint + Dockerfile) three ordered specs, step 1 of 3
do not refactor anything else prohibition not counted as an intent
make it better ## Unresolved inputs section enumerated, exit code 1
Injection inside a fenced Relevant state block SE002 error, handoff blocked
DROP TABLE in a 100/100 A4 spec at tier T2 SE003, handoff withheld
Repository's own artifacts 0 errors, 0 warnings, exit 0

Security posture

Refinement sits between untrusted text and a code-mutating agent, so it is an injection surface. The deterministic engine shrinks that surface structurally: no model participates in the transformation, so pasted text cannot steer a rewrite — fenced blocks are held out of clause parsing and re-emitted as quoted evidence. The linter's controls remain for scoring and CI:

  • SE002 — prompt injection. Instruction-override phrasing is screened over the full text including fenced blocks. Error severity; blocks handoff.
  • SE003 — destructive operations. DROP/TRUNCATE, rm -rf, force-push, disabling auth. Warning — an A7-OPS task may legitimately name them — but the A7 pack forces a rollback path and forbids applying changes to live environments.
  • SE001 — secrets. Literal credentials are errors. Telemetry stores prompt hashes only, never prompt text.

The agent's remaining discretion is a single command invocation; its hard rules forbid rewriting, filling slots, or substituting output. The ask travels to the engine via stdin (quoted heredoc / single-quoted PowerShell here-string), never interpolated into the command line — an ask containing quotes, backticks, or $(...) is data on a pipe, not shell input. Adapter conformance to this rule is CI-tested. The residual risk is a model that disobeys transport rules, which is why the acceptance criterion is byte-identity with engine stdout.

Two things worth knowing before you roll this out

The engine cannot judge whether stated criteria are the right criteria. It compensates by refusing to invent: anything not stated and not discoverable on disk becomes a visible [NEEDS INPUT:] slot. An output with unresolved slots is not a finished prompt; it is a prompt whose remaining defects are enumerated and impossible to miss. Exit code 1 exists so you can gate execution on it.

Measure K2 and K3, not K1. Median prompt score (K1) rises the moment you deploy this and is easy to game. Rework rate (K2) and diff precision (K3) are what justify the tooling existing. Baseline both for two weeks before enabling anything — spec §10 Phase 0 exists for this reason.

Releases

Packages

Contributors

Languages