Skip to content

About

Checks that the commands in your CI workflows actually exist — catches renamed scripts and phantom targets before CI does.

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

CI Command Truth — prove every run: command in your GitHub workflows exists

Tests

Prove every run: command in your GitHub workflows actually exists — before CI finds out it doesn't.

CI Command Truth terminal demo

A workflow that says npm run test:unit when package.json has no test:unit script, make deploy when the Makefile has no deploy target, or ./scripts/release.sh when that file was renamed fails only after you push, wait for a runner, and read the log. CI Command Truth checks those references statically, in milliseconds, with zero dependencies.

Install in one step: pip install git+https://github.com/RakTikoo/ci-command-truth, then run ci-command-truth . in the repo to check. Full options are under Get it.

First contribution? Start with a good first issue.

Demo

A two-file demo repo: package.json defines the scripts test and test:units — but the workflow asks for test:unit (no trailing "s"):

# .github/workflows/ci.yml — the last step
      - run: npm run test:unit
$ python3 ci_command_truth.py .
CI Command Truth — root: .
Workflows scanned: 1
  scanned: .github/workflows/ci.yml
Segments checked: 1
Segments skipped: 0
Findings: 1
  .github/workflows/ci.yml:8: `npm run test:unit` — missing-npm-script: expected script 'test:unit' (looked in: package.json) — did you mean 'test:units'?
FAIL — missing commands/targets/files found.
$ echo $?
1

Real captured output from the shipped v0.2 tool (only the root path is shortened). The phantom command is caught in milliseconds — locally, or as a PR annotation with --github-annotations — instead of after a push, a runner, and a log dive.

What it resolves

Workflow says Checked against
npm run X, npm test/start/..., pnpm run X, pnpm X, yarn X, bun run X scripts in package.json (repo root, or the step's working-directory)
npm run X --workspace=<path|name> / -w, pnpm --filter …, yarn workspace <name> … the workspace member's package.json when the member is identifiable; otherwise the segment is skipped with reason workspace
make X, make -C dir X, make -f file X, bare make targets in the Makefile / makefile / GNUmakefile, following include files; a target matching a pattern/suffix rule, or existing as a file/dir on disk, counts as present
just X, just -f file X, bare just recipes in justfile / Justfile / .justfile, following mod / import files; recipes with parameter defaults (build target='release':), @quiet recipes and alias names all count
task X, task ns:X, task -t file X, task --list, bare task tasks in Taskfile.yml / Taskfile.yaml / taskfile.yml (and the other names Task itself searches, incl. *.dist.* and lowercase variants), following includes: namespaces and flatten: includes; task aliases: names and * wildcard task names count, internal: tasks do not (Task refuses them from the command line); bare task needs a default task, --list only needs the file
./scripts/x.sh, bash x.sh, sh ./x.sh, python x.py, python3 ./x.py, node x.js the file existing on disk (per-interpreter flag tables: bash -e is a mode flag, only -c is inline code; python -W ignore -m pkg finds the module)
python -m pkg.mod the full dotted path, when the top-level package is repo-local (pkg/, pkg.py, or under src/): every submodule part must resolve (pkg/mod.py, pkg/mod/ regular or namespace package — namespace portions merged across the root and src/ trees, PEP 420 style — or a compiled module with a genuine platform extension suffix), and a package used as the final target must contain __main__.py. Conservative otherwise: repo-local existence is checked first (a local regular package even shadows a stdlib name); stdlib modules (http.server, unittest, …) and known installed tools (build, pytest, coverage, …) are assumed installed; if the workflow itself installs Python packages (pip install, poetry install, …), unknown modules are skipped as assumed installed; a package whose __init__.py really defines/imports __getattr__ or uses sys.modules can invent submodules at runtime, so its submodules are skipped as dynamic-module
pip install -r req.txt / -c constraints.txt the file existing on disk

It also unwraps runner prefixes (uv run make lint, poetry run …, bundle exec …), follows a leading cd dir inside a run block (subshell-scoped cd does not leak out; cd -- dir and cd - work), splits && / || / ; / | chains and checks each segment, strips env-var prefixes (FOO=bar cmd), sudo, and env -u/-C flags, skips heredoc bodies and $(...) / backtick / <(...) substitution contents, and strips trailing # comments (quote-aware). Windows backslash paths (.\scripts\run.sh, python tools\check.py) are normalised before checking. Files a workflow visibly creates earlier in the same run (> gen.sh, cp, touch, curl -o, git clone, a second actions/checkout with path:) are not flagged when later steps use them, and ./node_modules/.bin/* counts as an installed artifact.

What it deliberately does NOT flag

False positives are the bug this tool refuses to have:

  • shell builtins (echo, cd, export, [, source, …)
  • external/system tools (docker, curl, git, jq, sed, …) — plus a general rule: a bare command with no repo-local resolution target is never checked
  • any segment containing a ${{ }} GitHub expression or a $VAR that makes resolution impossible — those are counted as skipped, with a reason, never guessed at
  • steps whose working-directory: contains an expression — all path checks for that step are skipped, not guessed
  • steps (or whole jobs) with a literal if: false / if: ${{ false }}
  • run: bodies for non-shell shell: values (python, pwsh, powershell, cmd) — the body is not shell, so it is skipped with reason non-shell run body
  • segments whose failure is tolerated — joined by ||, or in a step with continue-on-error: true — are reported as warnings, not findings; they never affect the exit code
  • a resolver file (package.json / Makefile / justfile / Taskfile) that exists but is too large or unreadable makes its checks skipped (reason resolver-file-unknown, plus a warning), never treated as empty. The same applies to a Taskfile whose tasks cannot be enumerated statically (v2 layout, flow-style or anchor-built tasks: mappings). A malformed package.json is stronger: an error state (exit 2), listed explicitly like an unscanned file

Get it

One file, Python standard library only. Install it as a command:

pip install git+https://github.com/RakTikoo/ci-command-truth
ci-command-truth .          # scans the current repo

The same command via pipx or uv:

pipx install git+https://github.com/RakTikoo/ci-command-truth
uv tool install git+https://github.com/RakTikoo/ci-command-truth

Or just download ci_command_truth.py from this repo and run it with python3 — no install step. In CI, the GitHub Action below needs neither.

Usage

$ python3 ci_command_truth.py [ROOT]          # ROOT defaults to .
$ python3 ci_command_truth.py . --json        # machine-readable report
$ python3 ci_command_truth.py . --sarif       # SARIF 2.1.0 report (for GitHub code-scanning upload)
$ python3 ci_command_truth.py . --github-annotations   # ::error file=…,line=… lines for PR annotations

--json and --sarif are mutually exclusive, and --github-annotations pairs with the default text output only (appending annotation lines to a JSON/SARIF document would corrupt it, so the combination is rejected with exit 2). In SARIF output, findings are results (error findings → error, warning-severity findings → warning) located at the workflow file and line; files that could not be scanned are not results — they appear as error-level tool-execution notifications, with the unscanned count in the run properties, mirroring exit code 2. Artifact locations are percent-encoded repo-relative URIs; no absolute paths are emitted.

When a referenced script, target, recipe or module is missing and a close match exists, every output adds a did-you-mean suggestion (e.g. did you mean 'test:units'?) computed with stdlib difflib — deterministic, at most one suggestion, judged on the differing name segment only. A suggestion is advisory: it never changes a verdict, severity or exit code, and unrelated names get no suggestion.

Every report states: workflows scanned, segments checked, segments skipped (by reason), every finding as file:line, the offending segment, what was expected and where it was looked for — and every workflow file that could not be scanned (empty, tab-indented, oversized), listed explicitly. Nothing ever passes silently.

Exit codes: 0 everything checked resolved · 1 something missing · 2 usage/parse error (including any unscanned workflow file).

Use as a library

The module imports cleanly (import ci_command_truth), and scan() runs the whole check and returns the same result dict the CLI renders:

import ci_command_truth as cct

result = cct.scan("path/to/repo")
print(result["checked"], "segments checked, exit", result["exit"])
for f in result["findings"]:
    print(f["file"], f["line"], f["message"])

On a repo whose workflow calls npm run test:unit while package.json defines only test:units, that prints:

1 segments checked, exit 1
.github/workflows/ci.yml 8 missing-npm-script: expected script 'test:unit' (looked in: package.json) — did you mean 'test:units'?

render_text(result) produces the CLI's text report; the Checker class is there if you need step-by-step control.

Use it in GitHub Actions (2 lines)

      - uses: actions/checkout@v4
      - uses: RakTikoo/ci-command-truth@v0.4.0

The Action runs the checker with --github-annotations on (its default), so findings land as PR annotations; the job fails on any finding. Inputs (path, github-annotations) and a vendored-script variant are in examples/example-workflow.yml.

Use it as a pre-commit hook

# .pre-commit-config.yaml
- repo: https://github.com/RakTikoo/ci-command-truth
  rev: v0.4.0
  hooks:
    - id: ci-command-truth

The hook runs the checker over the repo whenever a workflow file is being committed, so a phantom command is caught before the push.

Use with an AI agent

This repo ships a SKILL.md written to the Agent Skills open standard. If your coding agent supports skills, point it at this repo and it can install CI Command Truth, run it on your workflows, and read the verdicts for you when you ask it to check your CI. The skill works as promised when the repo is cloned under its real name (ci-command-truth) — the standard ties the skill's name to its folder.

NOT in v1

  • No uses: action validation (that's actionlint's job — and it does it well)
  • No GitLab CI / Jenkins / other CI systems
  • No ${{ }} expression evaluation, no matrix expansion
  • No shell grammar beyond the documented subset (see ANALYSIS.md)
  • No execution of anything, ever: static parsing only

The YAML parser, honestly

PyYAML is not in the Python standard library, and this tool ships with zero dependencies, so workflow extraction uses a small indentation-based parser written for this project. It supports single-line run: values, | / > block scalars (including indent indicators like |2 and header comments; folded > scalars are joined into logical lines), plain multi-line scalars, anchors/tags on run: (stripped and parsed), - run: inline steps and bare - steps, and step-level plus defaults working-directory: — with defaults scoped per job (a job's defaults: overrides the workflow defaults for that job only). Nested run: keys under with: / env: and the bodies of non-run block scalars are never mistaken for steps, and only dashes inside a job-level steps: list create steps (strategy.matrix.include items do not). It does not support flow style or multi-document files; a file that mentions run: but yields no extractable steps is reported as unscanned (exit 2), never silently passed — as are empty workflow files, binary files, symlink loops, and an empty .github/workflows/ directory. See ANALYSIS.md for the full limitation list.

Safety

Static only: the tool never executes a scanned repository's code, scripts, or workflow commands (a test asserts the source contains no subprocess/eval/exec/os.system). File reads are size-capped (1 MB per file). No network, no LLM, deterministic output.

Tests

$ python3 run_tests.py     # 83 tests, zero dependencies
$ python3 spotcheck_fp.py  # 16 false-positive headline cases, fresh sandboxes

Fixture corpus in fixtures/: 35 clean repos must produce zero findings, 11 repos exactly one planted missing reference each, 8 edge-case repos (no-run, uses-only, empty, malformed, CRLF, nested workflow subdirectory, empty workflows dir, malformed package.json), and 1 downgrade repo (warning-severity findings, exit 0). A further 3 cargo repos (2 clean, 1 planted-missing) and 4 Go repos (2 clean, 2 planted-missing) sit alongside them with documented expectations in their EXPECTED.md files; they join the registered corpus when a resolver for their tool lands, since cargo and Go segments are currently skipped as external tools. See ANALYSIS.md for the score, including the v0.2 section.

Python tooling fixtures cover a Makefile target wrapping pytest, a repository-local script invoked from CI, and a missing script path. See ANALYSIS.md for the fixture score and coverage details.

License

MIT — see LICENSE.


More from this workshop: SkillScan · KiCad Preflight · Expiry Radar · PhantomDep PR Gate · Stranger Suite · Living Contract Diff · Handoff Reconciler

About

Checks that the commands in your CI workflows actually exist — catches renamed scripts and phantom targets before CI does.

Topics

Resources

Contributing

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages