Skip to content

[SG-320] feat(local): use tirith -policy-path as a CI gate — a directory of policies, masked input, and the verdict written out #283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
refeed wants to merge 2 commits into from
Closed

[SG-320] feat(local): use tirith -policy-path as a CI gate — a directory of policies, masked input, and the verdict written out #283

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1bc7275
[SG-320] feat(local): evaluate committed policy files with `tirith lo…
refeed Aug 18, 2026
5531d86
[SG-320] refactor(local): put CI reporting on `tirith -policy-path` i…
refeed Aug 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 37 additions & 3 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0




## [Unreleased]

## [1.3.0] - 2026-08-18

### Added
- `tirith -policy-path …` can be used as a CI gate, not only at a terminal. Three additions, all
optional and all off unless asked for, so an existing invocation behaves exactly as before -- which
matters because this command's `--json` output is a frozen contract:
- **`-policy-path` accepts a directory or a glob**, evaluating every policy it finds. A directory is
searched recursively for `*.tirith.json`, or failing that for any `.json` file shaped like a policy.
Pointing it at a directory previously failed with a bare `ERROR`. `-var-path` and `-var` apply to
every policy, so a parameterised policy behaves the same whether you name the file or its directory.
- **`--input-kind`** masks the document before evaluation. It matters even though nothing is uploaded:
evaluator messages embed the values they compared, and those messages are copied into whatever
comment a CI job posts. Opt-in because masking changes those messages, and they are the frozen
`--json` output.
- **`--output-json`, `--output-markdown`, `--comment-marker`, `--markdown-limit`, `--sha`** write the
verdict out for a job to publish, in the same shape `tirith platform check` writes. So one CI
integration drives either, and adding a StackGuardian organization later changes which policies
apply rather than how anything is wired.

Two paths fail closed regardless of `--fail-on-error`, because both are tool health rather than policy
decisions: no policy files found at `-policy-path`, and a policy that could not be evaluated. A run
where every check was skipped also exits 1 -- nothing was examined, and this command has always
treated that as a failure rather than a pass. That last one is a deliberate difference from
`platform check`, which counts skips separately and reports them as a pass.

This capability came from the GitHub Action, which owned the only implementation of it. It moved here
so a second front end -- the GitLab CI component -- drives one implementation rather than forking it.
- `tirith ui`: an interactive interface with three tabs.
- **Explorer** — read an evaluation's results down to the resource behind each one. The result
document has always carried the resource address, the planned action and the before/after
Expand All @@ -31,9 +56,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
a provider argument the operation does not read, an id referenced in `eval_expression` but
never defined, a single `&` where `&&` was meant, an evaluator that does not exist.

### Changed
- The `--output-json` document now carries a `mode` key (`platform` or `local`), and every key exists
in both modes -- one with no meaning on a path is `null` rather than omitted, so a consumer needs
one parser rather than two. Additive: no key was removed or retyped. Specified in
[docs/output-contract.md](docs/output-contract.md).
- `tirith platform check` writes `--output-json` and `--output-markdown` on its failure paths too,
rather than returning having written nothing. A caller editing a sticky comment in place needs a
marker-first body on that path, or it orphans the comment it was updating.

### Notes
- The local evaluation surface is untouched. `ui` is dispatched before the flat parser, like
`platform`, so `--json` output remains byte-identical to the golden file.
- The local evaluation surface is untouched. `ui` and `local` are dispatched before the flat parser,
like `platform`, so `--json` output remains byte-identical to the golden file.
- No new runtime dependencies for anyone who does not install the extra.

## [1.2.0] - 2026-08-03
Expand Down
77 changes: 64 additions & 13 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ verdict, same exit codes. That mode is optional and is the only part that talks
- [Run it in CI](#run-it-in-ci)
- [Exit codes](#exit-codes)
- [Evaluating against your StackGuardian organization](#evaluating-against-your-stackguardian-organization)
- [Evaluating committed policy files, in CI](#evaluating-committed-policy-files-in-ci)
- [Example Tirith policies](#example-tirith-policies)
- [error_tolerance](#error_tolerance-and-the-third-outcome)
- [Terraform Plan](#terraform-plan-provider)
Expand Down Expand Up @@ -177,29 +178,42 @@ pip install -e .

```
tirith --version
tirith 1.2.0
tirith 1.3.0
```

Congratulations! Tirith has been setup in your system

## Usage

```
usage: tirith [-h] [-policy-path PATH] [-input-path PATH] [-var-path PATH]
[-var PATH] [--json] [--verbose] [--fail-on-error] [--version]
usage: tirith [-h] [-policy-path PATH] [-input-path PATH] [-var-path PATH] [-var PATH] [--json]
[--verbose] [--fail-on-error] [--version] [--input-kind KIND] [--state-path PATH]
[--sha SHA] [--output-json PATH] [--output-markdown PATH] [--comment-marker TEXT]
[--markdown-limit N]

Tirith (StackGuardian Policy Framework)

options:
-h, --help show this help message and exit
-policy-path PATH Path containing Tirith policy as code
-input-path PATH Input file path
-var-path PATH Variable file path(s)
-var PATH Inline variable(s)
--json Only print the result in JSON form (useful for passing output to other programs)
--verbose Show detailed logs of from the run
--fail-on-error Exit 3 when a policy fails, instead of 0. Off by default for compatibility.
--version show program's version number and exit
-h, --help show this help message and exit
-policy-path PATH Path containing Tirith policy as code
-input-path PATH Input file path
-var-path PATH Variable file path(s)
-var PATH Inline variable(s)
--json Only print the result in JSON form (useful for passing output to other programs)
--verbose Show detailed logs of from the run
--fail-on-error Exit 3 when a policy fails, instead of 0. Off by default for compatibility.
--version show program's version number and exit

reporting:
Write the verdict out for a CI job to publish. All optional.

--input-kind KIND terraform_plan, terraform_state, kubernetes or json. Masks the input before evaluating
--state-path PATH Terraform state, when --input-kind is terraform_state
--sha SHA Revision these findings describe, recorded in the report
--output-json PATH Write the result document here (same shape as `tirith platform check`)
--output-markdown PATH Write a markdown report here, for a comment or a note
--comment-marker TEXT Opaque first line of the markdown, for comment stickiness
--markdown-limit N Truncate the markdown to this length. Default: 60000

Subcommands:

Expand All @@ -210,6 +224,7 @@ Subcommands:

About Tirith:


* Abstract away the implementation complexity of policy engine underneath.
* Simplify creation of declarative policies that are easy to read and interpret.
* Provide a standard framework for scanning various configurations with granularity.
Expand Down Expand Up @@ -385,7 +400,7 @@ GitLab-specific: any runner that can execute a container and produce a plan work
| 0 | Policies passed, or nothing was in scope to gate on |
| 1 | Tirith could not complete the evaluation — bad input, a policy it could not evaluate, unreachable API |
| 2 | Timed out waiting for a StackGuardian run |
| 3 | A policy failed. Only with `--fail-on-error`, on either surface |
| 3 | A policy failed. Only with `--fail-on-error`, on any of the three surfaces |
| 130 | Interrupted |

**Gate a CI job with `--fail-on-error`:**
Expand Down Expand Up @@ -443,6 +458,42 @@ Common flags:
Running this from GitHub Actions? Use [the action](#github-actions) instead — it wires up the plan
discovery, the sticky pull-request comment, the check run and the exit codes for you.

## Evaluating committed policy files, in CI

`tirith -policy-path … -input-path …` has always evaluated the policies in your repository. Three
additions make it usable as a CI gate rather than only at a terminal:

```
tirith -policy-path .tirith/policies -input-path plan.json \
--input-kind terraform_plan --output-json result.json --output-markdown report.md \
--fail-on-error
```

**`-policy-path` accepts a directory or a glob**, not just one file, and evaluates every policy it
finds — a directory is searched for `*.tirith.json`, or failing that for any `.json` file shaped like a
policy. Pointing it at a directory previously failed with a bare `ERROR`.

**`--input-kind` masks the document** before evaluation. That matters even though nothing is uploaded:
evaluator messages embed the values they compared, and those messages end up in whatever comment your
CI job posts. It is opt-in because masking changes those messages, and they are this command's `--json`
output, which is a frozen contract — with no `--input-kind` the file is read exactly as before.

**`--output-json` / `--output-markdown` / `--comment-marker`** write the verdict out for a job to
publish, in the [same shape](docs/output-contract.md) `tirith platform check` writes. So a CI
integration works against either, and adding a StackGuardian organization later changes which policies
apply rather than how anything is wired.

Two behaviours worth knowing, both of which fail closed:

- **No policy files found is an error, not a skip** — exit `1`, whether or not `--fail-on-error` was
passed. A green result for a change nothing was evaluated against is the one outcome this must never
produce.
- **A policy that could not be evaluated exits `1` regardless of `--fail-on-error`**, as does a run
where every check was skipped. "Could not evaluate" and "nothing was examined" are tool health, not
policy decisions.

Full reference: [docs/evaluating-policy-files.md](docs/evaluating-policy-files.md).

## Example Tirith policies

[Examples using various providers](tests/providers)
Expand Down
122 changes: 122 additions & 0 deletions docs/evaluating-policy-files.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
# Evaluating policy files in your repository

`tirith -policy-path … -input-path …` evaluates the policies committed in your repository. It needs no
account and makes no network calls, and it is the surface most open-source users are on.

```
tirith -policy-path .tirith/policies -input-path plan.json --fail-on-error
```

This page is about the flags that make it usable as a **CI gate**: several policies at once, the input
masked before its values reach a pull-request comment, and the verdict written out for a job to
publish. The alternative is
[`tirith platform check`](platform-check.md), which evaluates the policies a StackGuardian organization
enforces instead — and writes the [same result document](output-contract.md), so a CI integration built
against one works unchanged against the other.

## Several policies at once

`-policy-path` accepts a file, a **directory** or a **glob**.

- **A file** is evaluated as given. Naming one explicitly is an instruction, so "that is not a policy"
is reported rather than the file being skipped.
- **A directory** is searched recursively for `*.tirith.json`. If it holds none, any `.json` file
*shaped* like a policy is used — an object with both `meta` and `evaluators`.
- **A glob** is expanded and filtered by that same shape test.

The shape test is load-bearing rather than defensive: a policy directory routinely also holds the
document under evaluation. Without it, `plan.json` is evaluated *as a policy*, which reports a spurious
failure and buries the real findings.

`-var-path` and `-var` apply to every policy evaluated, so a parameterised policy behaves the same
whether you name the file or the directory holding it.

**No policy files found is an error, not a skip.** Pointed at a path with nothing in it, the command
exits `1`. A green result for a change nothing was evaluated against is the outcome this must never
produce, and "no policies found" is a configuration mistake rather than a deliberate skip.

## Masking, and why it is opt-in

`--input-kind` says what the document is, and supplying it **masks** the document before evaluation:
values terraform marked sensitive are replaced, root `variables` are dropped, `prior_state` is removed.

It matters even though nothing is uploaded. Evaluator messages embed the actual attribute values they
compared, and those messages are copied verbatim into whatever comment or note your CI job posts — so
an unmasked run publishes plan values to a code host. Masking also keeps this verdict identical to
`platform check`'s for the same plan, because that path evaluates the masked document too.

It is opt-in because masking changes the evaluator messages, and those messages are this command's
`--json` output, which is a frozen contract. With no `--input-kind` the file is read exactly as it
always has been.

`terraform_plan` and `terraform_state` are masked. `kubernetes` and `json` are passed through
untouched: they carry no sensitivity markers to mask by, and tirith reads YAML for them, which a JSON
round-trip would break.

## Writing the verdict out

| | |
|---|---|
| `--output-json PATH` | The result document, in the [same shape](output-contract.md) `platform check` writes |
| `--output-markdown PATH` | A rendered report, for a pull-request comment or a merge-request note |
| `--comment-marker TEXT` | An opaque first line for the markdown, so a job can find and edit its own comment |
| `--markdown-limit N` | Truncate the markdown. Default 60000 |
| `--sha SHA` | The revision the findings describe, shown in the report |

Both files are written on **every** exit path, including failures, and the markdown always begins with
`--comment-marker` when one is given. That matters to any caller that edits a sticky comment in place:
writing nothing on the failure path means the caller falls through to a body of its own with no marker
in it, and PATCHing that over a good comment orphans it permanently. It has happened.

Nothing in this group changes what the command prints.

## Exit codes

| Code | Condition |
|---|---|
| 0 | Policies passed or warned. Also a failing policy without `--fail-on-error` |
| 1 | No policy files found; the input was missing or unparseable; a policy could not be evaluated; nothing was evaluated |
| 3 | A policy failed, with `--fail-on-error` |
| 130 | Interrupted |

**A policy that could not be evaluated exits `1` regardless of `--fail-on-error`.** "Could not
evaluate" is tool health, not a policy decision. Such a policy also appears in the report as a visible
failure carrying its reason, so it is never mistakable for a pass and never silently dropped.

**A run of nothing but skips exits `1`.** If every check was swallowed by its `error_tolerance` —
usually because the resources the policies name are not in the document — then nothing was examined,
and this command has always called that a failure rather than a pass. Note this is a deliberate
difference from `platform check`, which counts skips separately and reports them as a pass.

## Flags

The full surface, as the command prints it:

usage: tirith [-h] [-policy-path PATH] [-input-path PATH] [-var-path PATH] [-var PATH] [--json]
[--verbose] [--fail-on-error] [--version] [--input-kind KIND] [--state-path PATH]
[--sha SHA] [--output-json PATH] [--output-markdown PATH] [--comment-marker TEXT]
[--markdown-limit N]

Tirith (StackGuardian Policy Framework)

options:
-h, --help show this help message and exit
-policy-path PATH Path containing Tirith policy as code
-input-path PATH Input file path
-var-path PATH Variable file path(s)
-var PATH Inline variable(s)
--json Only print the result in JSON form (useful for passing output to other programs)
--verbose Show detailed logs of from the run
--fail-on-error Exit 3 when a policy fails, instead of 0. Off by default for compatibility.
--version show program's version number and exit

reporting:
Write the verdict out for a CI job to publish. All optional.

--input-kind KIND terraform_plan, terraform_state, kubernetes or json. Masks the input before evaluating
--state-path PATH Terraform state, when --input-kind is terraform_state
--sha SHA Revision these findings describe, recorded in the report
--output-json PATH Write the result document here (same shape as `tirith platform check`)
--output-markdown PATH Write a markdown report here, for a comment or a note
--comment-marker TEXT Opaque first line of the markdown, for comment stickiness
--markdown-limit N Truncate the markdown to this length. Default: 60000
Loading
Loading