Say publicly why a finding that is not fixed does not apply (#89)

[iderex]

The issue this belongs to

Closes #89

What changed

security/statements.json says publicly, per finding, why something a scanner
raised here is not going to be fixed. It carries two kinds of entry, because there
are two ways a finding stops reaching the gate and they do not have the same
shape.

A findings entry is a finding a scanner raised against this repository and
somebody dismissed. It has an identifier, so it carries a per-finding disposition:
status, justification, impact, and the condition that retires it.

An exclusions entry is a class taken out of what a scanner is pointed at. It has
no identifier and no list of what would have been found, so it carries a scope,
and it carries in a field of its own what is consequently NOT being looked for.
That last field is the one a consumer needs and the one an author will otherwise
leave out - "the pedantic persona is not run" says less than the same sentence
ending "so no finding from that persona will appear in this document or in the
code-scanning tab at all".

.github/statements/statements.sh and .github/workflows/statements.yml are the
check. The comment at the head of .github/workflows/scorecard.yml said the
reasoning for a dismissal lived only in the dismissal; that stops being true with
this change, so it says what is true now and keeps the bound rather than dropping
it.

What failure it prevents

The cheaper path being the quieter one. Dismissing a finding costs a written
statement; narrowing the scanner so the finding never fires costs nothing, and the
configuration change looks like tuning. A rule whose honest route is more
expensive than its evasion selects for the evasion without anybody deciding to
evade. That is why the check reaches an exclusion and not only a dismissal, which
is the scope question this issue's comments raised three times.

It has already happened here in the weaker form. Four exclusions stand in this
tree today with their reasoning in a comment beside a pattern or a line in a
register, and four dismissed alerts stand on the code-scanning surface with their
reasoning in the dismissal itself, where a reader who cannot list the alerts
cannot reach it. Three of those four dismissals point at a repository a reader of
this board cannot open.

One of the four is worse than the others and the document is what fixes it. The
fuzzing check is dismissed permanently on the ground that there is no untrusted
input surface here for a fuzzer to reach. That is true of the tree today because
no parser is in it, and 0101, 0055 and open issue #86 say the opposite about the
surface that is coming. A dismissal comment carries no condition and its state does
not expire; a retires_when field does, and that entry now carries one.

Evidence

The commit these were run at:

git rev-parse HEAD
29391281378875f61633f1fbf3f853038d237684

The rule set and the fixtures behind it:

bash .github/statements/statements.sh selftest | awk '/^ok    /{t++} /^ok    bites/{b++} /^ok    passes/{p++} /^ok    reads|^ok    does not read/{r++} END{print "total="t" bites="b" passes="p" reads="r}'
total=21 bites=12 passes=5 reads=4

Twelve violations the rules must refuse, five near misses they must not, and four
readings of what a register line is. Every one runs before the gate judges the
document, so a rule cannot pass its fixture and refuse something else.

The tree it judges:

bash .github/statements/statements.sh check | grep -E '^Registers found|^Directives|^ok    every'
Registers found: .github/codeql/excluded-rules .github/lint/excluded-lints .github/shell-analysis/excluded-rules .github/thread-detector/suppressions
Directives in them: 2. Statements in the document: 4 finding(s), 4 exclusion(s).
ok    every directive these registers carry has a statement, and every statement names a directive that exists

Two directives, both in .github/shell-analysis/excluded-rules; the other three
registers are empty of entries today. Four exclusions rather than two, because the
two that are not directives - the persona .github/workflows/zizmor.yml does not
select and the codepoint .github/workflows/unicode-guard.yml does not list -
have entries with a null rule.

The four dismissed alerts the findings entries are written for, read live rather
than recalled:

gh api "repos/Flowfin/core/code-scanning/alerts?per_page=100&state=dismissed" --jq '.[] | "\(.number)\t\(.tool.name)\t\(.rule.id)\t\(.dismissed_reason)"'
9	Scorecard	CIIBestPracticesID	won't fix
8	Scorecard	CodeReviewID	won't fix
4	Scorecard	FuzzingID	won't fix
1	Scorecard	BranchProtectionID	won't fix

THAT READING NEEDS A PERMISSION A READER OF THIS BOARD DOES NOT HOLD, so it is a
reading made here rather than one a reader can reproduce from the command beside
it. The dismissal comment text is counted rather than pasted, for the reason the
document gives: three of the four send a reader outside this repository, and
pasting the reference would put a pointer of exactly the shape this issue exists
to replace into the thing that exists to replace it. The document's entries are
written in this repository's own words instead.

No dependency advisory exists to state. Every finding in the document is a policy
check from one scanner:

jq -r '.findings[].scanner' security/statements.json | sort -u
Scorecard

and the document a consumer reads is the one this change adds:

git ls-tree -r --name-only HEAD -- security/
security/statements.json

The rest of the gate, run locally at this commit:

cargo build --locked --all-targets
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.03s

cargo test --locked | grep -c '^test result: ok'
9

bash .github/doc-paths/doc-paths.sh check | tail -1
Every path these documents name resolves against the tracked set.

What a guard here refuses, and the proof it bites

The check refuses a document that does not parse before any rule reads it, an
entry missing a field or carrying one blank, a status outside the vocabulary the
document declares, one identifier used twice, a statement naming a configuration
that is not tracked, a directive a register excludes with no statement, and a
statement for a directive no register excludes any more.

Both conditions this issue's Done-when names as proofs were watched on the real
tree rather than only on a fixture.

A malformed edit. One comma added after "exclusions": [:

bash .github/statements/statements.sh check
      security/statements.json is not valid JSON, so nothing below could read it
::error::1 statement(s) do not hold. Each one is printed above.

A suppression with no statement. One line added to
.github/shell-analysis/excluded-rules:

echo 'SC2086 A reason nobody wrote a statement for.' >> .github/shell-analysis/excluded-rules
bash .github/statements/statements.sh check
      security/statements.json names no statement for `SC2086` in .github/shell-analysis/excluded-rules, which that configuration excludes
::error::1 statement(s) do not hold. Each one is printed above.

Both were then reverted and the run returns to the ok line quoted under
Evidence.

The five near misses are the other half of the proof and they are the mistakes
somebody actually makes: both arrays present and empty, which is a tree with
nothing to state; a null rule, which is an exclusion with no directive to name,
in both directions of the comparison; a register comment line, which is where
every one of these registers explains itself; and a blank line.

A blank field is refused as hard as an absent key, and that is a fixture of its
own. A field somebody emptied to get past a required-fields check is the silent
suppression this document exists to replace, wearing the shape of a statement.

The means, and why it fits

A JSON document read with jq, which is on the runner image, beside bash and awk
for the registers - the same means as doc-paths, invariants and
shell-analysis, and no language, runtime or dependency this tree does not
already carry.

JSON rather than this repository's own block-register shape, which is what
.github/fixture-scrub/values and .github/invariants/rules use. Those are read
by this gate and by nothing else; this document's audience is a downstream
consumer, and a bespoke format is not something a consumer acts on.

The document says of itself that it is NOT an OpenVEX document and does not claim
to be. No advisory against a dependency here has been raised, so there is nothing
for OpenVEX's vocabulary to describe and every entry is a policy check rather than
a vulnerability. status, justification and impact are named after OpenVEX's
fields so that projecting this document later is a mapping rather than a
translation. No such projection has been produced, and the shape a dependency
advisory takes here is decided when the first one arrives.

What this does not cover

Whether a statement is TRUE. Every field either carries text or it does not;
whether the reasoning is sound, and whether what an entry says is not looked for is
what is actually not looked for, is a judgement no reading of the tree makes. The
review is where a wrong one is caught.

An exclusion that is not a directive in a register. The persona a workflow does
not select and the codepoint a pattern does not list are stated in the document
and derived by nothing, so their entries carry a null rule and only their
configuration path is checked. An exclusion of that kind added tomorrow with no
statement is silent to every rule here. That is the larger residual and it is
printed on every run rather than only written down.

A finding dismissed on the code-scanning surface. A dismissal is a state on the
repository rather than a byte in the tree, no commit records one, and the listing
needs a permission a reader of this board does not hold. The four findings
entries are written by hand and nothing compares them against the live alerts, so
a fifth dismissal tomorrow adds no entry and reddens nothing.

The one open alert is not stated. MaintainedID is open rather than dismissed, so
it is a finding nobody has disposed of and this document is for the ones somebody
has. Nothing here dismisses it.

.github/workflows/zizmor.yml and .github/workflows/unicode-guard.yml are not
edited. Their comments still carry their reasoning, the document now carries it
too, and nothing keeps the two in step. Pointing each comment at the document
would be the next step and it is not taken here.

No run on GitHub had happened when this body was written. Every command above ran
on a Windows machine in Git Bash, and the gate runs on ubuntu-latest. One thing
was found that way and is worth naming: a jq built for Windows cannot open the
process substitution a --slurpfile is handed, and the failure is a message on
standard error with an empty verdict, which reads exactly like a run that found
nothing. Every input is a real file for that reason.

Who has read it

Nobody other than the author. There was no second reader available for it, and the
evidence above stands in place of one rather than the question being left open.