fix: fusa-asil-b CI gate — pin cpp-FuSa v0.17.1, stop wiping requirements, enforce real gap thresholds - #30
Open
SoundMatt wants to merge 1 commit into
Open
fix: fusa-asil-b CI gate — pin cpp-FuSa v0.17.1, stop wiping requirements, enforce real gap thresholds#30SoundMatt wants to merge 1 commit into
SoundMatt wants to merge 1 commit into
Conversation
…ents, enforce real gap thresholds
Three compounding problems in the fusa-asil-b (and sarif) CI jobs:
1. cpp-FuSa was pinned to v0.15.0, missing conformance fixes through
v0.17.1 that affect this repo's own generated evidence (sbom.json
component hashes, safety-case completeness — see cpp-FuSa's
CHANGELOG v0.17.0 entry). Both jobs now pin v0.17.1.
2. `cpfusa init --force` ran as the first step in both jobs, before the
traceability step, unconditionally overwriting the real, populated
`.fusa-reqs.json` (50+ KB of hand-maintained REQ-* entries) with an
empty scaffold. CI traceability coverage was always computed against
zero requirements regardless of the repo's actual state. Removed the
step entirely — `.fusa.json`/`.fusa-reqs.json` are already checked
in and match what `init` would have generated; the step is only
needed for first-time local setup.
3. The ISO 26262 / IEC 61508 gap-analysis steps were wrapped in
`|| true`, so they could never fail the build regardless of gap
count (15/20 and 12/18 gaps at last real measurement). Two changes:
- Reordered the job so boundary/tara/fmea/safety-case/sas/sci run
BEFORE the gap analyses: cpfusa grades several objectives (e.g.
§9-2.1 Safety case, §10.4 SCI) on whether safety-case.json/
sci.json already exist, so gap-checking first was undercounting
what the repo actually addresses. This alone closed 2 gaps.
- Replaced `|| true` with an explicit, intentional gate: fail the
build if the gap count regresses past the current, documented
baseline (13 for ISO 26262, 11 for IEC 61508). A "0 gaps" target
is not reachable with cpfusa v0.17.1 regardless of this repo's
actual documentation completeness — several objectives (e.g. §6.1
Software architectural design, §8-6.2 Safety manual) have no
evidence-detection logic implemented in the tool at all and report
Gap unconditionally (cpp-FuSa src/iso26262/iso26262.cpp's
detect_status()). The gate at least makes future regressions
visible instead of being structurally unable to fail.
Also fixed independently while investigating (2), a real bug the fix
exposed: cpp-FuSa v0.17.0 started actually enforcing `sourceDirs`
(previously ignored per its own CHANGELOG), and `.fusa.json`'s
sourceDirs (["src", "include"]) omitted "tests" and "cli" — so every
`fusa:test`/`fusa:req` annotation in those directories silently stopped
being seen by `trace`, dropping real test-coverage traceability from
93.5% to 0% the moment the pin bumped. Added both directories to
sourceDirs, and registered 6 real, already-annotated CLI requirements
(REQ-CLI-001..006, both impl- and test-annotated in cli/ and
tests/test_cli.cpp already) that this surfaced as dangling references
in requirements/requirements.json and .fusa-reqs.json.
README's ASIL-B badge/description is reworded to "ASIL-B target
(SEooC)" with a pointer to the real gap-report artifacts, per this
repo's own SAFETY_PLAN.md framing — the prior flat "ISO 26262 ASIL-B"
claim overstated completeness relative to the tool's own (now
enforced) gap reports.
Verified locally end-to-end against a clean checkout with a freshly
built cpfusa v0.17.1: check/lint/trace/cyber/qualify/boundary/tara/
fmea/safety-case/sas/sci/badge/vuln/metrics/report all exit 0; trace
reports 135/144 (93.8%) annotated and tested with zero dangling
references; iso26262 gate passes at 13 gaps (baseline); iec61508 gate
passes at 10 gaps (under the 11 baseline).
Closes #19, #20, #21
Signed-off-by: Matt <47545907+SoundMatt@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Three compounding problems in the
fusa-asil-b(andsarif) CI jobs:v0.17.1 that affect this repo's own generated evidence (
sbom.jsoncomponent hashes,
safety-casecompleteness).cpfusa init --forceran first, before the traceability step,unconditionally overwriting the real, populated
.fusa-reqs.json(50+ KB of hand-maintained
REQ-*entries) with an empty scaffold —CI traceability coverage was always computed against zero
requirements.
|| true—could never fail the build regardless of gap count (15/20 and 12/18
gaps at last real measurement), while
README.mddisplayed a flat"ASIL-B" badge alongside.
Fix
cpp-FuSatov0.17.1(current latest) in both jobs.cpfusa initstep entirely from both jobs —.fusa.json/.fusa-reqs.jsonare already checked in and match whatinitwould generate; the step is only needed for first-time localsetup.
boundary/tara/fmea/safety-case/sas/scirun before the gap analyses (cpfusa grades some objectiveson whether those artifacts already exist — this alone closed 2
gaps), and replaced
|| truewith an explicit gate: fail if the gapcount regresses past a documented baseline (13 ISO 26262, 11 IEC
61508). A "0 gaps" target isn't reachable with cpfusa v0.17.1 at
all — several objectives have no evidence-detection logic in the
tool and report Gap unconditionally — so the gate enforces
non-regression rather than an unreachable absolute.
Found while investigating (2)
cpp-FuSa v0.17.0 started actually enforcing
sourceDirs(previouslysilently ignored per its own CHANGELOG).
.fusa.json'ssourceDirsomitted
testsandcli, so everyfusa:test/fusa:reqannotationin those directories stopped being seen the moment the pin bumped —
test-coverage traceability would have silently dropped from 93.5% to
0%. Added both directories to
sourceDirs, and registered 6 real,already-annotated CLI requirements (
REQ-CLI-001..006) that thissurfaced as dangling references.
README
Reworded the ASIL-B badge/description to "ASIL-B target (SEooC)" with
a pointer to the real gap-report artifacts, matching this repo's own
SAFETY_PLAN.mdframing — the prior flat "ISO 26262 ASIL-B" claimoverstated completeness relative to the tool's own (now enforced) gap
reports.
Testing
Verified locally end-to-end against a clean checkout with a freshly
built
cpfusav0.17.1 (isolated clone, not the CI-cached one):check/lint/trace/cyber/qualify/boundary/tara/fmea/safety-case/sas/sci/badge/vuln/metrics/reportall exit 0.tracereports 135/144 (93.8%) annotated and tested with zerodangling references.
iso26262gate passes at 13 gaps (at baseline).iec61508gate passes at 10 gaps (under the 11 baseline). Also ran afull local build +
ctest(171/171 passing) to confirm the non-CIchanges (
.fusa.json, requirements JSON, README) don't affect thelibrary build.
Closes #19, #20, #21