Repository navigation
feat(coverage): read lcov tracefiles, closing the Rust coverage gap - #35
Open
modusensus wants to merge 10 commits into
Open
modusensus wants to merge 10 commits into
modusensus wants to merge 10 commits into
Conversation
…s in coverage was the one measurement with a confirmed gap: history and deps both work on Rust, but the rustls round came back wholly notEvaluated because the producer knew c8/istanbul JSON, coverage.py JSON and Go text profiles only -- with Rust emitting lcov (cargo-llvm-cov, cargo-tarpaulin, any gcov/llvm-cov target). This adds the fourth format, identified by shape like the others. The shape was read off a real tracefile rather than assumed. A 12-function, 2-file probe crate was built with RUSTFLAGS=-C instrument-coverage, run under cargo test, merged with llvm-profdata merge -sparse, and exported with llvm-cov export --format=lcov; the fixtures here carry that run's mangled names, not invented strings. What differs from the Go path: lcov records carry function names, so no source tree is needed and --source is not required. The price is that the names are mangled and this machine has no demangler (llvm-cov --Xdemangler was tried with rust, rustfilt, llvm-cxxfilt and c++filt -- all four report "could not find the demangler"). Matching therefore runs on the mangled form: legacy mangling encodes a path segment as <decimal byte length><name>, so the query "nested::dup" matches 6nested3dup while a bare "dup" matches every same-named function in the file, each reported as its own fact. An already-mangled query (_R/_Z) is matched exactly and never half-decoded. Segment matching is anchored rather than a concatenated needle, because a mangling may insert a disambiguator between segments: a #[cfg(test)] module comes out as 5testss_14exercises_some. The refusals are the point, and each is pinned: - an FN: with no matching FNDA: is notEvaluated, never count 0. A missing count is not a zero, and manufacturing "never invoked" out of an absent line is the one direction this producer must not get wrong. - a file absent from the tracefile is notEvaluated, deliberately the opposite of the c8 path: llvm-cov and gcov emit records only for what was instrumented and --sources filters further, so absence proves nothing about execution. - a tracefile whose records hold no FN: line at all is notEvaluated -- line coverage cannot answer a symbol question, since which lines belong to a function is in the source, not the tracefile. - the newer FNL:/FNA:/FNAB: encodings are recognised and rejected rather than read by a guessed shape (their spelling comes from the specification; no sample could be produced here, and the code says so). - a body that does not parse throws the same way a malformed Go profile does, with the reason printed instead of a silent JSON.parse failure. Two records for one SF: (a split tracefile, or one file instrumented twice) are merged by summing counts -- the one merge that cannot turn a nonzero count into zero, and zero is the only conclusion this path draws. A query answered only by refusals reports measured: false, same rule as the Go path. Suite: 275 tests, 275 pass, 0 fail (pwsh; Git Bash reports 274 + 1 skip, the bench case keyed on tar flavour).
`coverage --help` still described `--coverage` as "c8's coverage-final.json", which stopped being the whole story once the producer learned coverage.py JSON and Go profiles, and is now one short of the four shapes it accepts. The help lists them the way the producer resolves them -- by shape -- and says that `--source` is needed only for the Go profile, since the other three carry function names (lcov does, if mangled).
…n now answer The producer gained a fourth coverage format, so §6.2 gains the measured record of what that format actually looks like -- read off a real llvm-cov tracefile rather than transcribed from the spec: SF:/end_of_record record boundaries, FN:/FNDA: as two separate runs rather than adjacent pairs, legacy mangling and the four demangler names that all fail here, several FN: entries on one line (generic monomorphisation), counts summed across records, absence reported as not evaluated (the opposite of c8), and an FN: with no FNDA: never defaulting to 0. The language table also changes: Rust moves from "partial" to function-level falsification working, because lcov's per-function invocation count is exactly the quantity this category draws on -- with the call-site level still partial. And inlining was measured not to flatten a called function to 0 in a release build, which is the one counterexample that would have broken the count === 0 => never invoked inference; the scope of that measurement is stated with it.
Section 7 recorded Rust coverage as a confirmed gap, and that record is what was true in the round it describes -- it is not rewritten. A follow-up note states that the producer now reads lcov tracefiles, that the closing was done with a probe crate rather than cargo-llvm-cov (still absent here), and that this does not change the round's own conclusion, which is about rustls under 0.6.0.
…ference Both fact-producers references gain the lcov row in the format table plus the production commands for a Rust target, measured on the probe crate: RUSTFLAGS -C instrument-coverage, cargo test, llvm-profdata merge, llvm-cov export --format=lcov. The commands locate the llvm tools through `rustc --print sysroot` because neither is on PATH here -- pasting `llvm-cov` bare fails with "not recognized as a cmdlet", which is a defect in an instruction meant to be run. Six lcov caveats go with it, each one measured rather than inferred: mangled names and the four demangler names that all fail (so the query is written module::function and matched per segment); several FN: lines pointing at one line for generic monomorphisation; a missing FNDA: reported as not evaluated rather than defaulting to 0; counts summed across records; a file absent from the tracefile reported as not evaluated, the opposite of c8; and the newer FNL:/FNA: encoding refused explicitly instead of read by a guessed shape.
AGENTS.md's bin/+src/ row now names four coverage formats and says what the lcov path trades: no --source needed because FN: carries the names, but the names are mangled and the query is matched by segment against legacy mangling, since no demangler works on this machine. The test/ row moves to 275 tests, notes that a pwsh run and a Git Bash run of the same tree report 275 + 0 and 274 + 1 (the bench case keys on tar flavour, not the OS), and adds coverage-lcov.test.js to the list of suites carrying fixtures -- its mangled names copied from a real llvm-cov export rather than invented. README's verified list moves from three formats to four and states what was measured on the probe crate, including that inlining does not flatten a called function to 0, and what the path cannot do: demangle. A bare "dup" matching every same-named function in the file is written down as behaviour rather than discovered later.
The measured-shape section said "a 12-function, 2-file probe crate". Three different counts are in play and the sentence collapsed them into one: the source holds 14 function declarations, the tracefile holds 15 FN: entries (one generic function is monomorphised onto two symbols at the same line), and llvm-cov's own FNF: summary counts them as 12 and 2 because it treats the two monomorphisations as one function. The paragraph now states which count it means, so a reader checking the sample against the numbers is not left to guess which of the three was meant. No behaviour changes. Suite: 275 pass / 0 fail.
…measured on The test/ row carried two baseline pairs in one sentence without saying that they belong to different commits: 245+1 / 246+0 were measured on cdece0f, before this branch existed, while 275 / 274 is this branch. Read in order the sentence yielded the wrong current baseline, which a reader test caught. The row now states each pair's scope explicitly and says which one is current. No behaviour changes. Suite: 275 pass / 0 fail.
…identifier
`lcovSymbolMatches` anchored a segment at a proven `<length><value>` boundary,
but `_RNvCsi8SMnIpCoEi_13collide_probe8cfg_3get` also reads as `...cfg_` + `3get`:
the character before `3get` is `_`, which the separator rule accepts, while
`8cfg_3get` is itself a complete pair, so `3get` is inside an identifier rather
than standing at a boundary.
Measured with a tracefile from a real `llvm-cov export --format=lcov` run whose
crate has both `get` (invoked once) and `cfg_3get` (never invoked):
node bin/euthyna.js coverage --coverage <tracefile> --symbol get --lang en
reported an Established finding — "Symbol get was not invoked even once in this
test run (invocation count 0)" — for the function that same record shows was
invoked, because the zero belonged to `cfg_3get`. The direction is the dangerous
one: a query about code that was exercised comes back as a refutation.
A needle landing strictly inside a longer complete pair is no longer accepted as
a boundary (`lcovInsideLongerPair`). A qualified `module::function` query stays a
subsequence test rather than a path-exact one — `lcov_probe::dup` also answers
for `lcov_probe::nested::dup` — so that residual over-inclusion is now recorded,
replacing a note that said qualifying the query was the way around it, which was
measured not to help: `--symbol collide_probe::get` produced the same false hit.
The emitted statement also names the symbol the match resolved to, not the
pattern that was typed, so a query with several hits cannot read as a claim about
the query text.
Eight regressions added. Two of them fail against the pre-fix file — `a query
never answers for an identifier that sits inside a longer one` and `a path query
states the symbol it resolved to, not the pattern` — pinned by reverting
src/facts/coverage.js to HEAD and re-running the file: 2 fail, then restored. The
other six cover guards that had no test (a null count surviving a duplicate-record
merge, the first FN: line winning, an unclosed record at EOF, and repeated
segments each consuming what they matched).
Suite: `node --test` -> 283 tests, 283 pass, 0 fail, 0 skipped.
…baseline The written record disagreed with the code in four places, each checked against the source or a run rather than restated: - The format count. `detectCoverageFormat` (`src/facts/coverage.js:727`) tells c8 from classic istanbul by one key — `:748` is `'hash' in e`, over the same `fnMap` + `f` shape — so those are two formats, not one. With coverage.py format 3, Go's text `-coverprofile` and lcov, that makes five; `docs/fact-contract.md` called lcov "the fourth format", both skill references said four, and `README.md`'s list omitted Go. The skill tables gain the classic istanbul row (jest/nyc) they were missing. - lcov's scope. "any gcov / llvm-cov target" is wider than measurement supports: the format travels further than the symbol matching does. A pure-C name never resolves — `FN:10,main` answers `Could not locate symbol main` for the needle `4main` — which is a limitation to state, not a target to imply. - Which mangling. The names come from Rust's v0 mangling, which `-C instrument-coverage` implies (`rustc -C help`: it "implies -C symbol-mangling-version=v0"); the text called it "legacy", and the legacy Itanium shape (`_ZN…17h…E`) is what a bare `rustc` emits, not a shape this path was measured on. The old note also offered qualifying the query as the way around an ambiguous bare name. Measured, that is not what a qualified query is: it is a subsequence test (`lcov_probe::dup` also answers `lcov_probe::nested::dup`), and on a hand-built Itanium fixture a qualified `lib::dup` matched nothing while a bare `dup` did. - The baseline. 275 -> 283 (246 + 29 lcov + 8 regressions) in the `test/` row and in the handover; the `cdece0f` 245/246 record stays where it is, as a dated anchor rather than a current count. Both parity guards (`docs-pair`, `skill-mirror`) are green over this diff, and the suite is unchanged at 283 tests, 283 pass, 0 fail, 0 skipped.
|
Sorry @modusensus, you've used your own review budget of 250,000 diff characters for the last 7 days. You can request another review in 3 days and 4 hours by commenting |
Reviewer's GuideThis PR closes the Rust coverage gap by adding shape-detected lcov parsing and conservative function-level findings, including Rust v0 mangled-symbol matching with a guard against false matches inside longer identifiers. It adds extensive parser, matcher, and regression tests, then synchronizes CLI help and English/Chinese documentation around the fifth supported coverage format and its measured limitations. Sequence diagram for lcov coverage fact productionsequenceDiagram
participant CLI
participant Coverage
participant Parser
participant Matcher
participant Facts
CLI->>Coverage: loadCoverage(coverageFile)
Coverage->>Coverage: detectCoverageFormat(coverage)
Coverage->>Parser: parseLcov(raw)
Parser-->>Coverage: lcov files and function counts
Coverage->>Matcher: lcovSymbolMatches(segments, mangled)
Matcher->>Matcher: lcovInsideLongerPair(name, idStart, idEnd)
Matcher-->>Coverage: matching symbols
Coverage->>Facts: collectLcovCoverageFacts(coverage, targets, lang)
Facts-->>CLI: established, unknown, or notEvaluated findings
Flow diagram for conservative lcov coverage verdictsflowchart LR
A[LCOV tracefile] --> B{Recognized shape?}
B -- No --> C[Refuse as not evaluated]
B -- Yes --> D[parseLcov]
D --> E{FN and FNDA available?}
E -- No --> F[notEvaluated]
E -- Yes --> G[Match mangled Rust v0 symbol]
G --> H{Symbol match?}
H -- No --> I[Unknown: symbol not located]
H -- Yes --> J{Invocation count}
J -- 0 --> K[Established: never invoked]
J -- greater than 0 --> L[Unknown: invoked, call site not proven]
G --> M[Reject matches inside longer pair]
D --> N[Sum repeated records and FNDA counts]
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
euthyna coveragerecognised four report shapes: c8/V8-to-istanbul JSON, classic istanbul(
hash-keyed), coverage.py JSON (format 3), and Go's text-coverprofile. Rust had none. That isthe one measure the four validation runs left with a confirmed gap:
historyanddepswereexercised on all four languages, coverage was not, and the rustls round recorded it in public —
docs/case-study-rust-zh.md§7 says coverage on Rust is still a gap and thateuthyna coveragedoes not recognise any Rust artifact (cargo-llvm-cov's lcov tracefile, llvm-cov's JSON export,
cargo-tarpaulin's lcov/Cobertura).
This adds the lcov tracefile as a fifth shape. It is the one Rust coverage actually arrives in
(
cargo-llvm-cov,cargo tarpaulin,llvm-cov export --format=lcov, and gcov's.infoare allthe same format), and it is also the shape C and C++ coverage arrives in.
The README's count said three, because it had never listed Go. The count and its members are
corrected here to match
detectCoverageFormat, which is what actually decides.Why lcov needs no
--source, and what it costsThe Go path needs
--source <module root>because a-coverprofilecarries no function names —the producer has to read the source tree and map line ranges back to functions. An lcov tracefile
does carry names (
FN:<line>,<name>and, separately,FNDA:<count>,<name>— two runs of lines,not adjacent pairs), so this path takes none. The price is that the names are mangled.
The intended fix was to demangle. It does not work on this machine and the record says so rather
than leaving the next agent to rediscover it:
llvm-cov export --Xdemangler=<name>was triedwith all four plausible names —
rust,rustfilt,llvm-cxxfilt,c++filt— and each failswith
error: no such file or directory: could not find the demangler!. No*filt*binary existsin the rustlib bin directory, and
c++filt/llvm-cxxfilt/rustfiltare not onPATH.So the matching runs on the mangled form. The names are Rust v0 mangling — what
-C instrument-coverageimplies, perrustc -C help— and v0 encodes a path segment as<decimal byte length><segment name>, so a querynested::dupbecomes the needle6nested3dup,anchored at a segment boundary: the start of the name, right after the mangling's own
_separator, or at a proven
<length><value>pair boundary. That is enough to tell two same-namedfunctions in one file apart, which is the property the test suite pins. A query that is already
mangled (
_R…,_Z…) is matched exactly and never half-decoded.Two limits are stated rather than implied, because the format travels further than the symbol
matching does. A pure C name never resolves — a gcov target's
FN:10,mainis answeredCould not locate symbol mainfor the needle4main. And the legacy Itanium shape(
_ZN…17h…E) is what a barerustcemits, not a shape this path was measured on; on a hand-builtfixture a qualified
lib::dupmatched nothing there while a baredupdid.The behaviour worth reviewing, and the bug review caught
The producer has exactly two results —
count === 0→ established "never invoked",count > 0→ unknown ("was entered", which proves no particular call site ran). There is no code path
that emits "executed", and every refusal below exists to keep that true:
FN:line with noFNDA:lineFN:lines at allFNvalue that is not<line>,<name>FNL:/FNA:(the newer encoding)measured: false— CI must not read it as cleanFNDAfor one symbolThe bug, on the dangerous side. The first version of the matcher anchored a segment at a
proven pair boundary, but a name like
_RNvCsi8SMnIpCoEi_13collide_probe8cfg_3getalso reads as...cfg_+3get: the character before3getis_, which the separator rule accepts, while8cfg_3getis itself a complete pair. Measured against a tracefile from a realllvm-cov exportrun whose crate has both
get(invoked once) andcfg_3get(never invoked):returned an Established finding — "Symbol get was not invoked even once in this test run
(invocation count 0)" — for the function that same record shows was invoked, because the zero
belonged to
cfg_3get. A query about code that was exercised came back as a refutation, which isthe direction this producer must not fail in. The statement also named the query text rather
than the symbol it resolved to, so a query with several hits could read as a claim about the
pattern.
Both are fixed: a needle landing strictly inside a longer complete pair is no longer accepted as a
boundary (
lcovInsideLongerPair), and the statement names the matched symbol. Eight regressionswere added; two of them fail against the pre-fix file — verified by reverting
src/facts/coverage.jsto its parent and re-running the file (2 fail), then restoring it(sha256 unchanged). One residual over-inclusion is documented rather than hidden: a qualified
query is a subsequence test, not a path-exact one, so
lcov_probe::dupalso answerslcov_probe::nested::dup. Each fact names the symbol it is about.Changes
src/facts/coverage.js—isLcovTracefile,parseLcov,lcovSymbolNeedle,lcovSymbolMatches(with thelcovInsideLongerPairboundary guard),collectLcovCoverageFacts;detectCoverageFormatlearns'lcov';loadCoveragedispatches onshape like the other four.
test/coverage-lcov.test.js(new) — 37 cases, of which 8 are the review-driven regressions. Thefixtures carry mangled names copied from a real
llvm-cov export --format=lcovrun, notinvented strings.
src/cli.js— both language editions of the help name the fifth shape..agents/skills/euthyna{,-en}/references/fact-producers.md— the lcov path, the three-stepproduction recipe, six lcov-specific cautions, and the classic istanbul row the format table was
missing.
docs/fact-contract{,-zh}.md§6.2 — the measured shape and what Rust can now answer.docs/case-study-rust{,-zh}.md§7 — a follow-up block: the gap that section records has sincebeen closed. The original text stays; it was true when written.
AGENTS.md,README.md— the format count (five, with Go restored to it), the lcov scope, andthe stale test counts rescoped to the commit they were measured on.
Where the format was measured
Not from the spec — from a probe crate with 14 functions over 2 files (15
FN:entries: onegeneric is instantiated twice on the same line), built with
RUSTFLAGS=-C instrument-coverage, run undercargo test, merged withllvm-profdata merge -sparse, exported withllvm-cov export --format=lcov. That run is what established the shapethe parser is written against, including the details that are not obvious from documentation: the
FN:/FNDA:pairs are two blocks rather than adjacent lines, one line can carry more than oneFN:entry (generic monomorphisation, closures),SF:paths are absolute, andnever_calledisthe only function with count 0 in
lib.rswhileother.rs's two functions are both 0. Everycharacter of those 15 symbol names lies in
[A-Za-z0-9_].One inference was tested separately because it is the single fatal counterexample to
count === 0 ⇒ never invoked: inlining was measured not to flatten an invoked function to 0.A separate release-built probe with
#[inline(always)]still reportsFNDA:1for the inlinedfunction.
Reproduce end to end against the real file:
Not in this PR
chore(release): X.Y.Zcommit (see4359836,ef79576); this branch does not touchpackage.json.--format=json, the region model). Different field model; mixingboth in one round would leave the fixtures unable to say which shape they test.
FNL:/FNA:. They are refused with a reason today. The parser was written from a runthat does not emit them, so reading them would be guessing at a shape — noted as inference in
the code rather than presented as measured.
build" from "cannot be located". The lcov path reports both as unknown, which is the safe side
but a weaker answer; recorded as a known difference.
--help. It is prose in two languages, and a mechanicalguard there would pin wording rather than behaviour.
Test plan
node --test→ pwsh 283 pass / 0 fail / 0 skip; Git Bash 282 pass / 1 skip (thebsdtar-keyed exploit case, same as every Git Bash run on this machine).
246 existing + 29 lcov + 8 review-driven regressions = 283.
test/coverage-lcov.test.jsalone → 37 pass / 0 fail.test/docs-pair.test.js+test/skill-mirror.test.js→ both editions of the producerreference and both
fact-contractfiles stay structurally paired, over this diff.the refusal exit (
EXIT.UNMEASURED= 2, which is what a query answered only by refusalsreturns;
EXIT.USAGE= 1 is a different thing, a missing--symbol) and a malformed body.Re-run after the matcher fix: all 16 probe symbols resolve as before.
src/cli.js:EXIT = { CLEAN: 0, FLAGGED: 10, USAGE: 1, UNMEASURED: 2 }, and the coverage command returnsmeasured ? CLEAN : UNMEASURED. Notethat exit 0 means "facts were produced", not "clean" — an
unknownverdict also exits 0.38168625070: all six matrix legs (Node 20/22/24 × ubuntu/windows) pass, plusthe live licence check, the golden bench round, and CommitCheck. Sourcery skips.