Skip to content

ci: say which test failed, on the run's front page - #137

Merged
VDBBjorn merged 3 commits into
mainfrom
claude/ci-test-summary
Aug 8, 2026
Merged

ci: say which test failed, on the run's front page#137
VDBBjorn merged 3 commits into
mainfrom
claude/ci-test-summary

Conversation

@VDBBjorn

@VDBBjorn VDBBjorn commented Aug 4, 2026

Copy link
Copy Markdown
Member

Closes #139

What changed

A red build-test used to name the failing test only in the raw job log — on ~2000 tests
that is a scroll to find one line, and the run's front page said nothing at all.

It was worse than that. build/coverage.sh runs under set -e, so the failing dotnet test
aborted the script: no Cobertura.xml was rendered, the coverage summary printed nothing, and
Upload coverage failed on if-no-files-found: error. Two extra red steps, both burying the
one that mattered.

Three changes:

  • build/test-summary.py (new) parses .trx and prints a verdict line, a per-assembly
    table, and a <details> block per failure — message in full, stack trimmed to six frames.
    --annotate additionally emits ::error title=…:: workflow commands, which is what puts a
    failure on the Files-changed tab next to the test that produced it. It parses the trx rather
    than console output because the console format is a rendering choice of whichever logger was
    enabled, while the trx is the recorded result. Finding no trx files is not an error: a job
    that died before any test ran has nothing to summarise, and a stack trace from this script
    on top of the real failure helps nobody.
  • build/coverage.sh asks for a trx logger, carries the dotnet test exit status to the
    end instead of aborting on it, and tolerates reportgenerator finding nothing to report. It
    still exits non-zero, so the step is still red — it just stops taking the rest of the job
    down with it.
  • .github/workflows/ci.yml publishes both summaries to $GITHUB_STEP_SUMMARY with
    if: always() (the pattern mutation.yml already uses), and the examples run now writes a
    trx of its own so its failures land there too.

Verified locally both ways: a passing run renders

🟢 passed — 1487/1490 passed, 0 failed, 3 skipped

with the per-assembly table, and a deliberately failing test renders 🔴 failed, the
<details> block with xUnit's full message and a six-frame stack, and the matching
::error title=…:: line.

Checklist

  • dotnet test passes locally (Postgres-backed tests need a running Docker daemon)
  • Public API additions or removals are reflected in PublicAPI.Unshipped.txt / PublicAPI.Shipped.txt — no public API touched
  • breaking-change label applied if a consumer breaks — nothing breaks; build scripts and workflow only
  • CHANGELOG.md not hand-edited; the release workflow drafts it

A red `build-test` named the failing test only in the raw job log — a scroll
through ~2000 test lines to find one. Worse, `build/coverage.sh` aborted on the
failing `dotnet test` under `set -e`, so no report was rendered, the coverage
summary printed nothing and the artifact upload failed for a missing
Cobertura.xml: two extra red steps burying the one that mattered.

- `build/test-summary.py` parses the trx and prints a verdict line, a
  per-assembly table and a `<details>` block per failure (message in full,
  stack trimmed to six frames). `--annotate` also emits `::error::` commands,
  which puts each failure on the Files-changed tab.
- `build/coverage.sh` asks for a trx logger, carries the test exit status to
  the end instead of aborting, and tolerates reportgenerator finding nothing.
- CI publishes both summaries with `if: always()`, and the examples run now
  writes a trx too.

The trx is parsed rather than the console output because the console format is
a rendering choice of whichever logger was enabled; the trx is the record.
@VDBBjorn VDBBjorn added this to the 0.3.0 milestone Aug 4, 2026
@VDBBjorn
VDBBjorn enabled auto-merge (squash) August 4, 2026 19:15
@VDBBjorn
VDBBjorn merged commit 2212cba into main Aug 8, 2026
4 checks passed
@VDBBjorn
VDBBjorn deleted the claude/ci-test-summary branch August 8, 2026 05:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A failing CI run does not say which test failed

1 participant