Skip to content

fix: fail runs when a --file filter matches no test - #44

Merged
witqq merged 1 commit into
masterfrom
fix/missing-file-filter
Sep 24, 2026
Merged

witqq merged 1 commit into
masterfrom
fix/missing-file-filter

Conversation

@witqq

@witqq witqq commented Sep 24, 2026

Copy link
Copy Markdown
Owner

Problem

--file values are passed to Jest and Playwright as positional test-path filters, which both frameworks combine as a union. A path that matched nothing was silently dropped whenever another path matched, and the run reported success:

testfold unit --file tests/unit/shared/labels.test.ts --file tests/unit/shared/does-not-exist.test.ts --no-parallel
TESTFOLD_RESULT {"success":true,"passed":11,...,"exitCode":0}

Fix

  • Jest and Playwright parsers report testFiles, the test files each suite ran (Playwright paths are resolved against config.rootDir). Custom parsers may return testFiles; otherwise a complete testResults list is used.
  • After all selected suites finish, each --file filter must match a test file in at least one selected suite, using the frameworks' semantics (regular expression over the test path, optional Playwright :line suffix) plus literal path comparison.
  • Each unmatched filter becomes an infrastructure failure Unmatched File Filter: <filter> on the first selected suite; the run exits with code 2.
  • A filter that matches a file in another selected suite is not an error. The check is skipped when a selected suite has no file inventory (fail-fast skip, beforeSuite guard, timeout, missing result file, custom parser without file data).
  • With --file, reporters receive suite results after the check, so a suite is never printed as passed first.

Rule documented in README (File Filters), CLI help, architecture doc and changelog. Version 0.4.5.

Verification

  • npm run verify — exit 0; Unit 356 passed; integration 2 passed; workflow and package checks passed.
  • Regression: tests/unit/core/orchestrator-file-filters.test.ts — two of its tests fail on master sources and pass with the fix.
  • Consumer reproduction with the built CLI now reports ✗ unit: 11 passed, 1 failed, names Unmatched File Filter: tests/unit/shared/does-not-exist.test.ts, and ends with TESTFOLD_RESULT:{"success":false,"passed":11,"failed":1,...,"exitCode":2}.

Jest and Playwright combine positional test-path filters as a union, so a
requested --file path that matched nothing was silently dropped whenever
another path matched, and the run reported success.

Parsers now report the test files each suite ran. After all selected suites
finish, every --file filter must match a test file in at least one selected
suite; each unmatched filter is recorded as an infrastructure failure on the
first selected suite and the run exits with code 2. The check is skipped when
a selected suite has no file inventory (skipped, guarded, timed out, missing
result file, or a custom parser without file data).
@witqq
witqq merged commit 27ce4bd into master Sep 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant