Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ Notable changes to Testfold are documented here. The project follows Semantic Ve

## Unreleased

## 0.4.5 — 2026-09-24

- Fail the run with exit code 2 and name each `--file` filter that selected no test file in any selected suite, instead of silently dropping it when another filter matched. A filter that matches a file in another selected suite is not an error.

## 0.4.4 — 2026-09-07

- Count Playwright tests that pass on retry as successful tests in suite totals instead of silently omitting flaky outcomes from the summary.
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ testfold -r console,json,timing-text # Multiple reporters
testfold -r ./my-reporter.ts # Custom reporter from file
testfold -g "auth" # Filter by test name
testfold --grep-invert "slow" # Exclude tests by pattern
testfold -f auth.test.ts # Filter by file
testfold -f auth.test.ts # Filter by file (a filter matching no test in any selected suite fails the run, exit 2)
testfold --dry-run # Preview commands without running
testfold --dry-run unit # Preview only unit suite

Expand Down
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,18 @@ export default {

Returning `void` or `undefined` is treated as success (backward compatible).

### File Filters

`--file` (`-f`) values are passed to Jest and Playwright as positional test-path filters, which the frameworks treat as regular expressions searched in each test file's path; Playwright also accepts a `:line` suffix. The frameworks combine several filters as a union, so testfold checks each requested filter after the run:

- A filter must select at least one test file that one of the selected suites ran. A filter that matches a file in any selected suite is satisfied; it does not have to match every suite.
- Each filter that selected no test file in any selected suite is recorded as an infrastructure failure named `Unmatched File Filter: <filter>`, attached to the first selected suite. The run fails with exit code `2`, even when other filters matched and their tests passed.
- The check applies only when every selected suite ran and reported the test files it executed. When a suite was skipped by fail-fast, stopped by a `beforeSuite` guard, timed out, produced no result file, or uses a custom parser that reports neither `testFiles` nor a complete `testResults` list, the check is skipped because the filter may belong to that suite; such a run already fails or depends on that suite's own result.

With `--file`, reporters receive each suite's result after all selected suites finish, so a suite is never shown as passed before the check.

Custom parsers can return `testFiles` (the test files the framework ran) in their `ParseResult` to take part in the check.

### Path Prefix Resolution

Pass partial test file names and testfold resolves them to full paths:
Expand Down
3 changes: 2 additions & 1 deletion docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ testfold is a unified test runner that orchestrates multiple test frameworks (Je
### Core Layer (`src/core/`)

- **runner.ts** - `TestRunner` class, main entry point. Creates reporters (built-in + custom from file paths)
- **orchestrator.ts** - Coordinates suite execution (parallel/sequential), guard-aware hooks, filter flag passing
- **orchestrator.ts** - Coordinates suite execution (parallel/sequential), guard-aware hooks, filter flag passing, and `--file` coverage: a requested file that selected no test in any selected suite fails the run with exit code 2
- **executor.ts** - Spawns child processes with `detached: true`, captures output. SIGKILL escalation on timeout, workers→maxWorkers mapping, progress streaming via onOutput callback

### Parsers (`src/parsers/`)
Expand All @@ -63,6 +63,7 @@ testfold is a unified test runner that orchestrates multiple test frameworks (Je
- **ansi.ts** - Strip ANSI escape codes
- **sanitize.ts** - Sanitize strings for filenames
- **files.ts** - Directory cleanup, file writing
- **file-filters.ts** - Matches `--file` filters against the test files each suite ran; used by the orchestrator to fail runs where a requested file selected no test

### CLI (`src/cli/`)

Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "testfold",
"version": "0.4.4",
"version": "0.4.5",
"packageManager": "npm@11.19.1",
"description": "Unified test runner with Jest and Playwright support, parallel execution, and detailed failure reporting",
"type": "module",
Expand Down
6 changes: 4 additions & 2 deletions src/cli/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,9 @@ Options:
Custom: path to reporter module (e.g., ./my-reporter.ts)
--grep, -g <pattern> Filter tests by name pattern
--grep-invert <pattern> Exclude tests matching pattern
--file, -f <path> Filter by test file (repeat for multiple files)
--file, -f <path> Filter by test file (repeat for multiple files);
a file matching no test in any selected suite
fails the run with exit code 2
--dry-run Print planned commands without executing
--no-parallel Run suites sequentially
--fail-fast Stop on first failure
Expand All @@ -146,7 +148,7 @@ Options:
Exit Codes:
0 All tests passed
1 One or more tests failed
2 Infrastructure error (config, parse, spawn failure)
2 Infrastructure error (config, parse, spawn failure, unmatched --file)
3 Suite killed by timeout

Pass-through Arguments:
Expand Down
2 changes: 2 additions & 0 deletions src/config/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ export interface SuiteResult {
resultFile: string;
/** Individual test results for timing analysis */
testResults?: TestResult[];
/** Test files the framework ran, when the parser could determine them */
testFiles?: string[];
/** Error category for semantic exit code determination */
errorCategory?: ErrorCategory;
}
Expand Down
98 changes: 87 additions & 11 deletions src/core/orchestrator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import { CustomParserLoader } from '../parsers/custom.js';
import { executeCommand } from './executor.js';
import { readEnvFileContent } from '../config/env-loader.js';
import { createProgressCallback } from '../utils/progress.js';
import { findUnmatchedFileFilters, normalizeFileFilters } from '../utils/file-filters.js';

export interface OrchestratorOptions {
config: ValidatedConfig;
Expand Down Expand Up @@ -83,10 +84,26 @@ export class Orchestrator {
await this.config.hooks.beforeAll();
}

// With --file filters, suite results are reported only after every suite has
// finished, because an unmatched filter can be judged only across all of them.
const fileFilters = normalizeFileFilters(this.file);
const notifyPerSuite = fileFilters.length === 0;

// Execute suites
const results = this.config.parallel
? await this.runParallel(suitesToRun)
: await this.runSequential(suitesToRun);
? await this.runParallel(suitesToRun, notifyPerSuite)
: await this.runSequential(suitesToRun, notifyPerSuite);

if (!notifyPerSuite) {
this.applyFileFilterCoverage(fileFilters, suitesToRun, results);
results.forEach((result, index) => {
const suite = suitesToRun[index];
if (!suite) return;
for (const reporter of this.reporters) {
reporter.onSuiteComplete(suite, result);
}
});
}

// Aggregate results
const aggregated = this.aggregate(results);
Expand All @@ -104,16 +121,57 @@ export class Orchestrator {
return aggregated;
}

private async runParallel(suites: Suite[]): Promise<SuiteResult[]> {
const promises = suites.map((suite) => this.runSuite(suite));
/**
* Fail the run when a requested --file filter selected no test file in any
* selected suite. Frameworks combine positional filters as a union, so without
* this check a mistyped path is silently dropped while other paths pass.
*
* The check applies only when every selected suite ran and reported the test
* files it executed. Otherwise the run already fails for that suite's own
* reason, and an unmatched filter could belong to the suite whose files are
* unknown. Unmatched filters are recorded as infrastructure failures on the
* first selected suite and name every suite that was searched.
*/
private applyFileFilterCoverage(
filters: string[],
suites: Suite[],
results: SuiteResult[],
): void {
const target = results[0];
if (!target || results.length < suites.length) return;

const unmatched = findUnmatchedFileFilters(
filters,
results.map((result) => result.testFiles),
this.cwd,
);
if (!unmatched || unmatched.length === 0) return;

const searched = suites.map((suite) => suite.name).join(', ');
for (const filter of unmatched) {
target.failures.push({
testName: `Unmatched File Filter: ${filter}`,
filePath: '',
error: `--file ${filter} matched no test file in the selected suites (${searched}). No test from this path ran.`,
});
target.failed += 1;
}
target.success = false;
if (target.errorCategory !== 'timeout') {
target.errorCategory = 'infra_error';
}
}

private async runParallel(suites: Suite[], notify: boolean): Promise<SuiteResult[]> {
const promises = suites.map((suite) => this.runSuite(suite, notify));
return Promise.all(promises);
}

private async runSequential(suites: Suite[]): Promise<SuiteResult[]> {
private async runSequential(suites: Suite[], notify: boolean): Promise<SuiteResult[]> {
const results: SuiteResult[] = [];

for (const suite of suites) {
const result = await this.runSuite(suite);
const result = await this.runSuite(suite, notify);
results.push(result);

if (this.config.failFast && !result.success) {
Expand All @@ -124,7 +182,7 @@ export class Orchestrator {
return results;
}

private async runSuite(suite: Suite): Promise<SuiteResult> {
private async runSuite(suite: Suite, notify: boolean): Promise<SuiteResult> {
// Run before hook and check guard result
if (this.config.hooks?.beforeSuite) {
const guardResult = await this.config.hooks.beforeSuite(suite);
Expand All @@ -142,8 +200,10 @@ export class Orchestrator {
resultFile: '',
errorCategory: 'infra_error',
};
for (const reporter of this.reporters) {
reporter.onSuiteComplete(suite, result);
if (notify) {
for (const reporter of this.reporters) {
reporter.onSuiteComplete(suite, result);
}
}
return result;
}
Expand Down Expand Up @@ -258,6 +318,7 @@ export class Orchestrator {
logFile,
resultFile,
testResults: parseResult.testResults,
testFiles: errorCategory === 'timeout' ? undefined : this.getTestFiles(parseResult),
errorCategory,
};

Expand All @@ -276,13 +337,28 @@ export class Orchestrator {
}

// Notify reporters (after guard check so reporters see final state)
for (const reporter of this.reporters) {
reporter.onSuiteComplete(suite, result);
if (notify) {
for (const reporter of this.reporters) {
reporter.onSuiteComplete(suite, result);
}
}

return result;
}

/**
* Test files the suite ran. Parsers that do not report them (custom parsers)
* fall back to the files of their test results, but only when those results
* account for every counted test; a partial list cannot prove a filter missed.
*/
private getTestFiles(parseResult: ParseResult): string[] | undefined {
if (parseResult.testFiles) return parseResult.testFiles;
const tests = parseResult.testResults;
const counted = parseResult.passed + parseResult.failed + parseResult.skipped;
if (!tests || tests.length !== counted) return undefined;
return [...new Set(tests.map((test) => test.file))];
}

private getParser(suite: Suite) {
switch (suite.type) {
case 'jest':
Expand Down
1 change: 1 addition & 0 deletions src/parsers/jest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,7 @@ export class JestParser implements Parser {
success: data.success && crashedSuites === 0,
failures,
testResults,
testFiles: data.testResults.map((fileResult) => fileResult.name),
};
}

Expand Down
22 changes: 21 additions & 1 deletion src/parsers/playwright.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,12 @@

import { readFile } from 'node:fs/promises';
import { existsSync } from 'node:fs';
import { isAbsolute, resolve } from 'node:path';
import type { Parser, ParseResult, TestResult } from './types.js';
import type { FailureDetail } from '../config/types.js';

interface PlaywrightResult {
config: unknown;
config?: { rootDir?: string };
suites: PlaywrightSuite[];
stats: {
startTime: string;
Expand Down Expand Up @@ -160,9 +161,28 @@ export class PlaywrightParser implements Parser {
success: data.stats.unexpected === 0,
failures,
testResults,
testFiles: this.collectTestFiles(data.suites, data.config?.rootDir),
};
}

/**
* Playwright reports spec files relative to its rootDir. Resolve them so they
* can be compared with the paths requested through `--file`.
*/
private collectTestFiles(suites: PlaywrightSuite[], rootDir?: string): string[] {
const files = new Set<string>();
const visit = (items: PlaywrightSuite[]) => {
for (const suite of items) {
if (suite.file) {
files.add(rootDir && !isAbsolute(suite.file) ? resolve(rootDir, suite.file) : suite.file);
}
if (suite.suites) visit(suite.suites);
}
};
visit(suites);
return [...files];
}

private collectResults(
suites: PlaywrightSuite[],
parentTitle: string,
Expand Down
7 changes: 7 additions & 0 deletions src/parsers/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@ export interface ParseResult {
errorCategory?: ErrorCategory;
/** Raw test results for timing analysis */
testResults?: TestResult[];
/**
* Every test file the framework ran, including files that crashed before
* reporting a test. Used to confirm that each requested `--file` filter
* selected at least one test file. Custom parsers may omit it; `testResults`
* files are then used instead.
*/
testFiles?: string[];
}

export interface Parser {
Expand Down
Loading