//domains/platform/libs/server_pal:server_pal_test failed once on #1489 (job), on a head that touched no Rust:
thread 'tests::absent_headers_log_as_empty_fields' panicked at domains/platform/libs/server_pal/src/lib.rs:896:32:
no access line in:
The other 19 tests passed, the same test passes locally and on main's runs, and the PR merged through the failure. The captured buffer was empty, not wrong: the access line never reached the test's writer.
Where to look. access_line (lib.rs ~870-900) installs its subscriber with tracing::subscriber::set_default, which is thread-local, and captures through a SharedWriter buffer. access_log_middleware emits the line with tracing::info!. Anything that moves the emit off the test's thread, or any test in the binary that installs a dispatcher another way while this one runs (the tests run in parallel), loses the line. Candidates: a multi-thread runtime flavour somewhere in the file, init_logging() being called from a sibling test, or the middleware's span/emit crossing a spawn_blocking.
Fix shape. Make the capture not depend on thread identity: run the request inside tracing::subscriber::with_default(subscriber, async { ... }) (the future-scoped form), or give the test binary one global subscriber whose writer is a per-test slot. Then reproduce the flake by looping the test binary a few hundred times before and after.
Filed from the #1489 standing-down comment; the rule is a failing test is never an infra flake.
//domains/platform/libs/server_pal:server_pal_testfailed once on #1489 (job), on a head that touched no Rust:The other 19 tests passed, the same test passes locally and on main's runs, and the PR merged through the failure. The captured buffer was empty, not wrong: the access line never reached the test's writer.
Where to look.
access_line(lib.rs~870-900) installs its subscriber withtracing::subscriber::set_default, which is thread-local, and captures through aSharedWriterbuffer.access_log_middlewareemits the line withtracing::info!. Anything that moves the emit off the test's thread, or any test in the binary that installs a dispatcher another way while this one runs (the tests run in parallel), loses the line. Candidates: a multi-thread runtime flavour somewhere in the file,init_logging()being called from a sibling test, or the middleware's span/emit crossing aspawn_blocking.Fix shape. Make the capture not depend on thread identity: run the request inside
tracing::subscriber::with_default(subscriber, async { ... })(the future-scoped form), or give the test binary one global subscriber whose writer is a per-test slot. Then reproduce the flake by looping the test binary a few hundred times before and after.Filed from the #1489 standing-down comment; the rule is a failing test is never an infra flake.