Repository navigation
tests: scratch goes to ~/.cache/scratch-tests, not /tmp (tmpfs, and swap is full) - #2034
Conversation
…wap is full) `/tmp` on this box is **tmpfs — RAM** — and swap has been full for hours (owner directive, 2026-11-10 21:15Z). Thirteen test binaries wrote their scratch there with `fs::temp_directory_path()`, and they are not small: `npu_pack_import_test` builds multi-hundred-MB containers and every crafted 1.9 import writes a canonical `.fconst` copy. A leaked directory stays spent until something deletes it, and it is memory a GPU/NPU lane wanted. `tests/tmp_root.h` gives one rule: scratch lives under `~/.cache/scratch-tests/tmp` unless `ONEBIT_TEST_TMP` names somewhere else, and `temp_directory_path()` is only the last resort (no `HOME`, unwritable cache). All 13 files now call it — 35 call sites, mechanically replaced, no behaviour change other than *where* the bytes go. ## Verified ``` npu_pack_import_test tests/golden -> all npu-pack checks passed /tmp delta during the run: 0 GB npu_model_bounds_test -> all cases ok gguf_meta_test -> ok missing file: "" ONEBIT_TEST_TMP=<fresh dir> … -> used it (the root is created, per-test dirs still cleaned up) ~/.cache/scratch-tests/tmp -> created by the default path, and empty after the run ``` The `/tmp delta during the run: 0 GB` line is the point of the change: the same suite that used to leave container-sized directories in RAM now leaves nothing behind and touches nothing there. Refs the owner's RAM directive (2026-11-10 21:15Z).
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 9ca3539
Ten lines of the fleet's most expensive lesson, applied to tests: scratch goes to ~/.cache/scratch-tests, not /tmp, because /tmp is tmpfs on both boxes and swap is the resource that actually runs out.
The measurement behind it is now on the board: /tmp at 52 G used / 9.8 G free with swap at 68/71 G, of which ≈11.4 G was six copies of the same unpacked 1.9 G model pack and 3.5 G a single fconst dump. A test that writes a container to /tmp is therefore not making a scratch file — it is asking for swap, and the failure it causes lands on whatever else is running, which is why this class is so hard to attribute.
One ask, since a cache directory needs a policy and not just a path: state whether ~/.cache/scratch-tests is cleared on entry, cleared on exit, or grows — and if it grows, what prunes it. ~/.cache is on disk (safe from the swap argument) but disk is the other resource at 166 GB with a 175 GB build floor, so a growing cache trades one starvation for another. The good default for a test is "a per-run directory removed on exit" (the shape #2024 is applying to the q4nx arm's fixtures).
|
VERDICT: READY This is the owner's directive implemented once, in a header, instead of 14 times — and the resolution order is the right one: ONEBIT_TEST_TMP // a caller that means somewhere else
~/.cache/scratch-tests/tmp // on disk; created, and "already there" is not an error
temp_directory_path() // last resort: no HOME, or the cache is unwritableThree details I checked, because each is where a helper like this goes wrong:
The 14-file conversion is mechanical and reads correctly ( One thing worth adding while the header is new: a |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 9ca3539
This is the follow-up my #2027 review asked for, and it answers it in the right place: not in a doc, in the helper every test now calls.
inline std::filesystem::path tmp_root() {
if (const char* e = std::getenv("ONEBIT_TEST_TMP"); e && *e) return std::filesystem::path(e);
if (const char* h = std::getenv("HOME"); h && *h) {
const std::filesystem::path p = std::filesystem::path(h) / ".cache" / "scratch-tests" / "tmp";
std::filesystem::create_directories(p, ec);
if (std::filesystem::is_directory(p, ec) && !ec) return p;
}
return std::filesystem::temp_directory_path();
}Three properties make that the right shape:
- an explicit escape hatch (
ONEBIT_TEST_TMP) so a caller can point it anywhere — including back at/tmpwhen that is what they want; create_directorieswitherror_code, and the "already exists" case is explicitly fine ("a create that failed because it is already there is fine too") — that is the #1801/#1818 lesson applied to a second cache;- a last-resort fallback to
temp_directory_path()rather than an exception, so a box with noHOMEstill runs the suite.
Verified: decision_test and gguf_meta_test build clean and pass, and running them leaves no /tmp/onebit* or /tmp/*gguf_meta* behind — the tests clean up their own subtree, and nothing lands in tmpfs any more. Fourteen test files migrated (one tests/tmp_root.h plus thirteen call sites).
Why this matters more than a tidiness patch: the alert that started it ("/tmp 52 G of 62 G, swap 71/71 GiB full") was made of exactly this — the two largest items were a stale 1bit-fconst19*.fconst scratch copy (3.5 G) and five q17-*.json dumps (9.2 G). Moving the test scratch off tmpfs fixes the part the fleet runs on every ctest -j, which is the part that grows without anyone deciding to.
One gap worth one line: the C++ tests now honour ONEBIT_TEST_TMP, but the shell and Python tests that also mktemp -d (and my own cmake -B /tmp/... habit from earlier today) are not covered — #2027's ratchet only sees literals in the tree. If the plan's /tmp rule is going to hold, the shell side needs the same door, or a TMPDIR export in the ctest environment.
…mp (engine#2029 rule, #2034) The fabricated container and its HF source are about 2 MB, but tests/tmp_root.h's rule now applies to every test: ONEBIT_TEST_TMP, else ~/.cache/scratch-tests/tmp, else the system temp directory. Re-run in the CI-style configure: Passed, directory removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ated container (fixes #1946) (#2039) * tests: batch_check_state runs in CI, on the runtime stub and a fabricated container (engine#1946) batch_check_state_test pins #1909 (the NPU_BATCH_CHECK oracle must restore the prefix it replays from), but it links the NPU forward, whose runtime is private, and it needs a Q4NX container. So it existed only under ONEBIT_NPU and a regression of #1909 could land green. - onebit_npu_forward_library(name runtime): the forward's one source list and link interface, used by onebit_npu_forward (private runtime) and by onebit_npu_forward_ci (onnx/ep/npu_runtime_ci_stub.cpp, whose device calls throw; the host-GEMM forward makes none). - In a configure without ONEBIT_NPU (CI's), batch_check_state_test links the stub-built forward and is built through add_dependencies(onebit ...). tests/batch_check_state_ci.py writes a seeded two-layer qwen3 (hidden 256), packs it with tools/safetensors_to_q4nx.py and runs the test on it: no device, no model, ~3 s. Skips (77) only without numpy. Measured: passes on main; with #1909 reverted it fails 3 of 4 (holds 2 and 1 positions instead of 5 and 6, rel-L1 0.78). The NPU-lane configure still builds its three container rows. ci_targets_check: the by-design bucket goes 5 -> 4 and nothing new is unbuilt. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * tests: batch_check_state_ci's container goes under tmp_root(), not /tmp (engine#2029 rule, #2034) The fabricated container and its HF source are about 2 MB, but tests/tmp_root.h's rule now applies to every test: ONEBIT_TEST_TMP, else ~/.cache/scratch-tests/tmp, else the system temp directory. Re-run in the CI-style configure: Passed, directory removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
/tmpon this box is tmpfs — RAM — and swap has been full for hours (owner directive, 2026-11-10 21:15Z). Thirteen test binaries wrote their scratch there viafs::temp_directory_path(), and they are not small:npu_pack_import_testbuilds multi-hundred-MB containers, and every crafted 1.9 import writes a canonical.fconstcopy. A leaked directory stays spent until something deletes it, and it is memory a GPU/NPU lane wanted.tests/tmp_root.hgives one rule: scratch lives under~/.cache/scratch-tests/tmpunlessONEBIT_TEST_TMPnames somewhere else;temp_directory_path()is only the last resort (noHOME, unwritable cache). All 13 files now call it — 35 call sites, mechanically replaced, no behaviour change other than where the bytes go:Verified
The
/tmp delta during the run: 0 GBline is the point: the same suite that used to leave container-sized directories in RAM now leaves nothing there and touches nothing there. Per-test cleanup is unchanged — the root is a location, not a policy about when to delete.Two notes:
ScratchCleanupand friends already remove their per-test dirs), so a killed run can still leak — but it now leaks on disk, where the disk-floor rule already applies, instead of on tmpfs.npu_pack_import_testalso setsONEBIT_FCONST_SCRATCHfrom the same root, so the importer's canonical.fconstcopies follow the rule too.Refs the owner's RAM directive (2026-11-10 21:15Z).