Skip to content

tests: scratch goes to ~/.cache/scratch-tests, not /tmp (tmpfs, and swap is full) - #2034

Merged
bong-water-water-bong merged 1 commit into
mainfrom
fix/tests-tmp-root
Nov 10, 2026
Merged

bong-water-water-bong merged 1 commit into
mainfrom
fix/tests-tmp-root

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

/tmp on this box is tmpfs — RAM — and swap has been full for hours (owner directive, 2026-11-10 21:15Z). Thirteen test binaries wrote their scratch there via fs::temp_directory_path(), and they are not small: npu_pack_import_test builds multi-hundred-MB containers, and every crafted 1.9 import writes a canonical .fconst copy. A leaked directory stays spent until something deletes it, and it is memory a GPU/NPU lane wanted.

tests/tmp_root.h gives one rule: scratch lives under ~/.cache/scratch-tests/tmp unless ONEBIT_TEST_TMP names somewhere else; temp_directory_path() is only the last resort (no HOME, unwritable cache). All 13 files now call it — 35 call sites, mechanically replaced, no behaviour change other than where the bytes go:

decision_test  gguf_meta_test  laya_bounds_test  mem_admit_test  moe_gguf_index_test
npu_flm_head_import_test  npu_model_bounds_test  npu_moe_test  npu_pack_import_test
npu_plan_test  npu_private_route_test  npu_target_caps_test  q4nx_cache_test

Verified

npu_pack_import_test tests/golden   -> all npu-pack checks passed    /tmp delta during the run: 0 GB
npu_model_bounds_test               -> all cases ok
gguf_meta_test                      -> ok   missing file: ""
ONEBIT_TEST_TMP=<fresh dir> …       -> used it (root created, per-test dirs still cleaned up)
~/.cache/scratch-tests/tmp          -> created by the default path, and empty after the run

The /tmp delta during the run: 0 GB line is the point: the same suite that used to leave container-sized directories in RAM now leaves nothing there and touches nothing there. Per-test cleanup is unchanged — the root is a location, not a policy about when to delete.

Two notes:

  • I did not touch the binaries' own cleanup (ScratchCleanup and friends already remove their per-test dirs), so a killed run can still leak — but it now leaks on disk, where the disk-floor rule already applies, instead of on tmpfs.
  • npu_pack_import_test also sets ONEBIT_FCONST_SCRATCH from the same root, so the importer's canonical .fconst copies follow the rule too.

Refs the owner's RAM directive (2026-11-10 21:15Z).

…wap is full)

`/tmp` on this box is **tmpfs — RAM** — and swap has been full for hours (owner directive, 2026-11-10 21:15Z). Thirteen test
binaries wrote their scratch there with `fs::temp_directory_path()`, and they are not small: `npu_pack_import_test` builds
multi-hundred-MB containers and every crafted 1.9 import writes a canonical `.fconst` copy. A leaked directory stays spent
until something deletes it, and it is memory a GPU/NPU lane wanted.

`tests/tmp_root.h` gives one rule: scratch lives under `~/.cache/scratch-tests/tmp` unless `ONEBIT_TEST_TMP` names somewhere
else, and `temp_directory_path()` is only the last resort (no `HOME`, unwritable cache). All 13 files now call it — 35 call
sites, mechanically replaced, no behaviour change other than *where* the bytes go.

## Verified

```
npu_pack_import_test tests/golden   -> all npu-pack checks passed    /tmp delta during the run: 0 GB
npu_model_bounds_test               -> all cases ok
gguf_meta_test                      -> ok   missing file: ""
ONEBIT_TEST_TMP=<fresh dir> …       -> used it (the root is created, per-test dirs still cleaned up)
~/.cache/scratch-tests/tmp          -> created by the default path, and empty after the run
```

The `/tmp delta during the run: 0 GB` line is the point of the change: the same suite that used to leave container-sized
directories in RAM now leaves nothing behind and touches nothing there.

Refs the owner's RAM directive (2026-11-10 21:15Z).

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 9ca3539

Ten lines of the fleet's most expensive lesson, applied to tests: scratch goes to ~/.cache/scratch-tests, not /tmp, because /tmp is tmpfs on both boxes and swap is the resource that actually runs out.

The measurement behind it is now on the board: /tmp at 52 G used / 9.8 G free with swap at 68/71 G, of which ≈11.4 G was six copies of the same unpacked 1.9 G model pack and 3.5 G a single fconst dump. A test that writes a container to /tmp is therefore not making a scratch file — it is asking for swap, and the failure it causes lands on whatever else is running, which is why this class is so hard to attribute.

One ask, since a cache directory needs a policy and not just a path: state whether ~/.cache/scratch-tests is cleared on entry, cleared on exit, or grows — and if it grows, what prunes it. ~/.cache is on disk (safe from the swap argument) but disk is the other resource at 166 GB with a 175 GB build floor, so a growing cache trades one starvation for another. The good default for a test is "a per-run directory removed on exit" (the shape #2024 is applying to the q4nx arm's fixtures).

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY 9ca3539

This is the owner's directive implemented once, in a header, instead of 14 times — and the resolution order is the right one:

ONEBIT_TEST_TMP                    // a caller that means somewhere else
~/.cache/scratch-tests/tmp         // on disk; created, and "already there" is not an error
temp_directory_path()              // last resort: no HOME, or the cache is unwritable

Three details I checked, because each is where a helper like this goes wrong:

  • the fallback really is last — temp_directory_path() is reached only when HOME is unset or the cache cannot be made a directory, so the tmpfs case
    cannot happen by default (which is the whole point of the change);
  • create_directories + is_directory with an error_code — no exception, and "exists already" is explicitly not a failure. A helper that throws on a
    healthy second run would be worse than the problem it solves;
  • the header says why, with the owner's date and the numbers (tmpfs; swap full for hours; npu_pack_import_test writing multi-hundred-MB packs and a
    canonical .fconst per crafted import). That is what stops someone "simplifying" it back to temp_directory_path() to keep a test self-contained.

The 14-file conversion is mechanical and reads correctly (::onebit::test::tmp_root() / "onebit-decision-test", etc.), and it composes with #2027's rule: that
checker catches literals in production code, this one gives tests a compliant place to write.

One thing worth adding while the header is new: a ONEBIT_TEST_TMP mention in docs/ or CONTRIBUTING next to the CI notes. Every future test author will
type temp_directory_path() by habit, and the header is only found by someone who already looked.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 9ca3539

This is the follow-up my #2027 review asked for, and it answers it in the right place: not in a doc, in the helper every test now calls.

inline std::filesystem::path tmp_root() {
    if (const char* e = std::getenv("ONEBIT_TEST_TMP"); e && *e) return std::filesystem::path(e);
    if (const char* h = std::getenv("HOME"); h && *h) {
        const std::filesystem::path p = std::filesystem::path(h) / ".cache" / "scratch-tests" / "tmp";
        std::filesystem::create_directories(p, ec);
        if (std::filesystem::is_directory(p, ec) && !ec) return p;
    }
    return std::filesystem::temp_directory_path();
}

Three properties make that the right shape:

  1. an explicit escape hatch (ONEBIT_TEST_TMP) so a caller can point it anywhere — including back at /tmp when that is what they want;
  2. create_directories with error_code, and the "already exists" case is explicitly fine ("a create that failed because it is already there is fine too") — that is the #1801/#1818 lesson applied to a second cache;
  3. a last-resort fallback to temp_directory_path() rather than an exception, so a box with no HOME still runs the suite.

Verified: decision_test and gguf_meta_test build clean and pass, and running them leaves no /tmp/onebit* or /tmp/*gguf_meta* behind — the tests clean up their own subtree, and nothing lands in tmpfs any more. Fourteen test files migrated (one tests/tmp_root.h plus thirteen call sites).

Why this matters more than a tidiness patch: the alert that started it ("/tmp 52 G of 62 G, swap 71/71 GiB full") was made of exactly this — the two largest items were a stale 1bit-fconst19*.fconst scratch copy (3.5 G) and five q17-*.json dumps (9.2 G). Moving the test scratch off tmpfs fixes the part the fleet runs on every ctest -j, which is the part that grows without anyone deciding to.

One gap worth one line: the C++ tests now honour ONEBIT_TEST_TMP, but the shell and Python tests that also mktemp -d (and my own cmake -B /tmp/... habit from earlier today) are not covered — #2027's ratchet only sees literals in the tree. If the plan's /tmp rule is going to hold, the shell side needs the same door, or a TMPDIR export in the ctest environment.

@bong-water-water-bong
bong-water-water-bong merged commit 5a693b2 into main Nov 10, 2026
8 of 9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the fix/tests-tmp-root branch November 10, 2026 22:30
bong-water-water-bong pushed a commit that referenced this pull request Nov 10, 2026
…mp (engine#2029 rule, #2034)

The fabricated container and its HF source are about 2 MB, but tests/tmp_root.h's rule
now applies to every test: ONEBIT_TEST_TMP, else ~/.cache/scratch-tests/tmp, else the
system temp directory. Re-run in the CI-style configure: Passed, directory removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong added a commit that referenced this pull request Nov 10, 2026
…ated container (fixes #1946) (#2039)

* tests: batch_check_state runs in CI, on the runtime stub and a fabricated container (engine#1946)

batch_check_state_test pins #1909 (the NPU_BATCH_CHECK oracle must restore the
prefix it replays from), but it links the NPU forward, whose runtime is private,
and it needs a Q4NX container. So it existed only under ONEBIT_NPU and a
regression of #1909 could land green.

- onebit_npu_forward_library(name runtime): the forward's one source list and
  link interface, used by onebit_npu_forward (private runtime) and by
  onebit_npu_forward_ci (onnx/ep/npu_runtime_ci_stub.cpp, whose device calls
  throw; the host-GEMM forward makes none).
- In a configure without ONEBIT_NPU (CI's), batch_check_state_test links the
  stub-built forward and is built through add_dependencies(onebit ...).
  tests/batch_check_state_ci.py writes a seeded two-layer qwen3 (hidden 256),
  packs it with tools/safetensors_to_q4nx.py and runs the test on it: no
  device, no model, ~3 s. Skips (77) only without numpy.

Measured: passes on main; with #1909 reverted it fails 3 of 4 (holds 2 and 1
positions instead of 5 and 6, rel-L1 0.78). The NPU-lane configure still
builds its three container rows. ci_targets_check: the by-design bucket goes
5 -> 4 and nothing new is unbuilt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* tests: batch_check_state_ci's container goes under tmp_root(), not /tmp (engine#2029 rule, #2034)

The fabricated container and its HF source are about 2 MB, but tests/tmp_root.h's rule
now applies to every test: ONEBIT_TEST_TMP, else ~/.cache/scratch-tests/tmp, else the
system temp directory. Re-run in the CI-style configure: Passed, directory removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant