perf(cache): bound the response cache by bytes rather than entry count

[henry40408]

Third and last of the changes prompted by Cloudflare's DNS cache memory write-up, after #252 and #254. Not one of its five techniques — this is its premise: per-entry footprint is a quantity you manage, and you cannot manage it if the bound is in the wrong unit.

The problem

max_capacity(10_000) counted entries. A DNS response spans tens of bytes to tens of kilobytes, so the memory a full cache occupied was decided by whatever traffic it happened to see. A run of large DNSSEC or TXT answers grew it without limit — the wrong thing to leave open on the small hosts noadd targets, and the same concern that already drove the mimalloc choice.

The change

An entry is charged its response, its domain, its offsets table, and ENTRY_OVERHEAD_BYTES for the part that does not vary — moka's node, the Arc header and the CacheKey struct:

fn entry_weight(key: &CacheKey, value: &CacheValue) -> u32 {
    let variable = key.domain.len() + value.bytes().len() + value.ttl_offsets().len() * 4;
    ENTRY_OVERHEAD_BYTES.saturating_add(u32::try_from(variable).unwrap_or(u32::MAX))
}

That constant is measured, not estimated. cache_memory_bench gained a line that reports it, and it currently reads 364.6 against the 365 in the code — so a change to the entry's shape shows up as the constant drifting rather than as a cache quietly holding more than it was told to.

The default is 5 MiB, which at the ~499 bytes an entry measures on ordinary traffic holds roughly the 10,000 entries the old bound allowed. This is a ceiling, not a smaller cache.

new → with_capacity_bytes

The argument changed meaning, so keeping the name would have left all 33 call sites compiling while saying something else. Renaming makes the compiler find every one. Test call sites take a deliberately roomy bound — none of them ever meant to exercise eviction, and I checked that none depends on it.

Tests

Two in cache_test.rs, both comparing how many of N entries survive a fixed cap at two response sizes — asserting the property, not a number, since exactly which entries moka keeps is its business:

• capacity_is_measured_in_bytes_not_entries — 60 tiny responses all survive a 100 KB cap; 60 × 8 KB responses do not.
• a_large_response_displaces_more_than_a_small_one — a 20-byte response is worth several 2 KB ones.

Both observed failing without the weigher (60 of 60 survived, 400 small against 400 large) and passing with it.

Throughput

Unchanged, as expected — the weigher runs on insert and eviction, never on a hit: 4.23M, 4.24M, 4.35M qps over three cache_hit_bench runs, inside the 4.08–4.97M spread measured on the parent commit.

Not done here

No CLI flag or runtime setting for the figure; it stays a constant in main.rs, as the entry count was. Exposing it is a reasonable follow-up but a separate one — the ceiling existing at all is the change here.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.92%. Comparing base (d2769a8) to head (9c7c345).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #255   +/-   ##
=======================================
  Coverage   90.92%   90.92%           
=======================================
  Files          30       30           
  Lines       10763    10768    +5     
=======================================
+ Hits         9786     9791    +5     
  Misses        977      977           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.