tierstore

A generic storage-tier router: register N tiers (anything that can get/put/exists/list), and route reads and writes through them with explicit, pluggable policy. The first-class instantiation is a tiered rollover cache — hot (memory) over warm (disk) over cold (remote fetch, e.g. a database) — but the router doesn't know it's a cache; that's just one policy.

Layout

tierstore-core   no_std. Capability traits (TierRead / TierWrite / TierList /
                 TierReadRef), routing Policy, and ReadFlow — a sans-io state
                 machine that makes every read-path decision without I/O.
tierstore        std batteries, zero deps. Router (drives ReadFlow against
                 real tiers), MemoryTier (hot; entry- or byte-bounded),
                 DiskTier (warm), middleware tiers (VerifiedTier,
                 LimitedTier), SingleFlight, and two semantic presets built
                 entirely on the router: TieredCache (availability) and
                 TieredStore (authority).
tierstore-mmap   Adapter: MmapDiskTier, a kernel-evictable zero-copy warm
                 tier (file-per-key, served as mmap-backed bytes::Bytes,
                 snapshot-on-overwrite).
tierstore-moka   Adapter: MokaTier, a sharded TinyLFU hot tier for
                 high-throughput inclusive hierarchies (moka evicts
                 internally, so it reports no displacement — use MemoryTier
                 for exclusive rollover stacks).
                 Together these are the template for backend adapters
                 (redis, s3, postgres, …).

Cross-node invalidation deliberately lives elsewhere: it is a coordination
concern, not a tier. Pair a multi-node cache with groupnet's
`groupnet-consistency` crate (per-writer sequenced write feeds, Wrote/Gap
events, Frontier read-your-writes barriers) — the apply loop is
`cache.invalidate(&key)` per Wrote and a coarse flush per Gap.

Mechanism and policy are separated on purpose: the router executes, Policy decides. The same router is an inclusive read-through cache, an exclusive rollover cache, or a plain fallback chain depending only on policy.

Quick start

let cache = TieredCache::builder()
    .tier(Arc::clone(&hot))     // MemoryTier, bounded
    .tier(Arc::clone(&warm))    // DiskTier
    .tier(Arc::clone(&cold))    // your remote store
    .build();

let value = cache.get(&key).await?;   // read-through, promotes per policy

cargo run -p tierstore --example hot_warm_cold walks the whole story: remote fetch → hot hit → hot overflow rolls entries onto disk → disk hit without re-fetching.

cargo run -p rollover-demo is the assertion-gated proof on a real tier stack — hot is a fixed-slot memory-mapped file (memmap2), warm is disk, cold is a simulated remote with latency — and additionally proves the mmap hot tier survives a process restart (index rebuilt from the mapping, zero remote fetches). The MmapTier in that crate is the candidate to graduate into a real tier once we're happy with it.

Semantics worth knowing

• Rollover. TierWrite::put returns the entries it displaced; the router demotes them into the next tier down (cascading). Entries falling off the bottommost tier are returned to the caller — eviction is never silent.
• Batching is first-class. get_many / put_many / delete_many ship looping defaults so every tier supports them; backends override with real batch I/O (MemoryTier does one lock pass; a remote tier would use an MGET-style round-trip). The router probes each lower tier with only the still-missing keys.
• Partial success has per-key statuses. Batched router reads (read_many) and deletes (remove_many) return reports — each key is a Hit (with the tier that served it), a confirmed Miss, or Inconclusive when a failing tier left it unknown — with the tier failures alongside, so one bad key or tier never discards resolved values. The cache's get_many / invalidate_many return these reports.
• Honest misses. If a read falls through a failing tier and ends in a miss, you get RouterError::Inconclusive, not Ok(None) — the failed tier might have held the key.
• Deletes are all-tier. A delete attempts every tier even after failures, and partial failure is an error: a surviving upper copy would resurrect the key.
• Write-around invalidates. Writing only the bottom tier deletes the key from upper tiers, since a stale copy would shadow the new value.
• Routers compose — and so do stores. Router and TieredStore both implement the tier traits, so a "warm" tier can be a whole hierarchy, and the blessed layering for authority-backed systems is cache-over-store: TieredCache [ hot, warm, TieredStore [ … ] ]. Cache tiers stay lenient above; reads reaching the store are governed by its own fail-fast policy inside; writes hit the authority first.
• Store vs cache is policy, plus a stance on loss. Policy::default() is the neutral fallback chain (the router moves nothing on its own). TieredCache is the availability preset. TieredStore is the authority preset: fail-fast reads, strict deletes, and a write that pushes entries off the bottom tier returns them in StoreError::Evicted — data loss is an error carrying the data, never a silent shrink.
• Stampede protection is built in. TieredCache coalesces concurrent gets per key (single-flight: one caller fills, the rest wait and hit the promoted copy), dependency-free and executor-agnostic. Default on; .single_flight(false) opts out.
• Trust boundaries are explicit. Wrap an untrusted tier in VerifiedTier and every value it serves is checked once at the boundary; a rejected value is a tier failure (inconclusive read), never data, and is never promoted upward. Both patterns are adopted from shardstore's cache layer.
• Zero-copy reads are a capability. TierReadRef::get_ref returns a guard-held view that derefs to the value in place (the demo's MmapTier serves views pointing directly into the mapping). Direct-tier only: views can't cross the router's type-erased boundary, and promotion/demotion inherently copy (see open question 12).

Memory story

• Bound the hot set: MemoryTier::bounded (entry count) or MemoryTier::bounded_bytes(budget, weigher) (byte budget), with FIFO or LRU ordering (.with_eviction(Eviction::Lru) — O(1) touches via stamped tickets). Overflow rolls down; an entry heavier than the whole budget rolls straight through to the next tier instead of thrashing the hot set.
• Bound transit: values move between tiers as owned V clones, so pick a cheap-clone V for large values — bytes::Bytes turns every boundary clone into a refcount bump, and the router is already generic over it.
• Bound residency: tierstore-mmap's MmapDiskTier serves values as mmap-backed Bytes: zero-copy reads, snapshot-on-overwrite, and RAM residency managed by the kernel's page cache (evictable under pressure) — shardstore's warm-tier model.
• Keep useful warm data: bounded mmap tiers default to FIFO for compatibility and accept .with_eviction(Eviction::Lru) for access-aware retention. MmapDiskTier::stats() reports entries, mapped entries, disk bytes, budget, and lifetime eviction totals without touching mapped pages.
• Bound concurrency: wrap an origin in LimitedTier to cap in-flight operations against it; transient fill memory becomes ~limit × value size instead of callers × value size. Single-flight already dedupes same-key fills; this bounds distinct-key fan-in.
• Keep blocking I/O off the executor: wrap file-backed tiers in OffloadTier — a dependency-free, executor-agnostic worker pool that runs the inner tier's operations on its own threads.
• Watch it run: Router::stats() (also on the cache and store) reports per-tier hits, misses, errors, puts, and deletes.

The docres shape (node-local layered storage)

The target deployment: a node that caches an authoritative remote store locally — a docres/shardstore-style artifact node.

let cache = TieredCache::builder()
    .tier(MemoryTier::bounded_bytes(ram_budget, |_, v: &Bytes| v.len()))
    .tier(MmapDiskTier::open_bounded(cache_dir, disk_budget)?)
    .read_only_tier(LimitedTier::new(
        VerifiedTier::new(object_store_tier, checksum),
        max_inflight,
    ))
    .build();

• The origin is read_only_tier — only TierRead is required (no stub writes), the cache never writes or deletes through it, puts fill the cache layers, and invalidation clears local copies while the origin re-serves the key afterwards (resurrection by design for data you don't own).
• Disk is bounded: MmapDiskTier::open_bounded evicts over its byte budget (FIFO by default, or LRU with .with_eviction(Eviction::Lru)), rebuilds accounting from file sizes/mtimes on restart, and serves displaced values zero-copy from the evicted files themselves.
• Partial reads: TierReadRange::read_range serves byte ranges without materialising whole values (a positional read on DiskTier, a refcounted mapping slice on MmapDiskTier). Chunk-granular faulting à la shardstore is the same router with chunk keys ((artifact, chunk_no)) — promotion, rollover, and single-flight compose with it for free.
• Wrap the file tiers in OffloadTier to keep their blocking I/O off the async executor, and read hit ratios off cache.stats(). Background refresh remains deliberately out of scope.

Open questions (deliberately unresolved)

1. Send futures are part of the trait contract (server-first). Is a non-Send "local" variant worth the surface?
2. Read-only tiers are first-class (read_only_tier, requiring only TierRead); finer-grained capability splits (delete-but-not-put, promote-into-but-not-demote-into) are open.
3. TierList for the router itself (cross-tier cursor unification, dedup).
4. Write-back mode (dirty tracking + flush) — v2 at the earliest.
5. Single-flight granularity: get_or_load ships probe-then-gate (hits never touch the gate) for stacks whose origin lives in the loader; plain get still gates whole reads because an in-stack cold tier would be stampeded by ungated probes. Deadlock-free batched coalescing is open.
6. Per-tier TTL / staleness, negative caching.
7. Demotion churn when a lower tier is smaller than the one above it.
8. Typed-over-bytes is shipped (CodecTier: one-way key mapping, value codec embedding the original key so displacement decodes back typed); per-entry encode-failure granularity in batches is open.
9. MemoryTier offers FIFO and LRU ordering; LFU, segmented/scan-resistant policies, or a fully pluggable ordering trait are open.
10. Static (generic tuple) tier composition to avoid boxing on the hot path.
11. The batched trait methods stay lowest-common-denominator (Result<Vec<…>, _>), so a nested router driven through the tier traits degrades its per-key statuses to whole-batch errors; richer trait-level batch contracts and a sans-io batched read flow are open questions.
12. Zero-copy through the router: TierReadRef views are direct-tier (they hold the tier's lock). The refcounted-value path is shipped (tierstore-mmap + V = Bytes make boundary clones free); router-level borrowed views (boxed guards, top-tier fast path) remain open.
13. Richer observability: per-tier counters are in (Router::stats); latency histograms, eviction counters inside tiers, and tracing hooks are open.

Prior art

foyer is the mature memory+disk hybrid cache in Rust — if you want a fast two-tier cache product, use it. tierstore is the abstraction instead: N pluggable tiers behind small traits, policy separated from mechanism, no_std core, composable routers.

Toolchain

Rust edition 2024, tracking stable (no pinned MSRV). tierstore-core and tierstore are dependency-free (dev-deps included) and unsafe-free (forbid); tierstore-mmap carries the one documented unsafe block that memory mapping requires, plus memmap2 and bytes. CI gates: rustfmt, clippy pedantic + nursery at -D warnings, all tests including doctests, a no-alloc core build, docs with -D warnings, and the assertion-gated rollover demo.

Status

Early but real: the mechanism, both semantic presets, and the memory story are implemented and tested end to end. Version 0.1.x — expect API movement along the open questions above.

License

MIT. See LICENSE.