RANSKit

RANSKit is a prepared indexed-CDF rANS implementation and a small research toolkit for learned entropy coding. It accepts integer symbols, CDF indexes, quantized CDF rows, row lengths, and offsets without a model-framework dependency.

A model captures its entropy-coder calls once, then the toolkit inspects, verifies, and benchmarks implementations without loading the model or dataset.

Setup

RANSKit requires 64-bit Python 3.11 or newer, NumPy, CMake, and a C++17 compiler targeting a 64-bit architecture. CMake rejects older Python versions and other target widths.

PYTHON=python3.11 ./setup.sh
../.ranskit-venv/bin/python -m ranskit check

A regular install can also be created directly.

python3.11 -m pip install .
ranskit check

CompressAI is optional. It is imported only when check --oracle compressai or a compressai backend is requested.

API

The prepared API builds the table once and keeps conversion outside repeated coder calls.

import numpy as np

from ranskit import CdfTable, IndexedRansCoder

cdfs = np.array([[0, 32768, 65536]], dtype=np.int32)
lengths = np.array([3], dtype=np.int32)
offsets = np.array([0], dtype=np.int32)
symbols = np.array([0, 0, 0, 0], dtype=np.int32)
indexes = np.zeros(symbols.size, dtype=np.int32)

table = CdfTable(cdfs, lengths, offsets)
coder = IndexedRansCoder(table)
payload = coder.encode(symbols, indexes)
decoded = coder.decode(payload, indexes)
assert np.array_equal(decoded, symbols)

BufferedEncoder and StreamDecoder support ordered multi-call streams. RansEncoder, BufferedRansEncoder, and RansDecoder provide list-compatible entry points for existing indexed-rANS integrations.

Existing model call sites can replace the entropy-coder import directly.

from ranskit import BufferedRansEncoder, RansDecoder, RansEncoder

payload = RansEncoder().encode_with_indexes(
    symbols, indexes, cdfs, cdf_lengths, offsets
)
decoded = RansDecoder().decode_with_indexes(
    payload, indexes, cdfs, cdf_lengths, offsets
)

encoder = BufferedRansEncoder()
encoder.encode_with_indexes(
    first_symbols, first_indexes, first_cdfs, first_lengths, first_offsets
)
encoder.encode_with_indexes(
    second_symbols, second_indexes, second_cdfs, second_lengths, second_offsets
)
payload = encoder.flush()

decoder = RansDecoder()
decoder.set_stream(payload)
first = decoder.decode_stream(first_indexes, first_cdfs, first_lengths, first_offsets)
second = decoder.decode_stream(
    second_indexes, second_cdfs, second_lengths, second_offsets
)

No CompressAI import is required. Model-specific wiring stays in the model repository.

Inputs to the prepared API are C-contiguous native int32 arrays. Each CDF row starts at zero, is strictly increasing through its declared length, ends at 65536 for 16-bit precision, and has at least three entries because the terminal interval is the sentinel. The built-in RANSKit and CompressAI backends require 16-bit CDF precision. External trace backends may support other declared precisions.

Model call patterns

A hyperprior model normally prepares one static table for the hyper-latent and one indexed conditional table for the main latent. Each application sample then emits an ordered sequence of coder calls.

A two-pass checkerboard codec commonly emits:

1. hyper-latent
2. anchor latent
3. non-anchor latent

Capture the exact flattened symbol and index order passed to the coder. Keep the three calls inside one trace sample so deployment latency is measured over the original boundary. Table identifiers are content hashes, so calls sharing a table deduplicate automatically.

from ranskit import TraceWriter

with TraceWriter("capture.eect", provenance=provenance) as trace:
    y_table = trace.add_table(y_cdfs, y_lengths, y_offsets, precision_bits=16)
    z_table = trace.add_table(z_cdfs, z_lengths, z_offsets, precision_bits=16)
    with trace.sample("tile-0000", denominators={"pixels": 65536}) as sample:
        sample.add_stream(z_table, z_symbols, z_indexes, role="hyper")
        sample.add_stream(y_table, anchor_symbols, anchor_indexes, role="anchor")
        sample.add_stream(
            y_table,
            non_anchor_symbols,
            non_anchor_indexes,
            role="non_anchor",
        )

Model loading, dataset selection, and framework hooks stay in the model repository. TraceWriter is the only integration surface RANSKit requires.

Trace

A trace is a deterministic directory:

capture.eect/
  manifest.json
  arrays.i32le
  streams.bin

The integer blob stores trimmed CDF rows, lengths, offsets, symbols, and indexes as signed little-endian int32. The stream blob stores optional reference bitstreams. The manifest records byte offsets, content-addressed tables, sample and stream order, format identifiers, user-supplied provenance, and SHA-256 hashes. The writer rejects absolute path strings and non-finite metadata values. Trace producers control the remaining provenance fields.

ranskit inspect fixtures/cloudaware-g3-mini-v1.eect
ranskit inspect fixtures/cloudaware-g3-mini-v1.eect --json

The committed fixtures are:

• format-v1.eect, a small synthetic sentinel, escape, bypass, offset, and multi-table-index case
• cloudaware-g3-mini-v1.eect, eight trained-model call patterns selected from the CloudAware G3 capture with shared y/z tables and embedded reference bytes

The trained fixture is a regression and deployment-shape sample. Population or rate-distortion claims require a full preregistered trace.

Verify

Default verification requires only NumPy and ranskit._core.

ranskit check
ranskit verify fixtures/cloudaware-g3-mini-v1.eect
ranskit check --oracle compressai

Verification runs each backend round trip. When reference bytes and a matching format identifier exist, it also requires byte equality and reference decode. Two backends declaring the same format must emit equal bytes and cross-decode. Different formats receive round-trip and real-byte-count checks without a false byte-equality requirement.

A local experiment backend is selected without changing the runner.

ranskit verify TRACE \
  --backend baseline=ranskit \
  --backend candidate=my_experiment:open_backend \
  --backend-arg candidate:variant=\"candidate-a\"

The factory receives a JSON-compatible options mapping. It returns an object with info, prepare(table), and optional synchronize(). A prepared table object provides encode(symbols, indexes) and decode(payload, indexes).

Benchmark

Benchmarking verifies every backend first, prepares tables and decode payloads outside timing, preserves sample and stream boundaries, gives each trace sample the requested repetition count, and alternates backend order.

ranskit benchmark fixtures/cloudaware-g3-mini-v1.eect \
  --backend baseline=ranskit \
  --operations encode,decode \
  --warmup 20 \
  --samples 200 \
  --out results/baseline.json

Use --require-cov 0.05 when a coefficient-of-variation noise gate is part of the protocol. The gate uses the maximum CoV across within-sample repetition groups, requires at least two repetitions per sample, and runs before a successful result is written. CPU affinity, governor, power mode, and clock locking remain caller-controlled. taskset, device controls, and the exact invocation belong in the experiment record.

./run.sh
./sweep.sh

Results

Benchmark JSON stores the trace and manifest hashes, backend identity and configuration, native artifact hash, exact protocol, raw per-sample latency, symbols, real encoded bytes, median, p90, p99, mean, standard deviation, CoV, per-sample CoV, ns/symbol, Msymbol/s, architecture, affinity, Python version, allowlisted thread settings, and the noise-gate threshold and status. Result files are bounded at 256 MiB and must be regular, non-symbolic-link files.

ranskit compare results/baseline.json results/candidate.json

Comparison validates both result files and rejects different fixture content, operations, protocols, or environments. Real encoded bytes are the rate result. Inspection diagnostics do not substitute for real byte counts. Results containing multiple backends require --backend-left and --backend-right.

Scope

RANSKit covers indexed integer-CDF entropy-coder execution, portable traces, correctness comparison, and sample-scoped benchmarks. Model execution and trace capture remain in the model repository. Fixture results apply only to the captured calls. Codec-level claims require full-dataset rate-distortion evaluation at matched reconstruction quality.

Layout

cpp/                    native coder and bindings
src/ranskit/             Python API, trace format, verification, and CLI
fixtures/               concise format and trained-model traces
examples/               programmatic trace capture
setup.sh                sibling environment setup
run.sh                  default verification and benchmark
sweep.sh                bounded fixture/sample-count sweep