Skip to content

Two log2 calls on the encode path can be replaced by exact integer arithmetic #5

Description

@ztrencsenyi

Both of these compute something integer arithmetic gives exactly. The replacements are exact where the current code rounds, both are faster, and neither needs a feature flag.

encode.rs:3779 — Rice partition parameter

let bits_needed = ((partition_sum as f64) / f64::from(partition_samples))
    .log2()
    .ceil() as u32;

The branch is guarded by partition_sum > partition_samples, so the argument is always > 1. The quantity is "how many bits does this ratio need", which is exactly the smallest k such that partition_samples << k >= partition_sum — computable without a float:

let bits_needed = {
    let samples = u64::from(partition_samples);
    let ratio = partition_sum.div_ceil(samples);
    u64::BITS - (ratio - 1).leading_zeros()
};

64 - leading_zeros(r - 1) is the bit width of r - 1, which is floor(log2(r - 1)) + 1, which is ceil(log2 r) for every r >= 1. Boundaries: 2 → 1, 3 → 2, 4 → 2, 5 → 3, 2⁶³ → 63, 2⁶³+1 → 64, 2⁶⁴−1 → 64. Maximum return is 64, so nothing overflows.

One spelling worth avoiding, because it is the one you would naturally reach for: ratio.next_power_of_two().trailing_zeros() computes the same value and reads more obviously, but next_power_of_two needs 2⁶⁴ once ratio > 2⁶³ and overflows there. Not reachable on real material — the largest partition_sum we observed was 4,744,214,280 — but the leading_zeros form has no such boundary and costs nothing extra.

The current form rounds twice, once in the f64 division and once in log2; the integer form rounds not at all.

Measured: over 2,198,163 distinct (partition_sum, partition_samples) pairs collected from real encodes (44.1 kHz/16-bit through 352.8 kHz/24-bit, several minutes of audio), the two forms agree everywhere. So this is a correctness-and-speed improvement that does not move the bitstream on ordinary material, rather than a behaviour change dressed up as a cleanup. The largest partition_sum observed is comfortably inside 2⁵³, where the float division is still exact — but nothing prevents that being exceeded on longer or louder input, and past it the float form starts losing bits.

encode.rs:3360 — LPC shift

match ((u32::from(precision) - 1) as i32 - ((l.log2().floor()) as i32) - 1).min(MAX_SHIFT) {

l is > 0.0 (filtered on the preceding lines), so l.log2().floor() as i32 is the unbiased binary exponent — which an f64 already carries in its bit pattern. Two ways to take it out, depending on whether you want the dependency:

With libm (what we ship), one line and no commentary needed:

let exponent = libm::ilogb(l);

Without a new dependency, by hand — and the subnormal case is why this is not a one-liner:

// floor(log2(l)) for a strictly positive f64, exactly.
let exponent = {
    let bits = l.to_bits();
    let raw = ((bits >> 52) & 0x7FF) as i32;
    if raw == 0 {
        // Subnormal: no implicit leading 1, so the exponent comes from the
        // position of the highest set mantissa bit.
        -1022 - (52 - (63 - (bits & ((1u64 << 52) - 1)).leading_zeros() as i32))
    } else {
        raw - 1023
    }
};

The subnormal trap: the naive ((bits >> 52) & 0x7FF) as i32 - 1023 returns −1023 for a subnormal, where the true floor(log2(l)) goes as low as −1074. Here .min(MAX_SHIFT) clamps both answers to MAX_SHIFT so the difference happens to be invisible, but the extraction is worth writing correctly rather than resting on a clamp two tokens away.

Measured: we ran the hand-rolled form above against libm::ilogb, and both against the defining property 2^e <= l < 2^(e+1), over 2,006,294 arguments — every exact power of two in -1074..=1023, both neighbours of each, and 2,000,000 pseudo-random positive finite bit patterns. Zero disagreements and zero violations, so either form is safe to take.

Why this matters to us specifically

We build the same audio on Linux and Windows and require byte-identical output. log2 comes from the host C library, which no standard requires to be correctly rounded and which is not the same code on the two platforms — so any log2 on the encode path is a place two builds may disagree. Removing these two is better than making them portable: it closes the seam and makes the result exact.

We have verified that end to end. With these two log2 calls removed and the remaining transcendentals taken from libm rather than the host, our encoder now produces byte-identical FLAC on Linux (glibc 2.39) and Windows (UCRT 10.0.26100.8875) — pinned by a test suite that checks the encoded digests on both platforms, with the input signal's digest frozen separately so a platform-dependent generator cannot be mistaken for a portable encoder.

Happy to open a PR for either or both of these if that is easier than patching them yourself.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions