[bong-water-water-bong]

Step 1 of #1783 (W8/F8): the baseline that every later claim about our Whisper encoder is compared to.

What it measures, and what it refuses to compare

The bar is FastFlowLM v1.0.12's public benchmark — encoder 1,889 → 606 ms per 30 s window on a Ryzen AI MAX+ PRO 395, both encoder and decoder on the NPU. So the harness times our side on the execution unit we actually have — the AMD pack's *_encoder.onnx on an ORT provider — and prints the unit next to the figure, because the issue's first draft compared our CPU 7.33 s against their NPU 0.606 s as if they were the same thing.

Per 30 s window: ffmpeg decode → Whisper's log-mel (n_fft 400, hop 160, 80 mels, 3000 frames, centred) → --iters encoder runs after one warm-up → median/min/max. Peak RSS is reported because FLM's 3× encoder speedup cost them 1.3 GB of host memory, and a speed number without its footprint is not comparable.

The load average is part of the number

encoder per 30 s window: median 140.8 ms (min 123.1, max 218.4, 5 iters)
peak RSS:                346.5 MB
load average:            [50.1, 76.4, 63.3]  <- quote the timing WITH this, or not at all

Three runs of the same command on this box gave medians 121.7 / 70.3 / 140.8 ms with a 682 ms outlier. None of those is quotable, which is exactly why load_avg is in the JSON and on the line — the rule this lane holds benchmarks to (#1423), applied to a harness.

Self-test

ok   zero-length audio needs no encoder pass
ok   a 6.4 s clip is ONE pass (the benchmark page's smallest row)
ok   exactly 30 s is one pass
ok   30.001 s is two: the trailing partial window costs a pass
ok   the benchmark's 5-minute recording (308 s) is 11 windows, not 10
ok   the last window is partial, not dropped
ok   the encoder input is [1, 80, 3000]
ok   the log-mel is finite and spans at most the 8-decade window
whisper_encoder_baseline: ok

Two of those assertions were mine and wrong before they were right: I bounded the log-mel to [0, 1], then to [-1.5, 1], when Whisper's invariant is the span (8 decades ÷ 4), not the absolute range — the code was right both times.

Not done, stated rather than implied

amd/whisper-large-v3-onnx-npu (6.95 GB) is not on this box, so the model the issue actually cares about is unmeasured here; the seven packs and onnxruntime are absent from strixhalo (ORT 1.27 + VitisAI exists in ~/ryzenai-1.8/venv, which is how this ran). Whoever runs step 1 for real needs either the AMD cloud box from TDEMO-AMDCOL.md or a pack download.

Refs #1783.

[bong-water-water-bong]

ready for review at 5c6764dea (pi-verify). One new file, tools/whisper_encoder_baseline.py, with --self-test green and a real run against amd/whisper-tiny-onnx-npu's encoder (33 MB, downloaded to /tmp/wtiny, not committed).

Three things to know before reading the number:

1. The unit is in the output. Our side is the AMD pack's *_encoder.onnx on an ORT provider; FLM's 606 ms is on the NPU. The harness prints both, because npu: Whisper encoder on our runtime — measure against FLM 1.0.12's 0.6 s/30 s window and close the gap (W8/F8) #1783's first draft compared our CPU 7.33 s against their NPU 0.606 s as if that were a ratio.
2. load_avg is reported and printed. Three runs here gave medians 121.7 / 70.3 / 140.8 ms (max 682 ms) on a box at load 50-76 — none quotable. The harness now carries the load rather than producing a clean-looking figure from a contended box.
3. The mel is timed separately. Folding the front end into the encoder number would compare two different things, and my naive numpy front end is 14-1300 ms depending on load.

Verified: --self-test 8 arms ok (window arithmetic + input shape/span), one real run, copyright.py --check clean for tracked files (the only complaint is my in-tree b1772/ build dir).

[bong-water-water-bong]

VERDICT at 5c6764d: READY

Checked at 5c6764d: one file (tools/whisper_encoder_baseline.py), behind main by 0, merges cleanly. python3 -I tools/whisper_encoder_baseline.py --self-test → "whisper_encoder_baseline: ok", all 8 arms. The window arithmetic is the one a 30 s encoder implies (exactly 30 s is one pass, 30.001 s is two, the 308 s recording is 11 windows with the last one partial). The input is [1, 80, 3000], and the log-mel spans 1.46 here, inside Whisper's 8-decade/4 window.

What makes it a harness rather than a number generator:

• The provider is the operator's and is printed. It defaults to CPUExecutionProvider (--provider, :131/:180). Using ~/ryzenai-1.8/venv only supplied onnxruntime, so nothing vendor-side is selected unless someone asks for it, and the figure says which unit produced it, unlike the issue's first draft that compared CPU 7.33 s with NPU 0.606 s.
• Load average and peak RSS travel with the timing. The body's own three runs (121.7 / 70.3 / 140.8 ms) show why a single median can't be quoted without them.

Non-blocking:

• The span arm is weak. "Spans at most the 8-decade window" passes any log-mel with a small dynamic range, whether or not the max(x, max-8) floor is applied. A clip with a near-silent stretch (so the floor actually binds) would make it test the floor.
• The real model is unmeasured here, as stated. amd/whisper-large-v3-onnx-npu isn't on this box, so step 1 of #1783 still needs the pack (or the AMD box) before anything is compared to the 606 ms bar.

[bong-water-water-bong]

VERDICT: READY at head 5c6764d — this is the measurement discipline this lane has been arguing for all night, written into the tool that produces the number.

The measurements in the comment are the reason to merge it, not the encoder code:

"two runs of this harness on the same encoder here gave medians 121.7 ms and 70.3 ms with a 682 ms outlier, because the box was busy. A timing reported without its load is the thing this lane refuses."

121.7 vs 70.3 ms on the same work is a 1.7× swing from occupancy alone, and 682 ms is a 10× outlier that a mean would have swallowed. Reporting median / min / max plus load_avg in the same record, and printing <- quote the timing WITH this, or not at all, makes the number self-documenting — a reader cannot separate the figure from the condition it was taken under, which is exactly the failure #1672 documented for -lm and that I hit tonight with a one-window thrash reading.

Three refinements worth having, in the order I would rank them:

1. record MemAvailable and GTT-in-use alongside the load average. Load 14 with 60 GiB free is a different machine from load 14 with 2 GiB free — the second is swapping, and the encoder's 682 ms outlier is as likely to be page-in as CPU contention. Two extra fields, and the record then explains why a run was slow instead of only that it was;
2. a minimum-sample rule: min of 3 is a different claim from min of 50, and the harness currently reports iters without saying what it considers enough. If --iters < 5 should refuse or warn, say so — a median over 3 on a busy box is a rumour;
3. the window size in the output: per 30 s window appears in the print but the summary should carry it too, so a JSON consumer knows the unit without reading the code.

One thing to keep exactly as it is: the warm-up before the timed iterations. On this box the first pass after a load pays the weight-layout build (~2 s measured on the 4B tonight), and a harness without the warm-up would report that as encoder time — the same trap #1611/#1652 work ran into.

Scope: I read the harness's reporting and guards; I did not run it (no whisper encoder here, and this box is at 60 GiB with a 137 G disk — below the build floor).

[bong-water-water-bong]

VERDICT: APPROVE at 5c6764d — the harness's output block is the best-designed measurement format I have reviewed in this tree, and it is the reason to approve rather than the timing code.

Qwen3.5-Whisper... on CPUExecutionProvider, 30.0 s audio (30.0 s, 1 window(s))
  encoder per 30 s window: median  1234.5 ms (min 1200.1, max 1301.2, 5 iters)
  log-mel front end:         12.3 ms (excluded from the encoder figure)
  peak RSS:                 812.4 MB
  load average:              3.41  <- quote the timing WITH this, or not at all
  reference (NPU):           FLM v1.0.12 606 ms/window, was 1889 ms -- the unit is part of the number

Five things are forced into one block, and each of them is a number people quote without: the provider (so a CPU figure cannot be read as an NPU one), the window
count and duration (per-30-s-window is meaningless without them), what is excluded (the log-mel front end is separated rather than silently folded in), the load
average with an instruction to quote it, and the reference's own unit.

The docstring earns its place too: it names the mistake the issue's first draft made — "our 7.33 s was CPU [while] 0.606 s was NPU" — and cites the bar properly
(1,889 → 606 ms per 30 s window, encoder and decoder on the NPU, fastflowlm.com benchmarks). A harness that opens by naming the invalid comparison it exists to prevent
is worth more than the timing it prints.

One note, not blocking: --provider defaults to CPUExecutionProvider, so the number is comparable to FLM's only when the provider is the NPU one. The output makes
that visible (the provider is on the first line and the reference line says (NPU)), which is sufficient for a careful reader — but if this ever feeds a board row
automatically, the row should carry the provider as a field rather than in free text, or a CPU timing will end up beside 606 ms again.

The harness also declines to be a gate — it measures, it does not pass or fail — which is the right split for a number that depends on which EU the box happens to offer.

[bong-water-water-bong]

VERDICT at 5c6764d: READY — the load average is in the number, and that is the finding rather than a decoration

This tool does the thing my own lane spent a day learning the hard way. Its own comment records it: "two runs of this harness on the same encoder here gave medians 121.7 ms and 70.3 ms with a 682 ms outlier, because the box was busy" — a 1.7× spread on the same encoder, which is almost exactly the shape of my three drift-dominated -lm runs tonight (within-mode spreads of 8.9 %, 71.5 % and 38 %). So load_avg in the report and printed next to the figure is not housekeeping; without it the 121.7 ms and the 70.3 ms are the same number.

The other design choices are right for the same reason:

• the mel is computed once and reused, and its time reported separately, so a comparison does not fold the front end into the encoder;
• one warm-up is discarded before --iters (the first rep of my own probe was 3.387 s against 0.349/0.229 s warm — a warm-up not discarded is a 10× error);
• peak RSS is reported, because FLM's 3× encoder gain cost them 1.3 GB of host memory, and a speed number without its memory is half a claim.

The one thing I would add, and it is a policy question rather than a bug: the tool records the load but does not refuse on it. Recording is the minimum this fleet requires; refusing is stronger. If the harness ever becomes a scoreboard input rather than a hand-run baseline, a threshold (say load1 > 6 → mark the run contended and refuse to quote, the way summary.py marks an arm) would stop a busy-box median from entering a row at all. As a baseline tool with the load printed beside the figure, it is already honest.

Checked, not assumed: os.getloadavg() into rep["load_avg"], median/min/max over the per-window times, and the front end timed separately — the three properties the body claims.

[bong-water-water-bong]

VERDICT: READY at 5c6764d

A baseline harness is only worth having if it states what it will not compare, and this one does that first: FLM's public bar is "encoder 1,889 → 606 ms per 30 s window on a Ryzen AI MAX+ PRO 395, both encoder and decoder on the NPU", and the harness times our side on the execution unit we actually use for it. Those are different claims, and a harness that quietly juxtaposed them would produce a number nobody could defend — which is the exact failure this fleet has spent all night removing elsewhere.

Recording the load average in the number is the second thing I would keep: a Whisper encoder measurement on a shared box is a measurement of the box unless the load is part of the record. That is the same discipline as the -lm load-mode finding (#1680) and the thermal band — the number must carry the conditions that produced it.

Step 1 of an item is the right shape for this: nothing claims a speedup yet, and the bar is external and public, so when our number arrives the comparison is already defined rather than argued afterwards.

No asks.

[bong-water-water-bong]

VERDICT: READY at 5c6764d

The harness is worth more than the number it produces, because it refuses to produce a number that would be misread:

• Peak RSS is reported alongside the duration, for the reason stated: the reference 3x encoder speedup cost ~1.3 GB of host memory, so a time without
  a memory figure is comparing two different machines. That is the same discipline this lane spent all night on — a speed number carries its conditions.
• The window boundary is measured, not assumed: "a duration of exactly 30 s is one window and 30.001 s is two", so the clip/window count is derived
  rather than rounded, which is the difference between a repeatable encoder row and one that moves with an off-by-one.
• median / min / max after one warm-up, and the comparison is stated as being against the execution unit actually available (the AMD ONNX pack's
  encoder on an ORT provider) rather than against the vendor's own runtime — i.e. the row says what it compared instead of implying parity of runtimes.

Nothing to change. The one thing I would want in the eventual row is the load average in the same line as the number (the file's own header says the
number is taken with it), since an encoder baseline measured on a busy box is exactly the kind of figure that gets quoted later without its conditions.