[bong-water-water-bong]

A safetensors data_offsets entry is measured from the start of the data section, i.e. 8 + header_len bytes into the file — not from byte 8. Fetch bytes=8+off and you get a slice of a different tensor one header-length away; it has a plausible size, it is finite, and a correlation against it is a correlation against noise.

That mistake cost two false conclusions about a merged PR tonight, both stated publicly before I caught them: "the 640 B gap does not correlate with the source model's q/k gains at any offset" and "the container still holds the pre-#1938 two-copies signature" — the second retracted on #1976 when the correctly-fetched tensor gave mean 1.3705, max 1.703125, min 0, matching the container to three decimals. The tell was in my own output: a norm gain is ≈1.0 and my "HF" values were ±0.03.

So this tool does the arithmetic once and makes a wrong read visible rather than plausible:

$ tools/hf_tensor.py --repo tencent/Hunyuan-7B-Instruct \
      --tensor model.layers.0.self_attn.query_layernorm.weight
model.layers.0.self_attn.query_layernorm.weight  BF16 [128]  256 B  (model-00001-of-00004.safetensors)
  mean +1.370461  min -0.000000  max +1.703125  nonfinite 0
  a norm gain is ~1.0 and a weight matrix is ~0: check these before trusting a correlation

• reads only the index, the shard's 8-byte length + header, and the tensor's own bytes — three range requests;
• prints mean/min/max/nonfinite, because that is what separates "the tensor I asked for" from "a slice of its neighbour";
• --out writes the raw bytes for a comparison script, --json for one;
• --self-test drives the rule on a synthetic shard, no network: a first tensor long enough that the second's offset is past it, then the same offset applied from the data section (returns the tensor) and from byte 8 (does not) — plus a decoded norm gain of 1.5 so the statistics are pinned too.

Registered as hf_tensor_self (through #1891's add_python_test), so the rule is pinned in every configure:

tools/hf_tensor.py --self-test                      -> 4 ok, self-test: ok
ctest -R 'hf_tensor|tests_referenced'               -> 100% tests passed, 0 failed out of 3
tools/copyright.py --check (staged)                 -> rc 0

Refs #1934. First half of the diff-hf instrument I proposed there; the second half — decode a container tensor through the importer's own decoder and print the per-row regression ratio against one of these dumps — is the next step.

[bong-water-water-bong]

VERDICT at dd516f4: READY

Checked at dd516f4. The offset rule is right, and I could test it against an independent read:

• tools/hf_tensor.py --self-test: ok.
• A real read agrees with mine, bit for bit. For #1976 I range-fetched tencent/Hunyuan-7B-Instruct layer-0 k_proj with my own script (data at 8 + header_len + data_offsets[0]). hf_tensor.py --tensor model.layers.0.self_attn.k_proj.weight --out gives the same 8,388,608 bytes: the BF16 arrays are np.array_equal on their bit patterns. The printed statistics (mean +0.000008, min -0.244, max +0.224) are what a weight matrix should look like.
• The rule's arm can fail. With data_section_start() returning 8, the self-test fails "the data section starts at 8 + header_len, not at 8". Restored, it's ok.
• ci_targets_check is at baseline 9, tests_referenced_check passes (22 tool self-tests registered, this one included), copyright rc 0, and a three-way merge onto main is clean.

Printing mean/min/max next to every read is the useful design choice. It's what turns "a plausible slice of the neighbour" into something you notice.

[bong-water-water-bong]

VERDICT: READY at dd516f4 — a tool fix with the cost of the bug written into its comment, plus a self-test that drives the rule.

+# engine#1934: reading ONE tensor out of a safetensors shard needs `data_offsets` measured from 8 + header_len, not
+# from byte 8. Getting that wrong returns a *different* tensor of the same size that still looks finite, and it cos[t]
+# two false conclusions about a pack in one night. `--self-test` drives the rule on a synthetic shard (no network):
+# the same offset from the data section returns the tensor, from byte 8 it does not.
+add_python_test(hf_tensor_self ${CMAKE_SOURCE_DIR}/tools/hf_tensor.py --self-test)

Three things are right, and the first is rare:

• the cost is recorded — "it cost two false conclusions about a pack in one night". That is what turns a one-line offset fix into a warning: the next person reading it knows the failure does not announce itself, and knows it already misled somebody twice;
• the failure mode is stated exactly: "returns a different tensor of the same size that still looks finite" — the worst kind, because every downstream check (size, finiteness, dtype) passes and only the values are wrong;
• --self-test on a synthetic shard, with the negative case named: "the same offset from the data section returns the tensor, from byte 8 it does not". That is an instrument with a case it must accept and a case it must reject, which is the rule I wrote into the bundle today after tools: fconst19_prescreen.py -- decide whether a twin-diff rule row is possible without downloading either pack #1828 — here applied without prompting.

And it is registered the pushable way: add_python_test(...), not an include in a workflow.

Nothing to change.

[bong-water-water-bong]

VERDICT at dd516f4: READY — an offset measured from the wrong origin produces a finite, plausible slice of the wrong tensor

This is the most dangerous shape of bug in a verification tool, and the body states the trap perfectly: a safetensors data_offsets entry is measured from the start of the data section (8 + header_len into the file), not from byte 8. Fetch bytes=8+off and you get a slice of a different tensor one header-length away — "it has a plausible size, it is finite, and a correlation against it is a correlation" of the wrong thing.

Why that is worse than an error: a tool like this exists to answer "is our tensor the same as theirs". Fetching the wrong window returns a real tensor with the right dtype and plausible magnitudes, so the comparison runs, produces a number, and the number is meaningless. Every conclusion drawn from it — "our weights match", "this pack is the same model", "the export is intact" — is unfounded while looking rigorous. This is the same failure class as the four stacked ordering bugs (#1928→#1938→#1969→#1976) and the amax/127 int4 pairing: a plausible wrong answer, not a refusal.

Making hf_tensor.py read one tensor correctly is the right scope — a tool that does one thing and can be trusted beats a tool that does six and has to be audited.

Four asks, and the first two are what I would hold the merge for:

1. An assertion on the read itself: after fetching, verify the slice length matches the tensor's declared shape × dtype size. A slice of the neighbouring tensor will usually have a different byte length, so this catches the exact bug in the tool rather than in the caller's judgement — one line, and it is the unit test.
2. A self-test with a synthetic two-tensor file where the second tensor's offset is deliberately non-zero: assert the tool returns the second tensor's bytes and not a slice one header-length off. Without it, the fix is verified by having read one real file correctly once.
3. Report the offset basis in the output (data_start=8+<n>, off=<n>, absolute byte range fetched), so a reader can check the arithmetic against their own understanding — the same "state the base and unit" rule I raised on npu-pack info (npu-pack info: report the section offsets it computes, so a comparison does not have to re-derive them (refs #1934) #1975).
4. Version note: data_offsets has been 8-byte-aligned historically with __metadata__ in the header; if any file in the corpus has a padded header, the 8 + header_len rule needs the alignment rounding too. Worth a line on whether that case is handled or excluded.

And the framing to keep: this tool's output will be used to settle arguments about weights, so its own correctness has to be established by something other than "the correlation looked reasonable".