Skip to content

test(sim): commit the three remaining differential harnesses (Refs #2348) - #2381

Merged
gHashTag merged 4 commits into
masterfrom
sim/2348-harnesses
Aug 22, 2026
Merged

test(sim): commit the three remaining differential harnesses (Refs #2348)#2381
gHashTag merged 4 commits into
masterfrom
sim/2348-harnesses

Conversation

@gHashTag

Copy link
Copy Markdown
Owner

Commits the three differential harnesses that were still uncommitted under #2348, following the shape #2379 set for tb_bitnet_dma_write_address.v — same sim/ directory, same structure, same README section style.

harness issue emitter published claim
tb_bitnet_sequencer_zero_count.v #1977 (PR #2337) bitnet_pipeline.rs master never terminates for a zero-neuron request
tb_bitnet_prefetch_done_pulse.v #1985 (PR #2340) bitnet_buffers.rs prefetch_done read stale at the next request
tb_bitnet_dma_we_default.v #2006 (PR #2344) bitnet_dma.rs old and new observationally identical

Every published number reproduces

== CASE 1: zero neurons (num_neurons=0, num_chunks=4), 200000 cycles ==
    old: done_seen=0 done_pulses=0 valid_seen=1 max_nid=50000 final_nid=50000
    new: done_seen=1 done_pulses=1 valid_seen=0 max_nid=0 final_nid=0
  control 1: num_neurons=1 num_chunks=1   -> 0 mismatches
  control 6: num_neurons=2 num_chunks=255 -> 0 mismatches
RESULT: PASS (0 errors)
    old: t1 sampled_done=0  t2 sampled_done=1  done_rises=2  done_high_cycles=24  we_count=4
    new: t1 sampled_done=0  t2 sampled_done=0  done_rises=2  done_high_cycles=2   we_count=4
RESULT: PASS (0 errors)
== THREE-WAY: A=pre-#2006  B=#2006  C=#2006+#2003 ==
    A vs B: 0 mismatching cycle(s)                 <- the null result under test
    B vs C: 266 mismatching cycle(s), first at 5   <- anti-vacuity control
RESULT: PASS (0 errors)

#2006 is a null result, so it got a third rendering

Its claim is that old and new are observationally identical — a harness that "passes" by showing no difference proves nothing, since one wired to the wrong ports finds no difference either.

A, B and C form a linear chain: PR #2344's head rendering is byte-identical to PR #2345's base rendering. So one comparator, one stimulus, one simulation, one 293-bit vector of every output port sees all three, and B vs C is the control#2003 changed the READ_DATA address arm, so a working comparator has to see it. Substituting B into the C slot makes the run fail rather than certify its own null result:

FAIL anti-vacuity: B and C compared equal. #2003 changes the READ_DATA address arm,
so a working comparator MUST see a difference here. Since it does not, the A/B null
result below is meaningless and this run proves nothing.

Biting — one mutant per guard

mutant result
guard on num_neurons==0 only case 1 passes, case 2 fails — the two zero compares are independently guarded
guard forced to 1'b1 cases 1–2 pass, all six controls fail: valid never asserted (old=1 new=0) -- the control did no work and proves nothing
prefetch_done clear moved back inside the guard, new syntax kept FAIL new prefetch_done high-time is one cycle per completion: got 56, want 2 — measures ports, not emitter text
BRAM stride broken, flag left correct FAIL a non-prefetch_done output diverged ... baddr 1/2
READ_DATA else-clear deleted from both DMA renderings FAIL A vs B local writes: got 18, want 22 — A emits 4 spurious strobes; #2006 is latent today and a real backstop

Anti-vacuity in all three: placing the fixed rendering in the old slot makes each harness fail, so none can pass against a non-defective "before" — e.g. FAIL harness: old reported done for a zero-neuron request -- the non-termination this harness exists to demonstrate is not reproducing.

Three numbers did not reproduce at first — all three were my instrument

Diagnosed before any assertion was touched:

  • A posedge-sampled observer read registered outputs before their non-blocking update and reported max_nid=49999 while the port read 50000; 200000/4 confirms 50000 independently. Observers moved to the negedge.
  • A negedge observer raced the stimulus that drove start_prefetch, reporting t2 sampled_done=0 for both renderings. The sample moved into the driving process.
  • first_chunk/last_chunk are absent from the reset block of both renderings, so both sit at X until the first RUN cycle and X !== X. Now compared only while valid is high, with qual_cycles asserted equal to num_neurons*num_chunks so the qualification cannot void the check.

A fourth bug surfaced only by running a mutant: $display two-string continuation together with format args printed "expe" as 1702391909. Failure messages that are never exercised are not known to work.

Reporting, not gates

vvp exits 0 on RESULT: FAIL as well as RESULT: PASS, and no workflow runs these — cargo test -p t27c is invoked by none (#2292) and fpga-build.yml never calls vvp (#2241). No claim is made that anything here defends master. Wiring sim/ into CI is tracked separately.

Every emitter compiles standalone under rustc with a four-line driver — no cargo, no target directory — because the only use in each is use super::* inside #[cfg(test)]. Recipe and per-harness base/head shas are in sim/README.md.

Refs #2348, #1977, #1985, #2006

@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-22 02:04:22 UTC

Summary

Status Count
Total Open PRs 3
PRs with Failing Checks 1
PRs with All Checks Green 2
READY 1
FAILING 1
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fba627661c36 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@gHashTag
gHashTag enabled auto-merge (squash) August 22, 2026 02:04
claude added 4 commits August 22, 2026 09:09
…ng (Refs #1977)

`layer_sequencer` never left RUN when asked for zero work: both terminators
are `index == count-1` against an unsigned port, and the bare literal 1 widens
each subtraction to 32 bits, so a zero count borrows to 32'hFFFFFFFF while the
index zero-extends.

Elaborates the PR #2337 pre-fix and post-fix renderings in one simulation.
Reproduces the published numbers: 200,000 cycles with no `done` and
`neuron_id` reaching exactly 50,000, plus six non-zero controls that are
identical on every output every cycle.

Reporting, not a gate: `vvp` exits 0 on FAIL as well as PASS.

Refs #2348
#1985)

`weight_prefetch_ctrl` documented `prefetch_done` as a one-cycle pulse but
cleared it only inside the start guard, leaving it asserted for the whole idle
gap. A requester sampling it in the cycle it raises `start_prefetch` reads the
previous transaction's completion.

Elaborates the PR #2340 pre-fix and post-fix renderings in one simulation.
Reproduces the published numbers: t2 sampled_done OLD=1 / NEW=0, done_rises
2/2, we_count 4/4. A second case varies the idle gap to separate level from
pulse: old's high-time grows with the gap, new's does not.

Reporting, not a gate: `vvp` exits 0 on FAIL as well as PASS.

Refs #2348
)

#2006 defaults `local_we` low ahead of the case. The pre-fix and post-fix
renderings are observationally IDENTICAL: every reachable path already drove
the strobe, and the states that never mention it are never entered with it
high, because READ_DATA's only exit is DONE_ST.

A harness that passes by finding no difference proves nothing, so this one is
three-way: A = pre-#2006 (PR #2344 base), B = #2006 (PR #2344 head), C =
#2006 + #2003 (PR #2345 head). B and C are consecutive revisions -- #2344's
head rendering is byte-identical to #2345's base -- so one comparator sees all
three. A vs B must be identical; B vs C must differ. Same comparator, same
stimulus, same run, same 293-bit vector of every output port. Putting B in the
C slot makes the run fail rather than certify its null result.

Measured: A vs B 0 mismatching cycles, B vs C 266, over 373 cycles and seven
phases. Deleting READ_DATA's `end else local_we <= 1'b0;` from both renderings
makes A emit 22 local writes against B's 18 -- latent today, a real backstop
the moment an arm stops clearing the strobe.

Reporting, not a gate: `vvp` exits 0 on FAIL as well as PASS.

Refs #2348
…ecipe (Refs #2348)

Adds a README section per harness in the style #2379 established, plus the
shared emit recipe: every BitNet emitter compiles standalone under `rustc`
with a four-line driver, no cargo and no target directory, because the only
`use` in any of them is `use super::*` inside `#[cfg(test)]`.

Records the base/head shas each harness was rendered from, and the three
instrument faults found while reproducing the published claims -- a
posedge-sampled observer trailing the design by a cycle, an observer racing
the stimulus that drove `start_prefetch`, and `first_chunk`/`last_chunk`
having no reset in either rendering. Each was a fault in the instrument; every
published number reproduced once the instrument was corrected.

Refs #1977, #1985, #2006
@gHashTag
gHashTag force-pushed the sim/2348-harnesses branch from 416c40f to 0f9295e Compare August 22, 2026 02:09
@github-actions

Copy link
Copy Markdown
Contributor

PR Dashboard

Generated at: 2026-08-22 02:09:46 UTC

Summary

Status Count
Total Open PRs 3
PRs with Failing Checks 1
PRs with All Checks Green 2
READY 1
FAILING 1
PENDING 0

Seal Status

  • ⚠️ STALE -- sha256(compiler.rs)=fba627661c36 != manifest seal=87e5cbd3ad94.
    The committed NMSE numbers were certified against an older compiler.rs.
    Run scripts/reseal-check.sh locally for the two-step reseal command (advisory; not a merge gate).

@github-actions

Copy link
Copy Markdown
Contributor

📓 NotebookLM Notebook linked to this PR

This notebook contains session context, decisions, and artifacts for this work.

@gHashTag
gHashTag merged commit e53b9d0 into master Aug 22, 2026
18 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants