[FEA] Add reproducible benchmark harness foundation

[dleshchev]

Is your feature request related to a problem? Please describe.

DAQIRI benchmark runners do not share a reproducible lifecycle or result contract. A zero process exit can be mistaken for a successful measurement even when required application, kernel, NIC, or physical-path evidence is absent, and cross-host cleanup does not identify exact remote processes.

Describe the solution you'd like

Add a consolidated benchmark-harness foundation with one stable Python entrypoint, versioned experiment and site-profile schemas, deterministic plan/dry-run output, immutable resolved provenance, run-ID-based local/SSH process ownership, canonical valid/invalid/failed results, and derived CSV/Markdown reports. The first supported adapter should be externally configured physical cross-host UDP, with active-window counters, explicit placement/rate scope, strict loss-free validation, documentation, and hardware-free lifecycle tests.

Describe alternatives you've considered

Extending the existing shell controllers would retain fragmented schemas and transport-specific lifecycle behavior. Migrating every transport at once would make the change too broad to review safely.

Additional context

Follow-up issues can add transactional topology management, loopback modes, raw Ethernet, TCP/RDMA/workloads, and ResNet adapters. This request does not change published benchmark values and is separate from #294/#295 and the GPU-workload performance investigation.

[dleshchev]

the harness should also support multi-rx-queue for raw ethernet option to show core/queue scaling at low message sizes in particular

[dleshchev]

Requirements carried over from the PR #293 documentation cleanup

PR #293 is being narrowed to the DGX Spark results. The general reproducibility contract belongs with this harness instead of being duplicated in the performance report.

Shared contract:

• Record the evidence class explicitly: physical cross-host, two-port cabled loopback, NIC hardware loopback, or namespace/software loopback. Each class needs its own path proof and result label.
• Resolve and archive the exact commit/dirty state, container image and binary/library identity, host/NIC/GPU/software versions, topology and cabling, role configs, CPU/IRQ/NAPI placement, queue and buffer geometry, requested rate and rate scope, duration, drain, and repetition count.
• Measure an explicit sender-active window after receiver readiness. Keep startup, drain, and shutdown-tail samples separate; do not infer throughput from whole-process runtime.
• Preserve application, kernel, NIC, PHY/RDMA, and placement evidence for every repetition. Keep rejected/exploratory runs, but exclude them from published summaries.
• Produce one immutable bundle and a machine-readable valid / invalid / failed verdict with reason codes. Publication-ready should require every requested repetition to be valid (normally three independent 30-second windows), not a selection of clean runs.
• Own and clean up exact run processes/containers, restore any harness-owned topology transactionally, and verify that no owned work remains.

Transport/topology adapters that must fit this contract:

• UDP: matching application TX/RX totals; clean InErrors, RcvbufErrors, reassembly, and NIC-discard deltas; separate requested pacing, achieved wire rate, and delivered payload rate; distinct socket I/O, application-worker, IRQ, and NAPI placement; multi-pair/multi-link rate allocation.
• Raw Ethernet: physical and hardware-loopback path proof; DAQIRI per-queue packet/error/missed/no-buffer evidence; paced and unpaced rates; DPDK and ibverbs raw engines; multi-RX-queue/core scaling, especially at small message sizes, with the resolved queue-to-core map and per-queue balance retained.
• TCP: achieved/delivered rate plus retransmission and socket-error evidence; do not interpret flow-control backpressure as loss-free peak capacity.
• RoCE/RDMA: RDMA-CM and completion-queue status, send/receive completion totals, retry/resource errors, per-interface queue selection, TX/RX depth, and aggregate buffer/pool capacity.
• Loopback: transactional namespace/two-port setup and rollback, plus mlx5 hardware-loopback evidence using vport_loopback_bytes while PHY counters remain flat.
• GPU workloads and ResNet: active-window GPU/workload evidence and application-specific completion/quality counters under the same lifecycle.

PR #298's first UDP adapter already implements much of the shared contract. The unsupported transport/topology items should become explicit linked follow-ups before #296 is closed so they do not remain only as prose in a benchmark-results PR.