[BUG] DPDK header-data-split RX caps at ~3 Mpps; the ibverbs engine does not

[RamyaGuru]

Summary

daqiri_bench_raw_hds drops the large majority of received packets at small and medium frame
sizes. RX plateaus at roughly 3 Mpps regardless of frame size, while the plain
single-segment GPUDirect path sustains 20 Mpps on the same wire, same NIC, same frame size,
with zero loss.

Frame size is not the cause. It only determines how many packets/sec line rate implies, so the
loss disappears at 8 KB frames purely because the sender can no longer offer more than ~2.7 Mpps
— which happens to fit under the ceiling.

The ibverbs raw engine is not affected: its HDS path carried everything offered to it
(7.88 Mpps, 0.0% loss) where the DPDK engine collapses to 1.64 Mpps and sheds 87%. So this is a
DPDK-engine HDS problem, not an inherent cost of header-data split.

Evidence

Cable loopback between the two ports of one ConnectX-7, daqiri_bench_raw_hds, 6 s runs,
examples/daqiri_bench_raw_tx_rx_hds.yaml with the ipv4_len flow match and the TX/RX payload
buffer sizes scaled to each frame size:

Frame	TX offered	RX received	TX Mpps	RX Mpps	Loss
1064 B	77,660,160	9,844,736	12.94	1.64	87.3%
2064 B	65,382,400	18,121,202	10.90	3.02	72.3%
4064 B	33,699,840	19,973,334	5.62	3.33	40.7%
8064 B	16,240,640	16,070,620	2.71	2.68	1.0%

RX Mpps is flat at 1.6-3.3 across a 8x change in frame size, which is the signat
per-packet cost ceiling rather than a bandwidth limit.

The same sweep on daqiri_bench_raw_gpudirect (single segment, all-GPU, no HDS), same wire:

Frame	RX Mpps	Loss
1064 B	20.14	0.0%
2064 B	10.03	0.0%
4064 B	5.26	0.0%
8064 B	2.37	0.0%

At 1064 B the single-segment path does 12x the packet rate of HDS with no lo

Drops surface as port-level rx_missed, logged by the stats poller, e.g.:

[ERROR] daqiri_dpdk_stats.cpp:344: 'rx_port' interface (1), Rx:
        Dropped 5,632,830 packets since last poll 500ms ago (total: 5,925,886)

What it is not

• Not frame size. The single-segment path is clean at every size tested, including 1064 B.

• Not the GPU reorder path. daqiri_bench_raw_reorder_seq at the *identical
  (payload 1000 + header 64) sustains 15.2 Mpps with ~10k missed out of 76M (0.013%). The
  config-driven reorder kernels are not implicated.

• Not memory-region undersizing. The HDS config's regions are num_bufs: 51200 against an
  8192-deep ring and batch_size: 10240, which clears both the deadlock floor a
  starvation-free target from [BUG] IGX Thor DevKit raw GPUDirect Testing Follow-ups #242. rx_mbuf_allocation_errors is 0 or negligible in these runs
  while rx_missed is in the 100M range, so the pool is not the constraint.

• Not inherent to header-data split. The ibverbs engine implements HDS with a
  non-striding DevX regular RQ and multi-segment scatter WQEs
  (Regular RQ q0: num_segs=2 ... split=64), and it loses nothing:

  Engine, 1064 B	TX offered	RX received	RX Mpps	Loss
  ibverbs, HDS (2 seg)	47,292,416	47,288,632	7.88	0.0%
  ibverbs, single segment	121,044,992	120,590,336	20.10	0.4%
  DPDK, HDS (2 seg)	77,660,160	9,844,736	1.64	87.3%
  DPDK, single segment	-	-	20.14	0.0%

  ibverbs HDS RX sustained 4.8x the DPDK HDS RX rate with zero drops. Its true c
  unknown and is at least 7.88 Mpps: the ibverbs TX side was the limiter in that run, capped
  by max_qp_wr (see caveats), so RX was never saturated.

Likely cause (unconfirmed)

The HDS RX queue uses a physical header-data split: a multi-segment scatter
packet, header into a CPU memory region and payload into a GPU region. Relative to the
single-segment path that adds, per packet, a second scatter-gather entry, a seco
allocation, and a host-memory write for the header. Any of those could dominate at multi-Mpps.
Profiling is needed to attribute it.

Reproduction

# fill in PCIe addresses and eth_dst_addr for a cable loopback first
./build/examples/daqiri_bench_raw_hds  ./daqiri_bench_raw_tx_rx_hds.yaml --secon
./build/examples/daqiri_bench_raw_gpudirect ./daqiri_bench_raw_tx_rx.yaml --seconds 10

Compare Total packets transmitted/received by application between the two, and
rx_missed in the stats output. Note that changing payload_size in the HDS config also
requires updating the ipv4_len RX flow match (payload_size + header_size - 14 TX/RX payload region buf_size`, or the flow stops matching.

Open questions

• Does the ceiling scale with RX queues/cores? Only a single-queue HDS config was measured; no
  multi-queue HDS config ships today. If it scales per queue, this is a per-core
  workaround is more queues. If it does not, it is a shared bottleneck.
• What is the ibverbs HDS RX ceiling? It was never saturated (0% loss at 7.88 Mp
  ibverbs TX capped first. Driving it from a separate faster TX process would establish the
  real number and the true size of the gap.
• Since ibverbs handles the same two-segment scatter without loss, what is the DPDK path doing
  differently -- mempool pressure per segment, the rxconf_qsplit scatter setup,
  multi-segment RX loop?
• Is the documented HDS use case (headers to CPU, payload to GPU) expected to ru
  rates, or is ~3 Mpps within design expectations? If the latter, this is a documentation gap
  rather than a defect.

Caveats on the ibverbs comparison

The ibverbs run is not parameter-matched to DPDK. The ibverbs engine derives its
from num_bufs and provisions 2x that as send WRs, so the shipped num_bufs: 51200 fails QP
creation (device max_qp_wr 32768) and HDS with 2 SGEs fails even at 16384. The
therefore use num_bufs: 8192 and batch_size: 4096. That lowers ibverbs' offered TX rate but
does not weaken the conclusion: DPDK HDS RX drops 87% of a 12.9 Mpps offer and c
3 Mpps at every frame size, while ibverbs HDS RX absorbed 7.88 Mpps cleanly.

Two incidental findings from setting this up, both worth separate issues if they are news:

• ipv4_len RX flow matching is unusable on this NIC under the ibverbs engine:
  CREATE_GENERAL_OBJECT(FLEX_PARSE_GRAPH) failed: Remote I/O error (syndrome 0x examples/daqiri_bench_raw_tx_rx_hds.yamlusesipv4_len: 1050` for steering, that config
  cannot run on ibverbs unmodified; the comparison above steers on UDP ports ins
• The ibverbs engine stalls silently when batch_size exceeds the queue's slot count: TX
  reports posted=0 completed=0 indefinitely with no error, the same failure sh
  stall in [BUG] IGX Thor DevKit raw GPUDirect Testing Follow-ups #242.

Environment

IGX Thor, aarch64, kernel 6.8.0-1019-nvidia-tegra-rt, driver 580.00 / CUDA 13.0,
ConnectX-7 dual-port cable loopback (0004:03:00.0 <-> 0004:03:00.1), RTX PRO
(BAR1 8 GiB), container build, DPDK and ibverbs raw engines. daqiri 676b260.