Skip to content

Keep the write() submission probe's completion record off the stack - #47

Open
byrnedj wants to merge 1 commit into
intel:mainfrom
byrnedj:fix/write-probe-completion-record
Open

byrnedj wants to merge 1 commit into
intel:mainfrom
byrnedj:fix/write-probe-completion-record

Conversation

@byrnedj

@byrnedj byrnedj commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Fixes a density-dependent crash at process start when DTO submits through write().

When the portal mmap is refused (idxd requires CAP_SYS_RAWIO for user submission unless the device is marked user-submission safe), test_write_syscall() checks that the work queue accepts descriptors through write(). It submits a NOOP whose completion record is a local variable, spins 10,000 pauses, and returns on timeout with the descriptor still queued. On a shared work queue that other processes keep busy, the device completes the NOOP later and writes its 32-byte completion record into whatever has reused that stack slot by then. What follows depends on what is there: the stack protector aborts (stack smashing detected), argument parsing fails on a corrupted argv string, or the process segfaults somewhere in main(). Every failure is at startup and none of them points at DTO. The timeout is also reported as mmap error for DSA wq and leaves that process CPU-only.

Seen with 36 processes initializing DTO at the same moment on four shared work queues that other DMA traffic kept saturated (a burst of VM restores whose page copies go through the same queues from the kernel): 8 crashes in 18,066 process starts, all at 16 or more concurrent starts, and 55 silent CPU-only fallbacks in one burst of 647, which quietly skewed the CPU measurements of that configuration.

The change:

  • The completion record moves into struct dto_wq (static storage, 32-byte aligned), so a late completion lands in memory DTO owns for the life of the process.
  • The probe waits up to DTO_PROBE_TIMEOUT_MS (1 s, reading the clock every 1024 pauses) instead of a fixed spin count, and a timeout is logged as write() probe did not complete in 1000 ms ahead of the existing mmap error.
  • With the record off the stack a timeout is harmless: the work queue is rejected as before and the record is simply overwritten later.

Testing (Xeon 6787P, four shared work queues, portal mmap refused so every submit goes through write()):

  • The reproducer: a VM restore benchmark that starts 36 DTO-linked processes at once against the saturated queues. Before the fix, 2 crashes and 55 CPU-only fallbacks in one 647-session cell; after it, 0 crashes, 0 fallbacks and 0 probe timeouts over the 100 cells that followed (about 30,000 sessions).
  • The CI steps make, make dto-test and ./dto-test pass against the device.
  • cmake -B build -DDTO_BUILD_TESTS=ON then ctest --label-regex functional: 3/3 pass. DTO_TEST_EXPECT_DSA=1 ./dto-test-functional and ./dto-test-functional-explicit: 29/29 each, so the write() submission path still works after the probe.
  • -Wall -Wextra builds clean of new warnings.

test_write_syscall() submitted a NOOP with its completion record in a
local variable, spun 10,000 pauses and returned on timeout with the
descriptor still queued. On a shared work queue that other processes
keep busy the device completes the NOOP later and writes 32 bytes into
whatever has reused that stack slot by then: the stack protector aborts
("stack smashing detected"), argument parsing fails on a corrupted argv
string, or the process segfaults in main(), all at startup and none
pointing back to DTO. The timeout was also reported as an mmap error
and left that process CPU-only.

Seen with 36 processes initializing DTO at once on four shared work
queues that other DMA traffic kept saturated (a burst of VM restores):
8 crashes in 18,066 process starts, every one at 16 or more concurrent
starts, plus 55 silent CPU-only fallbacks in one burst of 647.

Keep the completion record in struct dto_wq, which is static storage and
32-byte aligned, so a late completion lands in memory DTO owns for the
life of the process. Wait up to DTO_PROBE_TIMEOUT_MS (1 s, the clock
read every 1024 pauses) instead of a fixed spin count, and log a probe
timeout as such. With the record off the stack a timeout is harmless:
the work queue is rejected as before and the record is simply
overwritten later.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant