Repository navigation
Conversation
test_write_syscall() submitted a NOOP with its completion record in a
local variable, spun 10,000 pauses and returned on timeout with the
descriptor still queued. On a shared work queue that other processes
keep busy the device completes the NOOP later and writes 32 bytes into
whatever has reused that stack slot by then: the stack protector aborts
("stack smashing detected"), argument parsing fails on a corrupted argv
string, or the process segfaults in main(), all at startup and none
pointing back to DTO. The timeout was also reported as an mmap error
and left that process CPU-only.
Seen with 36 processes initializing DTO at once on four shared work
queues that other DMA traffic kept saturated (a burst of VM restores):
8 crashes in 18,066 process starts, every one at 16 or more concurrent
starts, plus 55 silent CPU-only fallbacks in one burst of 647.
Keep the completion record in struct dto_wq, which is static storage and
32-byte aligned, so a late completion lands in memory DTO owns for the
life of the process. Wait up to DTO_PROBE_TIMEOUT_MS (1 s, the clock
read every 1024 pauses) instead of a fixed spin count, and log a probe
timeout as such. With the record off the stack a timeout is harmless:
the work queue is rejected as before and the record is simply
overwritten later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a density-dependent crash at process start when DTO submits through
write().When the portal mmap is refused (idxd requires
CAP_SYS_RAWIOfor user submission unless the device is marked user-submission safe),test_write_syscall()checks that the work queue accepts descriptors throughwrite(). It submits a NOOP whose completion record is a local variable, spins 10,000 pauses, and returns on timeout with the descriptor still queued. On a shared work queue that other processes keep busy, the device completes the NOOP later and writes its 32-byte completion record into whatever has reused that stack slot by then. What follows depends on what is there: the stack protector aborts (stack smashing detected), argument parsing fails on a corruptedargvstring, or the process segfaults somewhere inmain(). Every failure is at startup and none of them points at DTO. The timeout is also reported asmmap error for DSA wqand leaves that process CPU-only.Seen with 36 processes initializing DTO at the same moment on four shared work queues that other DMA traffic kept saturated (a burst of VM restores whose page copies go through the same queues from the kernel): 8 crashes in 18,066 process starts, all at 16 or more concurrent starts, and 55 silent CPU-only fallbacks in one burst of 647, which quietly skewed the CPU measurements of that configuration.
The change:
struct dto_wq(static storage, 32-byte aligned), so a late completion lands in memory DTO owns for the life of the process.DTO_PROBE_TIMEOUT_MS(1 s, reading the clock every 1024 pauses) instead of a fixed spin count, and a timeout is logged aswrite() probe did not complete in 1000 msahead of the existing mmap error.Testing (Xeon 6787P, four shared work queues, portal mmap refused so every submit goes through
write()):make,make dto-testand./dto-testpass against the device.cmake -B build -DDTO_BUILD_TESTS=ONthenctest --label-regex functional: 3/3 pass.DTO_TEST_EXPECT_DSA=1 ./dto-test-functionaland./dto-test-functional-explicit: 29/29 each, so thewrite()submission path still works after the probe.-Wall -Wextrabuilds clean of new warnings.