Repository navigation
ci: run the short suite under the race detector, close #2700 - #2797
Merged
Merged
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
|
gzliudan
requested review from
AnilChinchawale,
anunay-xin,
benjamin202410,
liam-lai and
wanwiset25
October 14, 2026 12:15
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 14, 2026 16:10
3cc51a0 to
cfebe79
Compare
wanwiset25
approved these changes
Oct 15, 2026
anunay-xin
approved these changes
Oct 15, 2026
gzliudan
force-pushed
the
ci-race-detector
branch
2 times, most recently
from
October 18, 2026 12:34
f4388bb to
6d09623
Compare
benjamin202410
approved these changes
Oct 19, 2026
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 20, 2026 04:33
6d09623 to
98c729d
Compare
gzliudan
force-pushed
the
ci-race-detector
branch
2 times, most recently
from
October 20, 2026 06:21
62a887e to
ff07e3c
Compare
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 20, 2026 06:45
ff07e3c to
a88ea90
Compare
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 20, 2026 07:08
a88ea90 to
d65d907
Compare
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 20, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel, with a timeout so a reorg that never sends still fails, and compare its log count against the chain that was reverted. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 20, 2026 07:26
d65d907 to
15f361e
Compare
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 20, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel, with a timeout so a reorg that never sends still fails, and compare its log count against the chain that was reverted. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 20, 2026
SimNode.Start registers the services before the devp2p node is started, and node.RegisterProtocols appends to the p2p server's protocol slice in that window. The simulations HTTP API marshals the node info from another goroutine: the event stream does it as soon as a control event is handed over, and the node handlers do it on request. Neither side takes a lock, so a node that is created and then started races with the HTTP layer reading the protocol slice. The new race job in XinFinOrg#2797 reported it as TestHTTPSnapshot failing on p2p.Server.NodeInfo reading srv.Protocols against node.RegisterProtocols appending to it. Hold the startup lock for the whole of Start and take it shared in NodeInfo. The lock order stays startMu -> node.lock, the only other lock in the path, and nothing reached from Start calls NodeInfo, so readers wait for the startup instead of observing a half-filled server. The change is confined to the in-memory adapter and leaves the event and node response payloads alone. Upstream geth deleted p2p/simulations in ethereum#30250 without fixing this, and both NodeInfo and RegisterProtocols are still unsynchronised there, so there is nothing to port. Tests: 100 runs of TestHTTPSnapshot as four concurrent processes under -race (short, count=25 each) report no race; dropping the read lock again makes 3 of the 4 processes fail, so the lock is what keeps it quiet. make all, go test ./p2p/simulations/... pass, golangci-lint 2.12.2 on ./p2p/simulations/... reports no new findings, and make tidy reports no untidy module files.
gzliudan
force-pushed
the
ci-race-detector
branch
2 times, most recently
from
October 20, 2026 07:54
a110646 to
786812a
Compare
This was referenced Oct 20, 2026
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 20, 2026 12:07
786812a to
b5a21a7
Compare
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 22, 2026 09:05
0b4be46 to
812a622
Compare
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel and compare its log count against the chain that was reverted. Both ends of that channel are bounded by a timeout, so a reorg that never sends fails the test without stranding the subscriber goroutine. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s, and dropping the send altogether fails it within the timeout while the subscriber goroutine leaves the receive. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 22, 2026
SimNode.Start registers the services before the devp2p node is started, and node.RegisterProtocols appends to the p2p server's protocol slice in that window. The simulations HTTP API marshals the node info from another goroutine: the event stream does it as soon as a control event is handed over, and the node handlers do it on request. Neither side takes a lock, so a node that is created and then started races with the HTTP layer reading the protocol slice. The new race job in XinFinOrg#2797 reported it as TestHTTPSnapshot failing on p2p.Server.NodeInfo reading srv.Protocols against node.RegisterProtocols appending to it. Hold the startup lock for the whole of Start and take it shared in NodeInfo. The lock order stays startMu -> node.lock, the only other lock in the path, and nothing reached from Start calls NodeInfo, so readers wait for the startup instead of observing a half-filled server. The change is confined to the in-memory adapter and leaves the event and node response payloads alone. Upstream geth deleted p2p/simulations in ethereum#30250 without fixing this, and both NodeInfo and RegisterProtocols are still unsynchronised there, so there is nothing to port. Tests: 100 runs of TestHTTPSnapshot as four concurrent processes under -race (short, count=25 each) report no race; dropping the read lock again makes 3 of the 4 processes fail, so the lock is what keeps it quiet. make all, go test ./p2p/simulations/... pass, golangci-lint 2.12.2 on ./p2p/simulations/... reports no new findings, and make tidy reports no untidy module files.
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel and compare its log count against the chain that was reverted. Both ends of that channel are bounded by a timeout, so a reorg that never sends fails the test without stranding the subscriber goroutine. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s, and dropping the send altogether fails it at once on the armed check, with the subscriber goroutine then leaving the receive within its own timeout. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel and compare its log count against the chain that was reverted. Both ends of that channel are bounded by a timeout, so a reorg that never sends fails the test without stranding the subscriber goroutine. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s, and dropping the send altogether fails it at once on the armed check, with the subscriber goroutine then leaving the receive within its own timeout. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel and compare its log count against the chain that was reverted. Both ends of that channel are bounded by a timeout, so a reorg that never sends fails the test without stranding the subscriber goroutine. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s, and dropping the send altogether fails it at once on the armed check, with the subscriber goroutine then leaving the receive within its own timeout. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel and compare its log count against the chain that was reverted. Both ends of that channel are bounded by a timeout, so a reorg that never sends fails the test without stranding the subscriber goroutine. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s, and dropping the send altogether fails it at once on the armed check, with the subscriber goroutine then leaving the receive within its own timeout. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel and compare its log count against the chain that was reverted. Both ends of that channel are bounded by a timeout, so a reorg that never sends fails the test without stranding the subscriber goroutine. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s, and dropping the send altogether fails it at once on the armed check, with the subscriber goroutine then leaving the receive within its own timeout. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 23, 2026 23:11
812a622 to
91f5e1d
Compare
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 24, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel and compare its log count against the chain that was reverted. Both ends of that channel are bounded by a timeout, so a reorg that never sends fails the test without stranding the subscriber goroutine. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s, and dropping the send altogether fails it at once on the armed check, with the subscriber goroutine then leaving the receive within its own timeout. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
21 tasks
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 26, 2026 04:10
91f5e1d to
9d72cc2
Compare
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 26, 2026
SimNode.Start registers the services before the devp2p node is started, and node.RegisterProtocols appends to the p2p server's protocol slice in that window. The simulations HTTP API marshals the node info from another goroutine: the event stream does it as soon as a control event is handed over, and the node handlers do it on request. Neither side takes a lock, so a node that is created and then started races with the HTTP layer reading the protocol slice. The new race job in XinFinOrg#2797 reported it as TestHTTPSnapshot failing on p2p.Server.NodeInfo reading srv.Protocols against node.RegisterProtocols appending to it. Hold the startup lock for the whole of Start and take it shared in NodeInfo. The lock order stays startMu -> node.lock, the only other lock in the path, and nothing reached from Start calls NodeInfo, so readers wait for the startup instead of observing a half-filled server. The change is confined to the in-memory adapter and leaves the event and node response payloads alone. Upstream geth deleted p2p/simulations in ethereum#30250 without fixing this, and both NodeInfo and RegisterProtocols are still unsynchronised there, so there is nothing to port. Tests: 100 runs of TestHTTPSnapshot as four concurrent processes under -race (short, count=25 each) report no race; dropping the read lock again makes 3 of the 4 processes fail, so the lock is what keeps it quiet. make all, go test ./p2p/simulations/... pass, golangci-lint 2.12.2 on ./p2p/simulations/... reports no new findings, and make tidy reports no untidy module files.
gzliudan
added a commit
to gzliudan/XDPoSChain
that referenced
this pull request
Oct 26, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel and compare its log count against the chain that was reverted. Both ends of that channel are bounded by a timeout, so a reorg that never sends fails the test without stranding the subscriber goroutine. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s, and dropping the send altogether fails it at once on the armed check, with the subscriber goroutine then leaving the receive within its own timeout. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan
added a commit
that referenced
this pull request
Oct 26, 2026
* fix(core): keep the removed-logs delivery test off the scheduler TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the subscriber goroutine after the channel handoff, and read that flag once the reorg returned. An unbuffered handoff only orders the send against the receive: whatever the subscriber does next runs in parallel with the rest of the reorg, so a subscriber the scheduler put off for a few hundred microseconds made a synchronous send look asynchronous. The new race job (-p 4 on a 4 vCPU runner) failed on it in #2797 with the reorg having waited the full subscriber delay. Announce readiness before parking on the receive instead: that write is sequenced before the handoff, which in turn happens before the send the reorg waits for returns. Carry the arrival over a buffered channel and compare its log count against the chain that was reverted. Both ends of that channel are bounded by a timeout, so a reorg that never sends fails the test without stranding the subscriber goroutine. Upstream geth has no equivalent test and sends synchronously as well, so there is nothing to port. Tests: 160 runs of the test as four processes pinned to four cores under GOMAXPROCS=4, 7 of which failed before this change and none after; turning the send into a goroutine still fails the test within 0.02s, and dropping the send altogether fails it at once on the armed check, with the subscriber goroutine then leaving the receive within its own timeout. make all, go test ./core and go run build/ci.go lint ./core/... (no new findings) pass. * test(core): widen the removed-logs delivery budget The subscriber delay is not only a stall: it is also how long an asynchronous send may take before the check can no longer tell it apart from a synchronous one. reorg hands the removals over near its start (both rmLogsFeed.Send calls precede the rollback and the head writes), so a spawned send has to outlast the delay before it hides. At 250ms that window was ~12x the ~20ms of remaining reorg work measured under -race, so a slow runner could pass a regression silently. Raise it to a second, which leaves ~50x, and record the premise next to the constant: the budget tracks where the send sits in reorg, not the subscriber. Spawning the send still fails the test within 0.01s, so the guard is unchanged. The test and its 250ms budget predate this branch - the previous commit only changed how the arrival is recorded - so this lands on its own commit instead of being folded into it. Tests: make all; go test ./core (24.3s); the test under -race (1.02s); 40 runs as four processes pinned to four cores under GOMAXPROCS=4, none failing; lint on ./core/... (26 issues the package already had, none in the diff).
gzliudan
added a commit
that referenced
this pull request
Oct 26, 2026
) SimNode.Start registers the services before the devp2p node is started, and node.RegisterProtocols appends to the p2p server's protocol slice in that window. The simulations HTTP API marshals the node info from another goroutine: the event stream does it as soon as a control event is handed over, and the node handlers do it on request. Neither side takes a lock, so a node that is created and then started races with the HTTP layer reading the protocol slice. The new race job in #2797 reported it as TestHTTPSnapshot failing on p2p.Server.NodeInfo reading srv.Protocols against node.RegisterProtocols appending to it. Hold the startup lock for the whole of Start and take it shared in NodeInfo. The lock order stays startMu -> node.lock, the only other lock in the path, and nothing reached from Start calls NodeInfo, so readers wait for the startup instead of observing a half-filled server. The change is confined to the in-memory adapter and leaves the event and node response payloads alone. Upstream geth deleted p2p/simulations in ethereum#30250 without fixing this, and both NodeInfo and RegisterProtocols are still unsynchronised there, so there is nothing to port. Tests: 100 runs of TestHTTPSnapshot as four concurrent processes under -race (short, count=25 each) report no race; dropping the read lock again makes 3 of the 4 processes fail, so the lock is what keeps it quiet. make all, go test ./p2p/simulations/... pass, golangci-lint 2.12.2 on ./p2p/simulations/... reports no new findings, and make tidy reports no untidy module files.
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 26, 2026 05:32
9d72cc2 to
de8bd9b
Compare
benjamin202410
approved these changes
Oct 26, 2026
gzliudan
force-pushed
the
ci-race-detector
branch
2 times, most recently
from
October 27, 2026 15:41
b8684a4 to
2837a13
Compare
gzliudan
force-pushed
the
ci-race-detector
branch
3 times, most recently
from
October 27, 2026 16:31
130160d to
9cc943d
Compare
make test, make quick-test and all eight shards of ci.yml call `go run build/ci.go test`, and none of them passes -race, so an access that is unsynchronised at run time stays invisible to CI. build/ci.go has carried the flag since it was ported from geth ethereum#23480 - nothing ever switched it on (XinFinOrg#2700). Every package the job started out excluding is fixed (XinFinOrg#2791, XinFinOrg#2793 by XinFinOrg#2802, XinFinOrg#2794) or gone with the package (XinFinOrg#2792, whose disposition was deletion via XinFinOrg#2796), so the bulk pass now covers the whole tree. metrics no longer needs a second, non-short run either: XinFinOrg#2795 (fixed by XinFinOrg#2801) removed the `testing.Short()` guard from TestUniformSampleConcurrentUpdateCount, so the short pass reaches it. rpc gets a slice of its own: the bulk pass runs four packages at a time, and TestClientWebsocketLargeMessage does not survive that on a GitHub-hosted runner - twice in a row the job failed after 30.00s with `websocket: close 1006 (abnormal closure): unexpected EOF`, the websocket codec's 30s ping/pong watchdog, with no data race reported in either attempt. The package is green when it does not compete with three other packages - 26s here, and 24.9s in the plain CI shards, which run one package at a time - so it leaves the bulk pass, which keeps the detector on the package instead of dropping it from the job. The job runs four slices of the bulk pass in parallel and then race-test-rpc, and the shard split, the rpc carve-out and the package list live in the Makefile, so make race-test (a single shard) runs exactly what CI runs and a local sweep predicts the job. It does not pass -failfast: the sweep exists to report every race in one go, and the first failing package would stop the rest. AGENTS.md mentions the target in its test checklist. Four slices because the detector makes the pass compile-bound: one job spent 8m28s on the 752s of test time that a twelve-core laptop covers in 5m38s. A slice is the name-sorted package list taken modulo four, 32 packages each, with -p 4 overlapping whatever shares one; side by side here they took 46s, 76s, 97s and 122s. actions/cache restores ~/.cache/go-build per slice, keyed by the dependency hash rather than the commit, so a slice refills when go.sum moves instead of once per push: an entry is 660 MB unpacked and 156 MB stored, and five of them fit inside the 10 GB a repository may cache. That cache also holds go's test results, so a slice reports a previous verdict for packages whose inputs did not move. Only packages with test files are listed: the other 70 would add build time and no coverage. -short leaves out the long consensus cases (skipLongInShortMode under consensus/tests/engine_v2_tests), the discover UDP/lookup cases and the tests/ performance cases. The plain shards run those without the detector, so make race-test-full exists for the local sweep a change to those paths needs. The target collects the package list before testing it: an argument list drops the substitution's exit status, and an empty slice would send build/ci.go back to ./... with rpc in it, so the slice is checked before it is used. make runs the recipe with /bin/sh, which has no pipefail, so that check is what covers go list failing outright. tag_build now waits for the race job as well, so a release artifact cannot ship with a known race. devnet_build_push keeps waiting for tests only: it fires on every merge, and a flaky detector would hold up the devnet image each time. Verified with the job's command against dev-upgrade 72ed9d6: rc=0, zero WARNING: DATA RACE, 126 packages ok in 3m49s. rpc on its own is ok in 26.0s, eth/bft in 1.7s, metrics in 6.6s (the concurrent sample test runs instead of skipping, 5.3s), make all exits 0, and no .go file is touched. make race-test against baf6d16 reports rc=0 with zero DATA RACE (rpc 23.0s, trie 48.8s), and pointing the recipe at a go that cannot list makes it exit instead of testing the whole tree. With the shards and the cache in place, the four slices run side by side report rc=0 and zero DATA RACE (46s/76s/97s/122s), their union equals the unsharded list, and an out-of-range slice exits before testing anything.
gzliudan
force-pushed
the
ci-race-detector
branch
from
October 27, 2026 16:38
9cc943d to
1a9b624
Compare
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Summary
Adds a
racejob to CI, plus a matchingmake race-test, so the short suite runs under the detector. #2700 asks for at least one test entry point with-race; every existing one -make test,make quick-test, the eight shards - callsgo run build/ci.go test, and none passes the flag.build/ci.gohas carried it since geth ethereum#23480 (da3da7c0e7, a Travis cron job upstream); nothing here ever switched it on. Closes #2700 - by hand: closing keywords only fire on merges into the default branch (main), so this one will not close itself when it lands ondev-upgrade.tag_buildwaits for the job, so a release cannot ship with a known race.devnet_build_pushkeeps waiting fortestsonly: it fires on every merge, and a flaky detector would hold up the devnet image each time.Exclusions. One carve-out:
rpc- not a race:TestClientWebsocketLargeMessagemoves 32 MiB and gives up on the codec's 30s pong watchdog while four packages run at once. Alone it is green in 26s, so it runs in a step of its own with the detector still on.Short mode.
-shortskips the long consensus cases (skipLongInShortModeunderconsensus/tests/engine_v2_tests), thep2p/discoverUDP/lookup cases and thetests/performance cases. The plain shards run all of them without-race, somake race-test-fullruns the non-short sweep locally when a change touches those paths.One definition. The lane steps call
make race-test-bulk RACE_SHARD=n RACE_SHARDS=4andmake race-test-rpc, so the shard split, the carve-out and the package list live in the Makefile only, andmake race-test(one shard) runs exactly what CI runs. The target collects the package list before testing it: an argument list drops the substitution's exit status, and an empty slice would sendbuild/ci.goback to./...withrpcin it, so the slice is checked before it is used;makeuses/bin/sh, which has nopipefail, so that check coversgo listfailing outright.AGENTS.mdmentions the target in its test checklist.Speed. One job spent 8m28s on the pass, 752s of which is test time a twelve-core laptop covers in 5m38s - the rest is compiled with the detector on, and no CI run reused that work. Four things follow:
-p 4overlapping whatever shares a lane; side by side here the four took 46s, 76s, 97s and 122s;actions/cacherestores~/.cache/go-buildper lane, keyed by the dependency hash rather than the commit, so the instrumentation is paid once pergo.sumand not once per push - an entry measures 660 MB unpacked and 156 MB stored, five of them well inside the 10 GB a repository may cache (the cache also holds go's test results, so an unchanged package reports its previous verdict rather than running again);rpccarve-out, where./...is 199 and the rest would add build time and no coverage;timeout-minutes: 12instead of one 25-minute budget, andfail-fast: falsekeeps a flaky lane from cancelling the others.p2p,p2p/enode,bmtandeth/bftwere on the list when the branch started. Every one of their fixes has since merged - #2791 (internal/testlog), #2794 (p2p/enode), #2792/#2796 (bmtremoved) and #2793 (eth/bft, fixed by #2802) - so all four lines are gone and the bulk pass covers the whole tree apart fromrpc.metricsused to need a second, non-short step becauseTestUniformSampleConcurrentUpdateCountskipped itself undertesting.Short(); #2795 (fixed by #2801) removed the guard, so the test now runs in the short pass and that step is gone as well.Evidence
Before - no entry point passes the flag:
git grep -n -- '-race' 72ed9d60ab -- .github Makefile build/ci.goreturns onlybuild/ci.go:249, the flag's own definition. A full sweep at branch start reported 14WARNING: DATA RACEentries and 11 failing tests in four packages.After - the job's commands, run locally on this branch (198 packages, 126 with test files):
Re-run through
make race-testonbaf6d167e9after the review follow-up: exit 0, zeroDATA RACE, rpc ok 23.0s, trie ok 48.8s. Re-run again once the commands moved into the Makefile targets: exit 0, 127 packages ok, zeroDATA RACE, rpc ok 23.0s. With the shards and the cache in place: the four slices, run side by side, report exit 0 and zeroWARNING: DATA RACE(46s/76s/97s/122s), their union equals the unsharded list (diffclean, 128 packages), and an out-of-range slice exits before testing anything.The two packages that left the exclusion list this round pass under the detector on their own as well:
go test -race -short -count=1 ./eth/bft/is ok in 1.7s, and./metrics/is ok in 6.6s withTestUniformSampleConcurrentUpdateCountrunning rather than skipping (5.3s).CI agrees: the race job has completed green at
cfebe7976b,f4388bb619and6d09623fe4(8m42s-11m20s per run).The job also proved the point while the branch was up: run 39379905697 reported two
WARNING: DATA RACEblocks inp2p.(*Server).NodeInfo()viap2p/simulations(TestHTTPSnapshot), and run 39377853847 failedcore.TestReorgDeliversRemovedLogsSynchronouslyunder detector load with no race reported - fixed by #2827 and #2826, both merged on 2026-10-26. A third run hung instead:TestGethClient/TestSubscribePendingTxHashessat until go test's 45m timeout whiletag_buildwaited on this job. Both test steps now carrytimeout-minutes- 25 on the bulk pass (green runs 10m43s and 10m31s) and 5 on the rpc step (36s and 31s). Green at91f5e1d1cd(11m9s).Merge Danger
Door: two-way - CI-only. Reverting means deleting the job and the Makefile target; no data, state, wire format or runtime behaviour moves.
Blast Radius: CI wall-clock and runner minutes. Every push to any branch now also runs the short suite under the detector: one job took 9-11 minutes, the same traffic as five lanes of a couple of minutes each (four bulk slices plus
rpc). Runner minutes go up by roughly a third, because a cold lane repeats some of the compile; the build cache is what keeps that from being five times the work.tag_buildwaits for the job, and the lanes run alongside the eight shards.devnet_build_pushkeeps waiting fortestsonly. The lane steps carrytimeout-minutes(12 and 5), so the worst a hung pass can do is fail after 12 minutes rather than sit out go test's 45m timeout.Types of changes
Put an
✅in the boxes that applyImpacted Components
Which parts of the codebase does this PR touch?
Put an
✅in the boxes that applyNone of the above applies: this changes CI configuration and a Makefile helper, and no Go file is touched.
Checklist
Put an
✅in the boxes once you have confirmed below actions (or provide reasons on not doing so) thatThe N/A covers the runtime items: the job only runs the existing test suite with the detector on, so there is no chain behaviour, wire format or documentation to update.
Local checks for the timeout addition: a YAML parse of
ci.ymlandmake all. No.goorgo.modfile is touched, so gofmt/goimports,make tidyandmake generatehave nothing to do, and the full suite runs in CI on the push itself.Sub-issues of #2700
All closed by work this branch depends on: #2791 (
p2p,internal/testlog), #2794 (p2p/enode), #2792/#2796 (bmt, the package is gone), #2793 (eth/bft, fixed by #2802) and #2795 (metrics, fixed by #2801).The branch no longer names any of them: the exclusion has a single line (
rpc, a watchdog failure rather than a race), and the short pass reaches every test, including the concurrent sample test that used to skip itself.