Skip to content

ci: run the short suite under the race detector, close #2700 - #2797

Merged
gzliudan merged 1 commit into
XinFinOrg:dev-upgradefrom
gzliudan:ci-race-detector
Oct 27, 2026
Merged

gzliudan merged 1 commit into
XinFinOrg:dev-upgradefrom
gzliudan:ci-race-detector

Conversation

@gzliudan

@gzliudan gzliudan commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Proposed changes

Summary

Adds a race job to CI, plus a matching make race-test, so the short suite runs under the detector. #2700 asks for at least one test entry point with -race; every existing one - make test, make quick-test, the eight shards - calls go run build/ci.go test, and none passes the flag. build/ci.go has carried it since geth ethereum#23480 (da3da7c0e7, a Travis cron job upstream); nothing here ever switched it on. Closes #2700 - by hand: closing keywords only fire on merges into the default branch (main), so this one will not close itself when it lands on dev-upgrade.

 jobs:
   tests:              # 8 shards; gofmt, tidy, generate; no -race
+  race:               # the short suite, detector on: lanes bulk 1/4 .. 4/4 and rpc
+    - make race-test-bulk RACE_SHARD=<1..4> RACE_SHARDS=4   # a quarter of the packages each
+    - make race-test-rpc                                    # rpc alone: 32 MiB vs the 30s pong watchdog
+    - actions/cache on ~/.cache/go-build                    # the detector recompiles every dependency
   tag_build:          # needs: [tests, race]
   devnet_build_push:  # needs: tests

tag_build waits for the job, so a release cannot ship with a known race. devnet_build_push keeps waiting for tests only: it fires on every merge, and a flaky detector would hold up the devnet image each time.

Exclusions. One carve-out:

  • rpc - not a race: TestClientWebsocketLargeMessage moves 32 MiB and gives up on the codec's 30s pong watchdog while four packages run at once. Alone it is green in 26s, so it runs in a step of its own with the detector still on.

Short mode. -short skips the long consensus cases (skipLongInShortMode under consensus/tests/engine_v2_tests), the p2p/discover UDP/lookup cases and the tests/ performance cases. The plain shards run all of them without -race, so make race-test-full runs the non-short sweep locally when a change touches those paths.

One definition. The lane steps call make race-test-bulk RACE_SHARD=n RACE_SHARDS=4 and make race-test-rpc, so the shard split, the carve-out and the package list live in the Makefile only, and make race-test (one shard) runs exactly what CI runs. The target collects the package list before testing it: an argument list drops the substitution's exit status, and an empty slice would send build/ci.go back to ./... with rpc in it, so the slice is checked before it is used; make uses /bin/sh, which has no pipefail, so that check covers go list failing outright. AGENTS.md mentions the target in its test checklist.

Speed. One job spent 8m28s on the pass, 752s of which is test time a twelve-core laptop covers in 5m38s - the rest is compiled with the detector on, and no CI run reused that work. Four things follow:

  • the bulk pass runs as four parallel lanes, each taking a slice of the name-sorted package list modulo four (32 packages), with -p 4 overlapping whatever shares a lane; side by side here the four took 46s, 76s, 97s and 122s;
  • actions/cache restores ~/.cache/go-build per lane, keyed by the dependency hash rather than the commit, so the instrumentation is paid once per go.sum and not once per push - an entry measures 660 MB unpacked and 156 MB stored, five of them well inside the 10 GB a repository may cache (the cache also holds go's test results, so an unchanged package reports its previous verdict rather than running again);
  • only packages with test files are listed - 128 after the rpc carve-out, where ./... is 199 and the rest would add build time and no coverage;
  • each lane carries timeout-minutes: 12 instead of one 25-minute budget, and fail-fast: false keeps a flaky lane from cancelling the others.

p2p, p2p/enode, bmt and eth/bft were on the list when the branch started. Every one of their fixes has since merged - #2791 (internal/testlog), #2794 (p2p/enode), #2792/#2796 (bmt removed) and #2793 (eth/bft, fixed by #2802) - so all four lines are gone and the bulk pass covers the whole tree apart from rpc. metrics used to need a second, non-short step because TestUniformSampleConcurrentUpdateCount skipped itself under testing.Short(); #2795 (fixed by #2801) removed the guard, so the test now runs in the short pass and that step is gone as well.

Evidence

Before - no entry point passes the flag: git grep -n -- '-race' 72ed9d60ab -- .github Makefile build/ci.go returns only build/ci.go:249, the flag's own definition. A full sweep at branch start reported 14 WARNING: DATA RACE entries and 11 failing tests in four packages.

After - the job's commands, run locally on this branch (198 packages, 126 with test files):

bulk (everything but rpc)   exit 0, 0 WARNING: DATA RACE, 3m49s
rpc (alone)                 ok 26.0s

Re-run through make race-test on baf6d167e9 after the review follow-up: exit 0, zero DATA RACE, rpc ok 23.0s, trie ok 48.8s. Re-run again once the commands moved into the Makefile targets: exit 0, 127 packages ok, zero DATA RACE, rpc ok 23.0s. With the shards and the cache in place: the four slices, run side by side, report exit 0 and zero WARNING: DATA RACE (46s/76s/97s/122s), their union equals the unsharded list (diff clean, 128 packages), and an out-of-range slice exits before testing anything.

The two packages that left the exclusion list this round pass under the detector on their own as well: go test -race -short -count=1 ./eth/bft/ is ok in 1.7s, and ./metrics/ is ok in 6.6s with TestUniformSampleConcurrentUpdateCount running rather than skipping (5.3s).

CI agrees: the race job has completed green at cfebe7976b, f4388bb619 and 6d09623fe4 (8m42s-11m20s per run).

The job also proved the point while the branch was up: run 39379905697 reported two WARNING: DATA RACE blocks in p2p.(*Server).NodeInfo() via p2p/simulations (TestHTTPSnapshot), and run 39377853847 failed core.TestReorgDeliversRemovedLogsSynchronously under detector load with no race reported - fixed by #2827 and #2826, both merged on 2026-10-26. A third run hung instead: TestGethClient/TestSubscribePendingTxHashes sat until go test's 45m timeout while tag_build waited on this job. Both test steps now carry timeout-minutes - 25 on the bulk pass (green runs 10m43s and 10m31s) and 5 on the rpc step (36s and 31s). Green at 91f5e1d1cd (11m9s).

Merge Danger

Door: two-way - CI-only. Reverting means deleting the job and the Makefile target; no data, state, wire format or runtime behaviour moves.

Blast Radius: CI wall-clock and runner minutes. Every push to any branch now also runs the short suite under the detector: one job took 9-11 minutes, the same traffic as five lanes of a couple of minutes each (four bulk slices plus rpc). Runner minutes go up by roughly a third, because a cold lane repeats some of the compile; the build cache is what keeps that from being five times the work. tag_build waits for the job, and the lanes run alongside the eight shards. devnet_build_push keeps waiting for tests only. The lane steps carry timeout-minutes (12 and 5), so the worst a hung pass can do is fail after 12 minutes rather than sit out go test's 45m timeout.

Types of changes

Put an ✅ in the boxes that apply

  • build: Changes that affect the build system or external dependencies
  • ci: Changes to CI configuration files and scripts
  • chore: Changes that don't change source code or tests
  • docs: Documentation only changes
  • feat: A new feature
  • fix: A bug fix
  • perf: A code change that improves performance
  • refactor: A code change that neither fixes a bug nor adds a feature
  • revert: Revert something
  • style: Changes that do not affect the meaning of the code
  • test: Adding missing tests or correcting existing tests

Impacted Components

Which parts of the codebase does this PR touch?
Put an ✅ in the boxes that apply

  • Consensus
  • Account
  • Network
  • Geth
  • Smart Contract
  • External components
  • Not sure (Please specify below)

None of the above applies: this changes CI configuration and a Makefile helper, and no Go file is touched.

Checklist

Put an ✅ in the boxes once you have confirmed below actions (or provide reasons on not doing so) that

  • This PR has sufficient test coverage (unit/integration test) OR I have provided reason in the PR description for not having test coverage
  • Tested on a private network from the genesis block and monitored the chain operating correctly for multiple epochs.
  • Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet.
  • Tested the backwards compatibility.
  • Tested with XDC nodes running this version co-exist with those running the previous version.
  • Relevant documentation has been updated as part of this PR
  • N/A

The N/A covers the runtime items: the job only runs the existing test suite with the detector on, so there is no chain behaviour, wire format or documentation to update.

Local checks for the timeout addition: a YAML parse of ci.yml and make all. No .go or go.mod file is touched, so gofmt/goimports, make tidy and make generate have nothing to do, and the full suite runs in CI on the push itself.

Sub-issues of #2700

All closed by work this branch depends on: #2791 (p2p, internal/testlog), #2794 (p2p/enode), #2792/#2796 (bmt, the package is gone), #2793 (eth/bft, fixed by #2802) and #2795 (metrics, fixed by #2801).

The branch no longer names any of them: the exclusion has a single line (rpc, a watchdog failure rather than a race), and the short pass reaches every test, including the concurrent sample test that used to skip itself.

@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4497638e-a2ae-4682-b21e-e52dc09d40b5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gzliudan
gzliudan force-pushed the ci-race-detector branch 2 times, most recently from f4388bb to 6d09623 Compare October 18, 2026 12:34
@gzliudan
gzliudan force-pushed the ci-race-detector branch 2 times, most recently from 62a887e to ff07e3c Compare October 20, 2026 06:21
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 20, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel, with a timeout
so a reorg that never sends still fails, and compare its log count against
the chain that was reverted.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s. make all,
go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 20, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel, with a timeout
so a reorg that never sends still fails, and compare its log count against
the chain that was reverted.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s. make all,
go test ./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 20, 2026
SimNode.Start registers the services before the devp2p node is started, and
node.RegisterProtocols appends to the p2p server's protocol slice in that
window. The simulations HTTP API marshals the node info from another
goroutine: the event stream does it as soon as a control event is handed
over, and the node handlers do it on request. Neither side takes a lock, so a
node that is created and then started races with the HTTP layer reading the
protocol slice. The new race job in XinFinOrg#2797 reported it as TestHTTPSnapshot
failing on p2p.Server.NodeInfo reading srv.Protocols against
node.RegisterProtocols appending to it.

Hold the startup lock for the whole of Start and take it shared in NodeInfo.
The lock order stays startMu -> node.lock, the only other lock in the path,
and nothing reached from Start calls NodeInfo, so readers wait for the
startup instead of observing a half-filled server. The change is confined to
the in-memory adapter and leaves the event and node response payloads alone.

Upstream geth deleted p2p/simulations in ethereum#30250 without fixing this, and both
NodeInfo and RegisterProtocols are still unsynchronised there, so there is
nothing to port.

Tests: 100 runs of TestHTTPSnapshot as four concurrent processes under -race
(short, count=25 each) report no race; dropping the read lock again makes 3
of the 4 processes fail, so the lock is what keeps it quiet. make all, go
test ./p2p/simulations/... pass, golangci-lint 2.12.2 on
./p2p/simulations/... reports no new findings, and make tidy reports no
untidy module files.
@gzliudan
gzliudan force-pushed the ci-race-detector branch 2 times, most recently from a110646 to 786812a Compare October 20, 2026 07:54
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel and compare its
log count against the chain that was reverted. Both ends of that channel are
bounded by a timeout, so a reorg that never sends fails the test without
stranding the subscriber goroutine.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s, and dropping the
send altogether fails it within the timeout while the subscriber goroutine
leaves the receive. make all, go test ./core and go run build/ci.go lint
./core/... (no new findings) pass.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 22, 2026
SimNode.Start registers the services before the devp2p node is started, and
node.RegisterProtocols appends to the p2p server's protocol slice in that
window. The simulations HTTP API marshals the node info from another
goroutine: the event stream does it as soon as a control event is handed
over, and the node handlers do it on request. Neither side takes a lock, so a
node that is created and then started races with the HTTP layer reading the
protocol slice. The new race job in XinFinOrg#2797 reported it as TestHTTPSnapshot
failing on p2p.Server.NodeInfo reading srv.Protocols against
node.RegisterProtocols appending to it.

Hold the startup lock for the whole of Start and take it shared in NodeInfo.
The lock order stays startMu -> node.lock, the only other lock in the path,
and nothing reached from Start calls NodeInfo, so readers wait for the
startup instead of observing a half-filled server. The change is confined to
the in-memory adapter and leaves the event and node response payloads alone.

Upstream geth deleted p2p/simulations in ethereum#30250 without fixing this, and both
NodeInfo and RegisterProtocols are still unsynchronised there, so there is
nothing to port.

Tests: 100 runs of TestHTTPSnapshot as four concurrent processes under -race
(short, count=25 each) report no race; dropping the read lock again makes 3
of the 4 processes fail, so the lock is what keeps it quiet. make all, go
test ./p2p/simulations/... pass, golangci-lint 2.12.2 on
./p2p/simulations/... reports no new findings, and make tidy reports no
untidy module files.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel and compare its
log count against the chain that was reverted. Both ends of that channel are
bounded by a timeout, so a reorg that never sends fails the test without
stranding the subscriber goroutine.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s, and dropping the
send altogether fails it at once on the armed check, with the subscriber
goroutine then leaving the receive within its own timeout. make all, go test
./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel and compare its
log count against the chain that was reverted. Both ends of that channel are
bounded by a timeout, so a reorg that never sends fails the test without
stranding the subscriber goroutine.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s, and dropping the
send altogether fails it at once on the armed check, with the subscriber
goroutine then leaving the receive within its own timeout. make all, go test
./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel and compare its
log count against the chain that was reverted. Both ends of that channel are
bounded by a timeout, so a reorg that never sends fails the test without
stranding the subscriber goroutine.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s, and dropping the
send altogether fails it at once on the armed check, with the subscriber
goroutine then leaving the receive within its own timeout. make all, go test
./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel and compare its
log count against the chain that was reverted. Both ends of that channel are
bounded by a timeout, so a reorg that never sends fails the test without
stranding the subscriber goroutine.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s, and dropping the
send altogether fails it at once on the armed check, with the subscriber
goroutine then leaving the receive within its own timeout. make all, go test
./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 22, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel and compare its
log count against the chain that was reverted. Both ends of that channel are
bounded by a timeout, so a reorg that never sends fails the test without
stranding the subscriber goroutine.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s, and dropping the
send altogether fails it at once on the armed check, with the subscriber
goroutine then leaving the receive within its own timeout. make all, go test
./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 24, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel and compare its
log count against the chain that was reverted. Both ends of that channel are
bounded by a timeout, so a reorg that never sends fails the test without
stranding the subscriber goroutine.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s, and dropping the
send altogether fails it at once on the armed check, with the subscriber
goroutine then leaving the receive within its own timeout. make all, go test
./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 26, 2026
SimNode.Start registers the services before the devp2p node is started, and
node.RegisterProtocols appends to the p2p server's protocol slice in that
window. The simulations HTTP API marshals the node info from another
goroutine: the event stream does it as soon as a control event is handed
over, and the node handlers do it on request. Neither side takes a lock, so a
node that is created and then started races with the HTTP layer reading the
protocol slice. The new race job in XinFinOrg#2797 reported it as TestHTTPSnapshot
failing on p2p.Server.NodeInfo reading srv.Protocols against
node.RegisterProtocols appending to it.

Hold the startup lock for the whole of Start and take it shared in NodeInfo.
The lock order stays startMu -> node.lock, the only other lock in the path,
and nothing reached from Start calls NodeInfo, so readers wait for the
startup instead of observing a half-filled server. The change is confined to
the in-memory adapter and leaves the event and node response payloads alone.

Upstream geth deleted p2p/simulations in ethereum#30250 without fixing this, and both
NodeInfo and RegisterProtocols are still unsynchronised there, so there is
nothing to port.

Tests: 100 runs of TestHTTPSnapshot as four concurrent processes under -race
(short, count=25 each) report no race; dropping the read lock again makes 3
of the 4 processes fail, so the lock is what keeps it quiet. make all, go
test ./p2p/simulations/... pass, golangci-lint 2.12.2 on
./p2p/simulations/... reports no new findings, and make tidy reports no
untidy module files.
gzliudan added a commit to gzliudan/XDPoSChain that referenced this pull request Oct 26, 2026
TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in XinFinOrg#2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel and compare its
log count against the chain that was reverted. Both ends of that channel are
bounded by a timeout, so a reorg that never sends fails the test without
stranding the subscriber goroutine.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s, and dropping the
send altogether fails it at once on the armed check, with the subscriber
goroutine then leaving the receive within its own timeout. make all, go test
./core and go run build/ci.go lint ./core/... (no new findings) pass.
gzliudan added a commit that referenced this pull request Oct 26, 2026
* fix(core): keep the removed-logs delivery test off the scheduler

TestReorgDeliversRemovedLogsSynchronously recorded the arrival in the
subscriber goroutine after the channel handoff, and read that flag once the
reorg returned. An unbuffered handoff only orders the send against the
receive: whatever the subscriber does next runs in parallel with the rest of
the reorg, so a subscriber the scheduler put off for a few hundred
microseconds made a synchronous send look asynchronous. The new race job
(-p 4 on a 4 vCPU runner) failed on it in #2797 with the reorg having waited
the full subscriber delay.

Announce readiness before parking on the receive instead: that write is
sequenced before the handoff, which in turn happens before the send the reorg
waits for returns. Carry the arrival over a buffered channel and compare its
log count against the chain that was reverted. Both ends of that channel are
bounded by a timeout, so a reorg that never sends fails the test without
stranding the subscriber goroutine.

Upstream geth has no equivalent test and sends synchronously as well, so
there is nothing to port.

Tests: 160 runs of the test as four processes pinned to four cores under
GOMAXPROCS=4, 7 of which failed before this change and none after; turning
the send into a goroutine still fails the test within 0.02s, and dropping the
send altogether fails it at once on the armed check, with the subscriber
goroutine then leaving the receive within its own timeout. make all, go test
./core and go run build/ci.go lint ./core/... (no new findings) pass.

* test(core): widen the removed-logs delivery budget

The subscriber delay is not only a stall: it is also how long an
asynchronous send may take before the check can no longer tell it apart
from a synchronous one. reorg hands the removals over near its start
(both rmLogsFeed.Send calls precede the rollback and the head writes), so
a spawned send has to outlast the delay before it hides. At 250ms that
window was ~12x the ~20ms of remaining reorg work measured under -race,
so a slow runner could pass a regression silently.

Raise it to a second, which leaves ~50x, and record the premise next to
the constant: the budget tracks where the send sits in reorg, not the
subscriber. Spawning the send still fails the test within 0.01s, so the
guard is unchanged.

The test and its 250ms budget predate this branch - the previous commit
only changed how the arrival is recorded - so this lands on its own
commit instead of being folded into it.

Tests: make all; go test ./core (24.3s); the test under -race (1.02s);
40 runs as four processes pinned to four cores under GOMAXPROCS=4, none
failing; lint on ./core/... (26 issues the package already had, none in
the diff).
gzliudan added a commit that referenced this pull request Oct 26, 2026
)

SimNode.Start registers the services before the devp2p node is started, and
node.RegisterProtocols appends to the p2p server's protocol slice in that
window. The simulations HTTP API marshals the node info from another
goroutine: the event stream does it as soon as a control event is handed
over, and the node handlers do it on request. Neither side takes a lock, so a
node that is created and then started races with the HTTP layer reading the
protocol slice. The new race job in #2797 reported it as TestHTTPSnapshot
failing on p2p.Server.NodeInfo reading srv.Protocols against
node.RegisterProtocols appending to it.

Hold the startup lock for the whole of Start and take it shared in NodeInfo.
The lock order stays startMu -> node.lock, the only other lock in the path,
and nothing reached from Start calls NodeInfo, so readers wait for the
startup instead of observing a half-filled server. The change is confined to
the in-memory adapter and leaves the event and node response payloads alone.

Upstream geth deleted p2p/simulations in ethereum#30250 without fixing this, and both
NodeInfo and RegisterProtocols are still unsynchronised there, so there is
nothing to port.

Tests: 100 runs of TestHTTPSnapshot as four concurrent processes under -race
(short, count=25 each) report no race; dropping the read lock again makes 3
of the 4 processes fail, so the lock is what keeps it quiet. make all, go
test ./p2p/simulations/... pass, golangci-lint 2.12.2 on
./p2p/simulations/... reports no new findings, and make tidy reports no
untidy module files.
@gzliudan
gzliudan force-pushed the ci-race-detector branch 2 times, most recently from b8684a4 to 2837a13 Compare October 27, 2026 15:41
@gzliudan gzliudan changed the title ci: run the short suite under the race detector ci: run the short suite under the race detector, close #2700 Oct 27, 2026
@gzliudan
gzliudan force-pushed the ci-race-detector branch 3 times, most recently from 130160d to 9cc943d Compare October 27, 2026 16:31
make test, make quick-test and all eight shards of ci.yml call
`go run build/ci.go test`, and none of them passes -race, so an access that
is unsynchronised at run time stays invisible to CI. build/ci.go has carried
the flag since it was ported from geth ethereum#23480 - nothing ever switched it on
(XinFinOrg#2700).

Every package the job started out excluding is fixed (XinFinOrg#2791, XinFinOrg#2793 by
XinFinOrg#2802, XinFinOrg#2794) or gone with the package (XinFinOrg#2792, whose disposition was
deletion via XinFinOrg#2796), so the bulk pass now covers the whole tree. metrics no
longer needs a second, non-short run either: XinFinOrg#2795 (fixed by XinFinOrg#2801) removed
the `testing.Short()` guard from TestUniformSampleConcurrentUpdateCount, so
the short pass reaches it.

rpc gets a slice of its own: the bulk pass runs four packages at a time, and
TestClientWebsocketLargeMessage does not survive that on a GitHub-hosted
runner - twice in a row the job failed after 30.00s with
`websocket: close 1006 (abnormal closure): unexpected EOF`, the websocket
codec's 30s ping/pong watchdog, with no data race reported in either
attempt. The package is green when it does not compete with three other
packages - 26s here, and 24.9s in the plain CI shards, which run one package
at a time - so it leaves the bulk pass, which keeps the detector on the
package instead of dropping it from the job.

The job runs four slices of the bulk pass in parallel and then race-test-rpc,
and the shard split, the rpc carve-out and the package list live in the
Makefile, so make race-test (a single shard) runs exactly what CI runs and a
local sweep predicts the job. It does not pass -failfast: the sweep exists to
report every race in one go, and the first failing package would stop the
rest. AGENTS.md mentions the target in its test checklist.

Four slices because the detector makes the pass compile-bound: one job spent
8m28s on the 752s of test time that a twelve-core laptop covers in 5m38s. A
slice is the name-sorted package list taken modulo four, 32 packages each,
with -p 4 overlapping whatever shares one; side by side here they took 46s,
76s, 97s and 122s. actions/cache restores ~/.cache/go-build per slice,
keyed by the dependency hash rather than the commit, so a slice refills when
go.sum moves instead of once per push: an entry is 660 MB unpacked and 156 MB
stored, and five of them fit inside the 10 GB a repository may cache. That
cache also holds go's test results, so a slice reports a previous verdict for
packages whose inputs did not move. Only packages with
test files are listed: the other 70 would add build time and no coverage.

-short leaves out the long consensus cases (skipLongInShortMode under
consensus/tests/engine_v2_tests), the discover UDP/lookup cases and the tests/
performance cases. The plain shards run those without the detector, so
make race-test-full exists for the local sweep a change to those paths needs.

The target collects the package list before testing it: an argument list
drops the substitution's exit status, and an empty slice would send
build/ci.go back to ./... with rpc in it, so the slice is checked before it
is used. make runs the recipe with /bin/sh, which has no pipefail, so that
check is what covers go list failing outright.

tag_build now waits for the race job as well, so a release artifact cannot
ship with a known race. devnet_build_push keeps waiting for tests only: it
fires on every merge, and a flaky detector would hold up the devnet image each
time.

Verified with the job's command against dev-upgrade 72ed9d6: rc=0, zero
WARNING: DATA RACE, 126 packages ok in 3m49s. rpc on its own is ok in 26.0s,
eth/bft in 1.7s, metrics in 6.6s (the concurrent sample test runs instead of
skipping, 5.3s), make all exits 0, and no .go file is touched. make race-test
against baf6d16 reports rc=0 with zero DATA RACE (rpc 23.0s, trie 48.8s),
and pointing the recipe at a go that cannot list makes it exit instead of
testing the whole tree. With the shards and the cache in place, the four
slices run side by side report rc=0 and zero DATA RACE (46s/76s/97s/122s),
their union equals the unsharded list, and an out-of-range slice exits before
testing anything.
@gzliudan
gzliudan merged commit 855afa6 into XinFinOrg:dev-upgrade Oct 27, 2026
19 checks passed
@gzliudan
gzliudan deleted the ci-race-detector branch October 27, 2026 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants