Skip to content

ci: balance the test shards and stop paying for what nothing reads - #2912

Open
gzliudan wants to merge 1 commit into
XinFinOrg:dev-upgradefrom
gzliudan:ci-speed-tests
Open

gzliudan wants to merge 1 commit into
XinFinOrg:dev-upgradefrom
gzliudan:ci-speed-tests

Conversation

@gzliudan

@gzliudan gzliudan commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Proposed changes

Summary

The tests job costs 29 runner-minutes and 10m32s of wall clock, because its eight shards are cut by the first letter of the import path: C-[n-o] holds 67 of the 199 packages and 10m32s, J-N holds 8 and 2m14s, and the job's wall clock is its slowest shard. This cuts the shards where the measured test time balances, and removes three things every one of the eight lanes was paying for.

 jobs:
+  static:               # gofmt, go mod tidy -diff, go generate - once, beside the shards
   tests:                # 8 lanes: no setup steps, no -coverage, go-build cache
   tag_build:            # needs: [tests, static]
   devnet_build_push:    # needs: [tests, static]

Shards. Slices of go list ./..., cut where the measured test time balances out, apart from the two packages that cost more than a slice on their own:

lane content local -count=1
root..bitutil 40 packages, XDCx .. common/bitutil 20s
compiler..randomize 39, common/compiler .. contracts/randomize 41s
randomize..txpool 18, contracts .. core/txpool 40s
txpool..downloader 20, core/txpool/legacypool .. eth/downloader 45s
ethconfig..enode 56, eth/ethconfig .. p2p/enode 34s
enr..version 24, p2p/enr .. version 44s
engine_v2_tests 1 216s
p2p/discover 1 111s

make test-shard SHARD_FROM=.. SHARD_TO=.. is the single definition of the list, and it filters those two packages out by name, so a package added to the tree cannot swallow one of them; the last range is open-ended. test-shard also refuses an empty slice, which would otherwise send build/ci.go back to ./....

-coverage is gone. Nothing reads the profiles: no codecov configuration, no upload-artifact, no later step. -covermode=atomic makes every test pay for counters that are then discarded.

static job. gofmt, go mod tidy -diff and go run build/ci.go generate ran in each of the eight lanes, and the last of them unpacks protoc-gen-go from the tools cache. They run once now, beside the shards; tag_build and devnet_build_push wait for that job as well as for the shards.

Build cache. The lanes restore ~/.cache/go-build, keyed by go.sum instead of the commit, so a slice compiles its dependency graph when the dependencies change rather than on every push - the same shape as #2797. Go keeps test results in the same cache, so a package whose inputs did not move re-reports its previous verdict.

Evidence

Before, run 40274777942: A-B 1m51s, C-[a-m] 2m23s, C-[n-o] 10m32s, C-[p-z] 1m35s, D-I 4m03s, J-N 2m14s, O-S 4m11s, T-Z 2m10s - 29 minutes of runner time to deliver a 10m32s job. The C-[p-z] lane spends 94 of its 95 seconds on setup, which is what moving the setup steps out removes.

After, twelve cores, every lane at once, -count=1: 20s / 41s / 40s / 45s / 34s / 44s / 216s / 111s.

After, from CI (run 40548209650, all eight lanes green): 17s / 27s / 27s / 61s / 86s / 119s / 129s / 5m25s, so the job's wall clock is now 5m25s against 10m32s, and the eight lanes cost about 13 runner-minutes against 29. The remaining tail is consensus/tests/engine_v2_tests on its own - a single package of ~100 test functions that no shard can subdivide; splitting it by test name would take the job to about two and a half minutes.

Partition check: the eight lanes cover go list ./... exactly - 199 packages, no duplicates, no gap - and each lane's command exits 0 locally.

Merge Danger

Door: two-way, CI only. Reverting means restoring the letter shards and the setup steps; no data, state, wire format or runtime behaviour moves.

Blast Radius: coverage is no longer collected, and a formatting or generation failure now blocks tag_build and devnet_build_push through the static job rather than through every test lane. Wall clock drops from 10m32s to 5m25s and runner time from ~29 minutes to ~13; the engine_v2_tests lane is the floor until that package is split by test name. Nothing else about what the job proves changes.

Types of changes

  • build: Changes that affect the build system or external dependencies
  • ci: Changes to CI configuration files and scripts

Checklist

  • I have read the contributing guidelines.
  • I have made corresponding changes to the documentation
  • The changes are covered by tests
  • All new and existing tests passed

@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: fc7bb3d2-7d63-4900-82d9-dbc9d6d91d3f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gzliudan
gzliudan force-pushed the ci-speed-tests branch 2 times, most recently from 55288d8 to e77e2fd Compare October 27, 2026 17:11
The eight shards were cut by first letter of the import path, which works as
a way to find a package and not as a way to split the work: C-[n-o] took
10m32s of the 29 minutes the suite costs while J-N took 2m14s, and the job's
wall clock is its slowest shard.

They are now slices of `go list ./...` cut where the measured test time
balances out - 20s to 45s each, from per-package timings of a full -count=1
run - apart from consensus/tests/engine_v2_tests (216s) and p2p/discover
(111s), which cost more than a slice each and run by path. `make test-shard`
takes a line range of the list, and it filters those two out by name, so a
package added to the tree cannot swallow one of them; the last range is
open-ended.

`-coverage` is gone: there is no codecov configuration, no artefact upload,
nothing reads the profiles, and -covermode=atomic makes every test pay for
counters that are then discarded.

Each shard also repeated gofmt, `go mod tidy -diff` and `go run build/ci.go
generate`, the last of which unpacks protoc-gen-go from the tools cache.
Those move to a `static` job that runs beside the shards; tag_build and
devnet_build_push now wait for it as well as for the shards.

The shards cache ~/.cache/go-build, keyed by go.sum instead of the commit, so
a slice compiles its dependency graph when the dependencies change rather
than on every push. Go keeps test results in the same cache, so an unchanged
package re-reports its previous verdict.

Measured on twelve cores with every lane running at once, -count=1: 18s to
294s for the first cut, which put consensus/tests/engine_v2_tests and
p2p/discover in the same lane as 35 other packages; the split above isolates
them, and the six ranges run in 20-60s. A hosted runner pays about 20s of
setup and roughly twice the local test time, so the job should land near four
minutes instead of ten and a half.

Verified: the eight lanes partition `go list ./...` exactly - 199 packages, no
overlap, no gap - each lane's command exits 0 locally, and no .go file is
touched.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant