You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] Flaky: DeterministicBB deterministic_across_runs/swath1_threads8 — run 3 diverges under work_limit #1793
DeterministicBB/DeterministicBBInstanceTest.deterministic_across_runs/swath1_threads8
fails intermittently in conda-cpp-tests. 1 of 528 tests; the rest pass.
Seen on run 32773153161
(13.3.0, 3.13, arm64, ubuntu26.04, l4, latest-driver, latest-deps). Surfaced on #1763, which changes only .github/workflows/*.yaml for wheel publishing, so it
is not caused by that PR.
What happened
Params: ("/mip/swath1.mps", 8, 60, 4) — 8 CPU threads, time_limit=60, work_limit=4. Seed for this occurrence: 480747044 (the test draws from std::random_device per run and prints it, so each CI run exercises a different
seed).
The test solves the same problem three times with the same seed and compares.
Runs 1 and 2 agreed exactly; run 3 diverged:
Run
Nodes
Simplex iterations
1
1004
10931
2
1004
10931
3
986
10364
Zero Run 1 vs 2 assertion failures, 128 Run 1 vs 3 failures.
determinism_test.cu:217: Failure
solution1.get_objective_value() Which is: 382.30780375000012
solution3.get_objective_value() Which is: 384.26733044999986
determinism_test.cu:220: Failure
solution1.get_solution_bound() Which is: 341.0518313
solution3.get_solution_bound() Which is: 341.16572324999987
Why this looks like a real bug rather than tolerance
All three runs terminated on the work limit, not the clock:
Work limit reached. Stopping the solver...
Explored 986 nodes (10364 simplex iterations) in 3.55s.
Best objective 3.842673e+02, best bound 3.411657e+02, gap 11.22%.
work_limit exists precisely so termination does not depend on wall time. Three
runs with identical settings and seed consuming different amounts of work before
the counter trips means the divergence is upstream of termination — the search
itself took a different path — rather than a formatting or floating-point
tolerance question. The differing node and simplex-iteration counts point the
same way.
The failure is also not marginal: variable 1 differs by 2.0
(1.78e-14 vs 2.0) and variable 11 by 1.0 (1.0 vs 0), i.e. different
integer assignments, not last-bit noise.
Notes
Reproducing needs the seed pinned; the test's use of std::random_device
means a rerun exercises a different seed and will often pass.
Two of three runs matching suggests a race that usually resolves the same way
— consistent with 8 threads and CUOPT_MODE_DETERMINISTIC.
[BUG] Flaky Determinstic B&B test #1134 ([BUG] Flaky Determinstic B&B test, closed 2026-04-24) was the same
test with a different instance (seymour1_threads16), so this class of
failure has recurred.
Another occurrence, on run 33427068595, same shard (13.3.0, 3.13, arm64, ubuntu26.04, l4, latest-driver, latest-deps). Surfaced on #1778, which only changes logging, and the same shard passed on that PR's previous head with equivalent code — so not caused by it.
Worth noting this run failed two tests in NUMOPT_INTERNAL_TEST, not just the parameterized one this issue names:
reproducible_solution_vector compares two solution vectors element-wise and diverges on adjacent binaries:
cpp/tests/mip/determinism_test.cu:44: Failure
Expected equality of these values:
x1[i] Which is: -0
x2[i] Which is: 1
Variable 6717 differs
Seeds drawn in this run: 1957747793, 424238335, 719885386.
That it is not confined to the work_limit=4 parameterization may narrow things down — reproducible_solution_vector is a separate test hitting the same underlying nondeterminism, so a fix aimed only at the swath1_threads8 case would likely leave this one.
🔔 Hi @anandhkb@akifcorduk, this issue has had no activity for 21 days. Please update or let us know if it can be closed. Thank you!
If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.
Activity
ramakrishnap-nv commentedon Aug 31, 2026
Another occurrence, on run 33427068595, same shard (
13.3.0, 3.13, arm64, ubuntu26.04, l4, latest-driver, latest-deps). Surfaced on #1778, which only changes logging, and the same shard passed on that PR's previous head with equivalent code — so not caused by it.Worth noting this run failed two tests in
NUMOPT_INTERNAL_TEST, not just the parameterized one this issue names:reproducible_solution_vectorcompares two solution vectors element-wise and diverges on adjacent binaries:Seeds drawn in this run: 1957747793, 424238335, 719885386.
That it is not confined to the
work_limit=4parameterization may narrow things down —reproducible_solution_vectoris a separate test hitting the same underlying nondeterminism, so a fix aimed only at theswath1_threads8case would likely leave this one.github-actions commentedon Sep 22, 2026
🔔 Hi @anandhkb @akifcorduk, this issue has had no activity for 21 days. Please update or let us know if it can be closed. Thank you!
If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.