Skip to content

[BUG] Flaky: DeterministicBB deterministic_across_runs/swath1_threads8 — run 3 diverges under work_limit #1793

Open

Description

@ramakrishnap-nv

DeterministicBB/DeterministicBBInstanceTest.deterministic_across_runs/swath1_threads8
fails intermittently in conda-cpp-tests. 1 of 528 tests; the rest pass.

Seen on run 32773153161
(13.3.0, 3.13, arm64, ubuntu26.04, l4, latest-driver, latest-deps). Surfaced on
#1763, which changes only .github/workflows/*.yaml for wheel publishing, so it
is not caused by that PR.

What happened

Params: ("/mip/swath1.mps", 8, 60, 4) — 8 CPU threads, time_limit=60,
work_limit=4. Seed for this occurrence: 480747044 (the test draws from
std::random_device per run and prints it, so each CI run exercises a different
seed).

The test solves the same problem three times with the same seed and compares.
Runs 1 and 2 agreed exactly; run 3 diverged:

Run Nodes Simplex iterations
1 1004 10931
2 1004 10931
3 986 10364

Zero Run 1 vs 2 assertion failures, 128 Run 1 vs 3 failures.

determinism_test.cu:217: Failure
  solution1.get_objective_value()  Which is: 382.30780375000012
  solution3.get_objective_value()  Which is: 384.26733044999986

determinism_test.cu:220: Failure
  solution1.get_solution_bound()   Which is: 341.0518313
  solution3.get_solution_bound()   Which is: 341.16572324999987

Why this looks like a real bug rather than tolerance

All three runs terminated on the work limit, not the clock:

Work limit reached. Stopping the solver...
Explored 986 nodes (10364 simplex iterations) in 3.55s.
Best objective 3.842673e+02, best bound 3.411657e+02, gap 11.22%.

work_limit exists precisely so termination does not depend on wall time. Three
runs with identical settings and seed consuming different amounts of work before
the counter trips means the divergence is upstream of termination — the search
itself took a different path — rather than a formatting or floating-point
tolerance question. The differing node and simplex-iteration counts point the
same way.

The failure is also not marginal: variable 1 differs by 2.0
(1.78e-14 vs 2.0) and variable 11 by 1.0 (1.0 vs 0), i.e. different
integer assignments, not last-bit noise.

Notes

Activity

  1. ramakrishnap-nv commented on Aug 31, 2026

    @ramakrishnap-nv
    CollaboratorAuthor

    Another occurrence, on run 33427068595, same shard (13.3.0, 3.13, arm64, ubuntu26.04, l4, latest-driver, latest-deps). Surfaced on #1778, which only changes logging, and the same shard passed on that PR's previous head with equivalent code — so not caused by it.

    Worth noting this run failed two tests in NUMOPT_INTERNAL_TEST, not just the parameterized one this issue names:

    [  FAILED  ] DeterministicBB/DeterministicBBInstanceTest.deterministic_across_runs/swath1_threads8, where GetParam() = ("/mip/swath1.mps", 8, 60, 4)
    [  FAILED  ] DeterministicBBTest.reproducible_solution_vector
    

    reproducible_solution_vector compares two solution vectors element-wise and diverges on adjacent binaries:

    cpp/tests/mip/determinism_test.cu:44: Failure
    Expected equality of these values:
      x1[i]  Which is: -0
      x2[i]  Which is: 1
    Variable 6717 differs
    

    Seeds drawn in this run: 1957747793, 424238335, 719885386.

    That it is not confined to the work_limit=4 parameterization may narrow things down — reproducible_solution_vector is a separate test hitting the same underlying nondeterminism, so a fix aimed only at the swath1_threads8 case would likely leave this one.

  2. github-actions commented on Sep 22, 2026

    @github-actions

    🔔 Hi @anandhkb @akifcorduk, this issue has had no activity for 21 days. Please update or let us know if it can be closed. Thank you!

    If this is an "epic" issue, then please add the "epic" label to this issue.
    If it is a PR and not ready for review, then please convert this to draft.
    If you just want to switch off this notification, then use the "skip inactivity reminder" label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions