enable merge queues

[cgwalters]

Basically merge queues are the next logical step after going to automerge; they can help us solve two things:

• PR 1 and PR 2 pass tests but break when merged together (rare but happens)
• Reduce impact of flaky tests if we set "Only merge non-failing pull requests" to No (effectively the inverse of the above - PR 1 and PR 2 will merge if they pass tests together)

Plan:

• Change all of our actions to enable merge_group
• Test it out on one repo (maybe this one? ci-sandbox?)
• Investigate packit: https://packit.dev/docs/configuration mentions this so hopefully it can be made to work
• Try it out on bootc

[cgwalters]

OK I set this up on https://github.com/containers/composefs-rs/ and it seems to be working OK. Next step: all the repos ehre

[hone]

I was pointed to merge queue is susceptible to GitHub incidents as a potential thing to watch out for. I think it's still low risk.

[cgwalters]

Ouch, thanks for pointing that out!

It is indeed a risk, but...a while back when I was working on OpenShift core dev we had an infra issue which temporarily made our Prow instance (that effectively implemented a merge queue strategy before Github supported it natively) merge PRs that weren't passing tests. It was a huge mess.

There's always a tension here in basically how much one outsources infrastructure.

Note that in not using merge queues, if one is merging PRs one by one "manually" then it's absolutely possible to hit the "A xor B" problem, that just happened recently in bootc-dev/bcvk#238 (review) to us.

I hope Github ensures after this incident they have really good tests for their merge queues at least. Running a merge queue externally just shifts risk around I think.

[cgwalters]

Now enabled in bootc!

[cgwalters]

Note from discussion: We can generally shift CI focus/weight to the merge_group target, and e.g. there's generally much less need for postsubmit "push to main" jobs for example, and pull_request can be lighter weight

[cgwalters]

Investigating this now,

Note from discussion: We can generally shift CI focus/weight to the merge_group target, and e.g. there's generally much less need for postsubmit "push to main" jobs for example, and pull_request can be lighter weight

We actually need to do this with our required_checks now to really gain benefit.

[cgwalters]

OK so...I had a fundamental misunderstanding here, I thought merge queues could serve the same role as e.g. https://github.com/rust-lang/bors

And they do have the "tree is always green" property.

But they don't actually help reduce CI cost by batching expensive tests in a group.

[cgwalters]

OK I created https://github.com/cgwalters/merge-queue-like-bors after some experimentation...may look at deploying this on bootc.

[cgwalters]

Another big downside of merge queues I've stumbled upon is that one can't restart/retry a single job that failed due to an infrastructure flake like one can do with normal jobs that run on a pull request etc.

I think ultimately we're going to need to refactor our build system to fetch all dependencies up front with retries.