Skip to content

[LXC] RHEL setup is now correctly documented and supported in CI - #1316

Merged
Elliot (theelliotm) merged 13 commits into
mainfrom
user/emichlin/fix-rhel-lxc-failure
Sep 29, 2026
Merged

Elliot (theelliotm) merged 13 commits into
mainfrom
user/emichlin/fix-rhel-lxc-failure

Conversation

@theelliotm

@theelliotm Elliot (theelliotm) commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

📖 Description

This PR updates documentation and CI image setup for RHEL (and similar distros). After investigating LXC failures on RHEL systems, it became clear that firewalld wasn't properly configured with lxcbr0 in a trusted zone. CI now uses the trusted zone, while documentation recommends picking any zone as long as lxcbr0 has the desired access.

Other changes:

  • Peer listeners were consolidated into tests/scripts/lib/lxc_peer_listener.sh.
  • During container cleanup, expected "errors" (like missing ip keychains or rules) no longer get logged to sysout. A missing rule is expected after it has been deleted. 🙂 Avoids clogging up sysout with multiple long irrelevant error messages.

🔗 References

Resolves #1274

🔍 Validation

Validated in CI: https://github.com/microsoft/mxc/actions/runs/36478382638/job/109119872066

✅ Checklist

📋 Issue Type

  • Bug fix
  • Feature
  • Task
Microsoft Reviewers: Open in CodeFlow

Elliot (theelliotm) and others added 9 commits September 25, 2026 15:37
Containers on RHEL 10 came up with an IPv6 address but never an IPv4
lease, so every LXC network test failed its readiness gate. The image
runs firewalld with an nftables backend, whose default zone permits
router advertisement but rejects IPv4 DHCP on the bridge. lxc-net's own
accept rules cannot override that: nftables evaluates every base chain
registered at a hook, so a reject in the firewalld chain still applies.

Host preparation now assigns the bridge to the trusted zone at runtime,
and the LXC guide records the same step for developer machines.

The NAT check alongside it derived the subnet from the bridge's host
address and looked for a POSTROUTING rule matching 10.0.3.1, which
lxc-net never writes -- it matches the network. The check therefore
never found the existing rule and appended a duplicate on every run.

Separately, the peer-backed network tests started a listener, slept a
second, and treated `kill -0` as proof it was serving. A background
child that has already exited remains a zombie, so that check succeeds
on a listener that never bound, and the fixed sleep turns interpreter
start-up on a loaded host into a test failure. They now poll the socket
and report what the listener wrote when it stays unreachable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Every container run logged six iptables failures while installing its
inbound chain and two more while removing it, on runs that succeeded.

Both sets are the designed outcome. A fresh container has no chain to
reset, so the pre-install reset is expected to report one missing, and
teardown drains INPUT references until iptables reports none left --
that final error is how the drain knows it is done. Only the caller can
tell those apart from a real error, but the runner logged every non-zero
exit before the caller ever classified it, so a genuine teardown failure
read exactly like the eight routine ones surrounding it.

The runner is now silent and each caller logs what it judges to be a
failure. A reset that does remove something says so, because a chain
present there escaped an earlier teardown and is worth reporting.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The diagnostics block, the standalone workflow and the matrix entry existed
only to reproduce the firewalld bridge failure in CI. The fix is verified, so
the matrix returns to its committed state.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
lxc-net installs masquerade for the bridge subnet as part of bringing the
bridge up, choosing nft or iptables at startup. On a host where nft is usable
it writes its own table, which iptables -t nat -S POSTROUTING does not list,
so the probe here concluded NAT was absent and appended a rule every run.

start_lxc_bridge only ever starts lxc-net, so the bridge cannot come up without
the accompanying NAT rule, and the check has nothing left to cover.

Document the Red Hat setup the CI host performs, so the firewalld zone step and
the EPEL prerequisite are discoverable outside the script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restores the standalone workflow and the matrix entry, and reports the bridge
NAT state so the run shows which backend lxc-net used. Revert before merging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Widens the enabled plan to Debian 13 and both Ubuntu pools alongside RHEL 10,
so the NAT change is exercised where lxc-net picks iptables as well as where it
picks nft. Revert before merging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The standalone workflow, the enabled matrix entries and the bridge NAT
diagnostics existed only to exercise the LXC lane while the fix was in
progress. The matrix returns to its committed state.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings September 28, 2026 21:09
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot AI review requested due to automatic review settings September 28, 2026 23:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The changes are consistent, covered by targeted tests, and the referenced RHEL validation completed with 29 passed and no failures or skips.

Review effort: Balanced
Findings: None

@theelliotm
Elliot (theelliotm) marked this pull request as ready for review September 28, 2026 23:24
@theelliotm
Elliot (theelliotm) requested a review from a team as a code owner September 28, 2026 23:24
Comment thread src/backends/lxc/common/src/network_ingress.rs
Copilot AI balanced review requested due to automatic review settings September 29, 2026 17:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The changes are coherent, tested, syntactically valid, and the referenced RHEL CI run completed successfully.

Review effort: Balanced
Findings: None

@theelliotm
Elliot (theelliotm) merged commit 1f86ae5 into main Sep 29, 2026
31 checks passed
@theelliotm
Elliot (theelliotm) deleted the user/emichlin/fix-rhel-lxc-failure branch September 29, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LXC: Network tests in CI fail due to not receiving an IPv4 address

3 participants