[PRODENG-3623] Allow vendor change when installing MCR packages on SLES by james-nesbitt · Pull Request #652 · Mirantis/launchpad

james-nesbitt · 2026-08-04T08:27:18Z

What

Interim workaround: on SLES, install the MCR packages (containerd.io, docker-ee) with a direct zypper -n install -y --allow-vendor-change call instead of the generic InstallPackage helper, so the install succeeds on SLES cloud images that ship a distro-vendor containerd pre-installed.

Why

On current SUSE SLES 15 cloud images, containerd is pre-installed from SUSE. The MCR containerd.io package set obsoletes/replaces it, which zypper treats as a vendor change; zypper install -y refuses that non-interactively and exits 4, surfacing as package manager could not install containerd.io. Reproduced on SLES 15 SP7 against both stable-25.0 and stable-29.2; --allow-vendor-change resolves both.

Why this is a workaround, not the final fix

The root cause is in the package-manager layer, not in launchpad: zypper install -y should allow the vendor change. That belongs in k0sproject/rig, whose zypper provider builds the install command. I've raised it upstream:

That upstream change is an RFC (open question on whether --allow-vendor-change should be a default or an opt-in option) and, even if accepted, won't land + tag + get vendored into launchpad in time for the v1.5.18 release. So this PR takes the direct-zypper workaround in the SLES configurer as the interim fix, deliberately bypassing rig's InstallPackage for exactly the calls that need the flag.

Once the rig change lands and is vendored, this workaround should be revisited — reverted back to the generic InstallPackage path (if rig makes it the default) or switched to rig's opt-in option (if that's the shape maintainers choose). Tracking that follow-up under PRODENG-3623 / the upstream PR.

How

Replace the two InstallPackage calls in the SLES InstallMCR path with zypper -n refresh + zypper -n install -y --allow-vendor-change <pkg> (preserving the prior refresh behaviour, non-interactive throughout).
Wrap the underlying zypper error with %w so future package failures are diagnosable rather than swallowed.

Testing

Validation was manual/integration only — this path runs live zypper against a SLES host and has no unit coverage in-repo (nor did the code it replaces). No automated tests were added or updated.

go build ./pkg/configurer/..., go vet ./pkg/configurer/sles/..., and gofmt are clean.
Manually reproduced the failure and verified the fix (--allow-vendor-change -> exit 0) on a standalone SLES 15 SP7 EC2 instance for both stable-25.0 and stable-29.2.
End-to-end coverage is exercised by the SLES leg of the smoke-test suite (smoke-modern).

Links

JIRA: PRODENG-3623
Upstream root-cause fix: zypper Install fails on vendor-change (from mirantis/launchpad) k0sproject/rig#417, feat(packagemanager): allow vendor change on zypper install k0sproject/rig#419

Checklist

Tests added or updated (N/A — live-host install path, not unit-testable in-repo; validated manually + via smoke tests)
Docs updated if user-visible behaviour changed
No debug output or dead code left in

Written by AI: claude-sonnet-5

SLES cloud images (e.g. SUSE's SLES 15 SP7 AMIs) ship a SUSE-vendor containerd package pre-installed. The Mirantis containerd.io package set must obsolete/replace it, which zypper treats as a vendor change and refuses to perform non-interactively without --allow-vendor-change -- it silently cancels and exits non-zero, surfacing as 'package manager could not install containerd.io'. Replace the generic InstallPackage calls in the SLES InstallMCR path with direct zypper installs that pass --allow-vendor-change, and wrap the underlying zypper error (%w) so future failures are diagnosable. Reproduced on a standalone SLES 15 SP7 instance against both stable-25.0 and stable-29.2; --allow-vendor-change resolves both. Written by AI: claude-sonnet-5

Note in-code that the direct-zypper workaround bypasses rig's InstallPackage only until k0sproject/rig#417/#418 land and are vendored, at which point it should revert to the generic path (or rig's opt-in option). Written by AI: claude-sonnet-5

Copilot

Pull request overview

Implements an interim SLES-specific workaround in Launchpad’s SLES host configurer to ensure MCR installation succeeds when the Mirantis containerd.io package replaces a preinstalled SUSE-vendor containerd (vendor-change scenario).

Changes:

Replaces SLES InstallPackage calls for containerd.io/docker-ee with direct zypper installs using --allow-vendor-change.
Adds an explicit zypper refresh step and improves install error diagnostics by wrapping underlying errors (%w).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Pass -n to zypper refresh so an unattended SSH run cannot block on a repo/GPG trust prompt, matching the -n install calls that follow. Addresses PR review feedback. Written by AI: claude-sonnet-5

The committed .terraform.lock.hcl pinned hashicorp/aws 6.43.0, but the provision-aws module tree now requires aws >= 6.56.0. terraform init (non-upgrade, as CI runs it) cannot reconcile the pin against the constraint and fails with 'Failed to query available provider packages' -- which terratest misreports as a transient network error and retries until the job dies. This blocks every smoke-test job, not just SLES. Regenerate the lock (aws 6.57.1) across linux_amd64/arm64 and darwin_amd64/arm64 so init resolves without --upgrade on CI and dev machines alike (the previous lock only carried a single platform's hashes). Written by AI: claude-sonnet-5

Rebased onto main. Resolved the SLES InstallMCR conflict to retain the --allow-vendor-change fix (PRODENG-3623 / #652) expressed in rig v2's API. Build/test fixes required by the migration: go.mod/go.sum tidied for github.com/k0sproject/rig/v2, validate_facts_test.go updated to rig v2 CompositeConfig/ssh.Config, and %w error wrapping in the EL/SLES/Ubuntu configurers. Adds TestUpgradeModernClusterFromLegacy. Signed-off-by: Kimmo Lehto <kimmo.lehto@gmail.com> Written by AI: claude-sonnet-5

* refactor: migrate to rig v2 Rebased onto main. Resolved the SLES InstallMCR conflict to retain the --allow-vendor-change fix (PRODENG-3623 / #652) expressed in rig v2's API. Build/test fixes required by the migration: go.mod/go.sum tidied for github.com/k0sproject/rig/v2, validate_facts_test.go updated to rig v2 CompositeConfig/ssh.Config, and %w error wrapping in the EL/SLES/Ubuntu configurers. Adds TestUpgradeModernClusterFromLegacy. Signed-off-by: Kimmo Lehto <kimmo.lehto@gmail.com> Written by AI: claude-sonnet-5 * fix: restore GET-based MKE health check and unblock its deadlock Two bugs made every Linux smoke job hang until the harness killed it. MKE installed fine, then 'Validating MKE Health' looped 'waiting for MKE at https://<ip>/_ping to become healthy' every 30s until the test panicked. 1. GET -> HEAD regression (introduced by the rig v2 migration) CheckHTTPStatus was switched to remotefs.HTTPStatusInsecure, which issues a HEAD request on both PosixFS (curl -kIso) and WinFS (Method='HEAD'). The previous per-OS implementations issued a GET, and the MKE health endpoints do not answer HEAD with 200, so the check could never succeed. rig v2's Windows path additionally does not skip TLS verification on PowerShell 5.x, which breaks against MKE's self-signed certs. Reinstate HTTPStatus on the Linux and Windows configurers (GET, TLS skipped), restoring the pre-migration semantics, and have CheckHTTPStatus delegate to the configurer again. Keeps the per-OS logic in the per-OS layer. 2. pingHost deadlock (pre-existing on main, latent until the check fails) On the error path pingHost sent twice on errCh (the error, then nil) while errCh is buffered to len(hosts) and only drained after wg.Wait(). Any failure overflowed the buffer, blocked the second send, and left waitgroup.Done() unreached, so wg.Wait() blocked forever -- converting a bounded ~5 minute failure into an indefinite hang. Now sends exactly one value and defers Done. Renamed the pingHost host parameter for varnamelen after the added comment extended the function scope. Written by AI: claude-sonnet-5 * fix: restore WinRM HTTPS port derivation lost in the rig v2 migration Both Windows smoke jobs failed at Open Remote Connection with 'connect <ip>:5985: All attempts fail'. Windows hosts listen for WinRM over TLS on 5986; 5985 is the plaintext port. creasty/defaults applies rig's winRM port struct-tag default (5985) during Host.UnmarshalYAML, before rig ever sees the config, so the port is never zero by the time rig defaults it. rig v0 corrected this by bumping 5985 -> 5986 when useHTTPS was set; rig v2 only derives the port when it is zero (and only infers useHTTPS when the port is already 5986). A host with 'useHTTPS: true' and no explicit port -- exactly what the terraform modules generate -- was therefore left attempting TLS against the plaintext port. Restore the bump in Host.UnmarshalYAML so existing configs keep working without having to name the port explicitly. Adds TestHostWinRMHTTPSPortDefault, which covers the compatibility matrix (useHTTPS with no port, with 5985, with 5986, a custom port, and no useHTTPS). Verified the test fails without the fix, reproducing the CI symptom exactly (expected 5986, got 5985). Written by AI: claude-sonnet-5 * fix: stop Windows hosts poisoning Linux OS detection Mixed Linux/Windows clusters failed OS detection on every Linux host: Detect host operating systems => failed to resolve configurer for <host>: unsupported OS: linux rig's os.DefaultRegistry holds [ResolveLinux, ResolveLinuxCompat, ResolveWindows, ResolveDarwin] and is a process-global. On a successful match it promotes the winning resolver to the front of the slice so later hosts hit it first. That optimisation is only sound while no resolver matches a superset of another. ResolveLinuxCompat breaks it: it is a fallback for hosts with no /etc/os-release and reports ID "linux" for any Linux host at all. Resolving a Windows host swaps ResolveWindows from index 2 to index 0, which leaves the order [Windows, LinuxCompat, Linux, Darwin] -- the fallback now sits ahead of the real resolver. Every Linux host detected after a Windows host is reported as "linux", matches no configurer and fails the phase. This is why smoke-fips and smoke-windows failed while smoke-modern, smoke-legacy and smoke-upgrade passed: only the mixed clusters ever resolve a Windows host, and the CI logs show the Windows host resolving immediately before the Linux host failed. Give the client its own registry, identical to rig's minus the compat fallback, via rig.WithOSReleaseProvider. Reordering is then harmless because every remaining resolver matches exactly one OS family. Dropping the fallback also restores rig v0's semantics: v0 had no compat resolver, so an unreadable os-release produced a real error instead of a silent misclassification. Every OS launchpad supports ships an os-release file, so the fallback could never yield a usable configurer. The AMI was not at fault -- verified against a live Ubuntu 22.04 FIPS instance, where rig resolves ID=ubuntu Version=22.04 correctly, 20 runs out of 20. Both tests fail without the fix, the first reproducing the exact CI symptom (expected "ubuntu", got "linux"). Written by AI: claude-sonnet-5 * fix: retry WinRM auth rejections while waiting for hosts smoke-fips abandoned a Windows host on the first connect attempt: Open Remote Connection => connect 44.211.52.127:5986: All attempts fail: #1: retry: abort condition reached after 1 attempts: operation cannot be completed: create shell: http response error: 401 - invalid content type The Connect phase exists to wait for hosts to become reachable, and on Windows an auth rejection is part of that wait: the WinRM HTTPS listener answers before provisioning has finished configuring authentication, so a freshly booted host returns 401 for a while with entirely correct credentials. rig v2 wraps 401/403 as protocol.ErrNonRetryable, and this phase's RetryIf honours that, so the wait ended on attempt 1 of 60. rig v0 did not classify these errors, so launchpad retried them and the condition healed itself. The migration swapped the predicate from ErrCantConnect to ErrNonRetryable and inherited the new classification with it. The contrast is visible within a single CI run: smoke-windows retried an i/o timeout to attempt 36 of 60, while smoke-fips aborted a 401 at attempt 1. Both are the same underlying condition - a Windows host whose provisioning has not finished - and the same host connected fine on the previous run, so this is timing, not credentials. Treat 401/403 as retryable again while still honouring ErrNonRetryable for everything else: bad certificates, host key mismatches and misconfigured bastions, none of which waiting can fix. The cost is that genuinely wrong credentials take the retry budget to report rather than failing at once. That is the right trade here: a cluster that would have come up must not fail, and the budget is bounded. Matching is by substring because the WinRM library returns untyped formatted errors and rig exports no sentinel; rig's own isAuthError does the same, and both message shapes it covers are matched. The test drives shouldRetryConnect, the predicate handed to RetryIf. Its three auth cases fail without this change while the four others pass. Written by AI: claude-sonnet-5 * fix: recognize exit code 3010 across rig v1/v2 error phrasing isExitCode3010 matched the exact substring "non-zero exit code: 3010", which is rig v1's wording. rig v2's WinRM transport formats the same error as "command exited with a non-zero exit code: exit code 3010" (note the doubled "exit code"), so the substring never matched and a successful-but-reboot-required MCR install (ERROR_SUCCESS_REBOOT_REQUIRED) was treated as a hard failure instead of triggering the reboot phase. Reproduced on smoke-windows and smoke-fips CI runs for PRODENG-3594: both failed identically with "failed to install container runtime: ... command exited with a non-zero exit code: exit code 3010" during "Install Mirantis Container Runtime on the hosts". Match on the exit code number via regexp instead of a fixed phrase, so future wording changes in either rig transport don't silently break reboot handling again. Added pkg/configurer/windows_test.go covering both known phrasings, an unrelated exit code, and a numeric substring collision (13010) that a naive `strings.Contains(err, "3010")` fix would have wrongly matched. Signed-off-by: James Nesbitt <jnesbitt@mirantis.com> * fix: bound Windows MCR install/uninstall/restart execs with a timeout smoke-windows CI hung for the full 60-minute test timeout: a Windows host rebooted mid-uninstall, reconnected successfully, and the very next remotefs.Upload call then hung indefinitely. A goroutine dump from the test's own timeout panic traced the root cause to rig v2: command.Wait() (protocol/winrm/connection.go) and the remotefs rigrcp helper's command loop (remotefs/winfile.go) had no timeout at all, so a WinRM session that silently died post-reboot could block the caller forever. Filed and fixed upstream as k0sproject/rig#472 / k0sproject/rig#473. The upstream fix only helps if a caller actually supplies a bounded context: rig's plain Exec/ExecOutput hardcode context.Background(), which never has a deadline. Add that bound here as defense in depth, independent of when the upstream fix lands: - pkg/configurer/host.go: extend the Host interface with cmd.ContextRunner so configurers can call ExecContext/ ExecOutputContext. - pkg/product/mke/config/host.go: add matching ExecContext/ ExecOutputContext wrapper methods on Host, mirroring Exec/ ExecOutput's existing sudo/SudoDocker routing (the promoted methods from the embedded *rig.Client would silently skip that routing). - pkg/configurer/windows.go: bound every Exec/ExecOutput call in InstallMCR, UninstallMCR, and RestartMCR -- the MCR lifecycle operations that run immediately around host reboots -- with a new windowsExecTimeout (15 minutes, generous for legitimate slow installs/image pulls, not a tight SLA). remotefs.Upload calls in these functions are unaffected by this change; they are already self-bounded by the upstream winfile.go fix regardless of caller context. Points the go.mod replace directive at the fork commit carrying the upstream fix (k0sproject/rig#473) until it merges and is tagged upstream, at which point the replace should be dropped and the version bumped normally. Added TestExecCtxIsBoundedAndCancelable pinning that the new helper actually carries a deadline and that cancel works. Verified: go build, go vet, full go test --tags 'testing' ./pkg/..., golangci-lint run pkg/configurer/... pkg/product/mke/config/... (one pre-existing, unrelated gofumpt finding in hosts.go, unchanged by this commit). End-to-end verification is the smoke-windows re-run this was found on. Signed-off-by: James Nesbitt <jnesbitt@mirantis.com> * test: temporarily exclude $ from generated Windows password Unblocks this branch's CI. launchpad.tf embeds the generated windows_password directly into the launchpad_yaml output's password/adminPassword fields, which launchpad's config loader then runs through envsubst (pkg/config/config.go). PRODENG-3751 made an unescaped "$word" in that YAML fail config loading loudly if the implied variable is unset, correctly replacing silent password truncation -- but the test's password generator was never updated to account for it, so smoke-windows/smoke-fips fail whenever the random password happens to contain "$" (e.g. run with password "Nq4@CLkhD6%HvwO&$K2f": "variable ${K2f} not set"). This is a workaround, not the fix: the real fix is to escape "$" as "$$" where launchpad.tf embeds the password into that YAML, so a literal "$" -- legitimate in a real password -- keeps working end to end, then restore "$" to this generator's symbol set. Tracked under PRODENG-3751. Written by AI: claude-sonnet-5 Signed-off-by: James Nesbitt <jnesbitt@mirantis.com> * fix: confirm swarm NodeID is reported before JoinWorkers returns smoke-windows failed at the (much later) Label nodes phase: failed to label node 100.53.69.105:5986 (): command result: process finished with error: Process exited with status 1 ("docker node update" requires exactly 1 argument. ...) The empty "()" is swarm.NodeID(h) having returned an empty string with no error. docker swarm join succeeding only means the join command itself completed; it does not guarantee this host's own docker engine has finished updating its local view of swarm state, particularly right after the reconnect JoinWorkers already does for Windows hosts (swarm join tears down and re-establishes the WinRM connection). LabelNodes runs several phases later and had no reason to expect this, so it used the empty NodeID as-is. Add a bounded retry (20 attempts, 3s delay) after joining -- and after the Windows reconnect -- confirming swarm.NodeID(h) actually returns a non-empty NodeID before JoinWorkers considers the host joined. Applies to all hosts, not just Windows, since the race is generic (docker's local swarm-state sync lag), even though it was only actually observed on Windows in this session's testing, likely because the Windows reconnect makes the race window more visible. No dedicated unit test: swarm.NodeID takes the concrete *Host type (not an interface), consistent with the rest of this package, so testing this meaningfully would need a live connection; verified via build, vet, golangci-lint (clean), the full unit suite, and the smoke-windows run this was found on. Written by AI: claude-sonnet-5 Signed-off-by: James Nesbitt <jnesbitt@mirantis.com> * chore: trigger a single clean CI run after AWS quota exhaustion from duplicate label runs Signed-off-by: James Nesbitt <jnesbitt@mirantis.com> --------- Signed-off-by: James Nesbitt <jnesbitt@mirantis.com> Co-authored-by: Kimmo Lehto <kimmo.lehto@gmail.com>

james-nesbitt added 2 commits August 4, 2026 11:26

james-nesbitt requested review from Copilot and quadespresso August 4, 2026 11:05

Copilot started reviewing on behalf of james-nesbitt August 4, 2026 11:06

View session

james-nesbitt requested review from nekwar and pgedara August 4, 2026 11:07

Copilot AI reviewed Aug 4, 2026

View reviewed changes

Comment thread pkg/configurer/sles/sles.go Outdated

Comment thread pkg/configurer/sles/sles.go

fix(sles): make zypper refresh non-interactive

b82b61e

Pass -n to zypper refresh so an unattended SSH run cannot block on a repo/GPG trust prompt, matching the -n install calls that follow. Addresses PR review feedback. Written by AI: claude-sonnet-5

james-nesbitt added the smoke-modern Run modern smoke test label Aug 4, 2026

james-nesbitt merged commit b2325ff into main Aug 4, 2026
11 checks passed

james-nesbitt deleted the PRODENG-3623 branch August 4, 2026 12:43

Navigation Menu

Repository navigation

[PRODENG-3623] Allow vendor change when installing MCR packages on SLES - #652

[PRODENG-3623] Allow vendor change when installing MCR packages on SLES#652
james-nesbitt merged 4 commits into
mainfrom
PRODENG-3623

james-nesbitt commented Aug 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

james-nesbitt commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Why this is a workaround, not the final fix

How

Testing

Links

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

james-nesbitt commented Aug 4, 2026 •

edited

Loading