Upgrades across minor/major RHEL/Fedora releases can skip bootloader updates due to EVR comparison

[Johan-Liebert1]

When upgrading across RHEL minor releases (e.g., 9.6 -> 9.8), bootupd may skip updating EFI components like shim because the currently installed package has a higher EVR than the one in the target image, even though the target image contains the correct signed binary.

Example

A system running RHEL 9.6 has:
shim-aa64-16.1-8.el9_6.aarch64

The RHEL 9.8 image contains:
shim-aa64-16.1-7.el9.aarch64

bootupd compares these and determines 16.1-8.el9_6 > 16.1-7.el9, so it skips the shim update entirely. However, the 9.8 shim binary has proper signatures and is the one that should be installed for that release.

Should we also check the os-release when comparing EVR?

[cgwalters]

Yes I think in previous discussion I actually wanted to get away from comparing versions and just treat "changed content wins" which is exactly the semantics behind OCI.

I really think this shim situation is a pure RHEL breakage. I mean won't a dnf upgrade from 9.6 to 9.8 also leave the old shim?

[Johan-Liebert1]

Yes, I think changed content wins is a good idea. We can just compare the sha which we store anyway

[Johan-Liebert1]

JIRA: https://redhat.atlassian.net/browse/OCPBUGS-112308

[Johan-Liebert1]

I mean won't a dnf upgrade from 9.6 to 9.8 also leave the old shim?

From preliminary testing, the answer seems to be yes

$ rpmdev-vercmp 'shim-aa64-16.1-8.el9_6.aarch64' 'shim-aa64-16.1-7.el9.aarch64'

WARNING: hyphen in release1: aa64-16.1-8.el9_6.aarch64
WARNING: hyphen in release2: aa64-16.1-7.el9.aarch64

rpmdev-vercmp <epoch1> <ver1> <release1> <epoch2> <ver2> <release2>
rpmdev-vercmp <EVR1> <EVR2>
rpmdev-vercmp # with no arguments, prompt

Exit status is 0 if the EVR's are equal, 11 if EVR1 is newer, and 12 if EVR2
is newer.  Other exit statuses indicate problems.

shim-aa64-16.1-8.el9_6.aarch64 > shim-aa64-16.1-7.el9.aarch64

[Rolv-Apneseth]

Does this relate back to #468? I was wondering about just using hashes in #1130 (comment)

[krolmiki2011]

Does this relate back to #468? I was wondering about just using hashes in #1130 (comment)

Im using a script now

[cgwalters]

The command is supposed to be invoked with just the EVR, not the full package name:

> rpmdev-vercmp '16.1-8.el9_6.aarch64' '16.1-7.el9.aarch64'
16.1-8.el9_6.aarch64 > 16.1-7.el9.aarch64

[Johan-Liebert1]

Does this relate back to #468?

Not really, as this could happen with other package managers as well

I think @cgwalters suggestion of "changed content wins" is the way to go here

[Rolv-Apneseth]

To clarify, we would still be storing versions, but move away from using them to decide if an update is required, correct? Meaning we could probably drop uapi-version.

[Johan-Liebert1]

Yeah probably. Storing the versions are still useful

[cgwalters]

Hold up though did anyone file an issue against RHEL shim?

OK looks like this is not the first time https://redhat.atlassian.net/browse/RHEL-78691

[Johan-Liebert1]

We also have https://redhat.atlassian.net/browse/RHEL-248162

[Johan-Liebert1]

Hold up though

I think this requires a bit more nuance. There could be times where we would not want to update the bootloader/shim etc even if they change

[cgwalters]

I think this requires a bit more nuance. There could be times where we would not want to update the bootloader/shim etc even if they change

Isn't that use case systemctl mask bootloader-upgrade.service?