[vgulati-apphelix]

Description

Deployments that include the common role, such as xqueue, sometimes fail at "Take security updates during ansible runs" on freshly launched Jammy (22.04) instances. This PR makes the security role wait for any background apt/dpkg activity to finish before it runs unattended-upgrade. It also spaces out the existing retries.

Failure

edx.yml enables both of the following settings:

• SECURITY_UPGRADE_ON_ANSIBLE: true

  • Ansible runs unattended-upgrade during the play.

• SECURITY_UNATTENDED_UPGRADES: true

  • The apt-daily / apt-daily-upgrade timers also run upgrades in the background.

When a new instance boots, the background timers can start upgrading packages immediately. The Ansible run may then compete with the background upgrade process for the dpkg lock and fail with a non-zero return code.

The existing Ansible retries do not include a delay, so all retry attempts can be exhausted quickly, often before the background upgrade finishes.

Example from a stage-xqueue run:

MSG:
non-zero return code

FAILED - RETRYING: Take security updates during ansible runs (10 retries left).
...

10.2.93.205 : ok=20 changed=15 unreachable=0 failed=1

This creates a race condition between:

1. The background apt-daily / apt-daily-upgrade process.
2. The unattended-upgrade command explicitly triggered by Ansible.

A possible fix would be to either prevent both mechanisms from running at the same time or introduce an appropriate delay/wait for the package manager lock before retrying.

Changes

In playbooks/roles/security/tasks/security-ubuntu.yml:

New task: wait for the dpkg lock, then repair dpkg. It runs only when SECURITY_UPGRADE_ON_ANSIBLE is true.
Waits until no process holds the dpkg/apt lock files, giving up after 30 minutes.
Runs dpkg --configure -a to finish any interrupted package installs.
If the 30 minutes run out, it fails immediately with a clear error.
It always reports ok rather than changed.
delay: 30 on the existing retries. The 10 retries now cover about 5 minutes instead of about 50 seconds. This handles the background timer firing again between the wait and the upgrade.

Impact

Applies to every play that includes the security role (through common) with SECURITY_UPGRADE_ON_ANSIBLE enabled.
When nothing holds the lock, the new task adds only a second or two.
No new variables or defaults, and no changes needed in internal repos.
Needs fuser (from the psmisc package), which is on standard Ubuntu server and cloud images.

Testing

YAML parses
stage-xqueue pipeline re-run on this branch: "Wait for background apt/dpkg…" shows ok, and "Take security updates" passes without FAILED - RETRYING

---

Make sure that the following steps are done before merging:

• Have a Site Reliability Engineer review the PR if you don't own all of the services impacted.
• If you are adding any new default values that need to be overridden when this change goes live, update internal repos and add an entry to the top of the CHANGELOG.
• Performed the appropriate testing.