Auto Assessment job terminated with SIGTERM after ~90s due to systemd start timeout and the service type model

[tiagojfernandes]

Describe the bug
MsftLinuxPatchAutoAssess.service is created with Type=forking, but the script runs Python in the foreground and never daemonizes. systemd waits for the fork-and-exit signal, hits the default TimeoutStartSec=90s, and SIGTERMs the cgroup. On any host where Python startup + the first package-manager call exceeds 90s, every auto-assessment is killed before it writes a result and the Portal shows "Pending" forever.
The same unit template is written for all distros - issue is latent everywhere, only reliably reproducible on slow hosts.

To Reproduce

1. RHEL 8.4 VM (Arc or Azure) as example
2. Make yum slow enough to exceed 90s. Wrap /usr/bin/yum:

 # wrap yum so every call sleeps 120 min first
sudo mv /usr/bin/yum /usr/bin/yum.real
sudo tee /usr/bin/yum >/dev/null <<'EOF'
#!/bin/bash
sleep 120
exec /usr/bin/yum.real "$@"
EOF
sudo chmod +x /usr/bin/yum

3. Install the extension, enable auto-assessment (assessmentMode=AutomaticByPlatform)
4. sudo systemctl start MsftLinuxPatchAutoAssess.service
5. journalctl -u MsftLinuxPatchAutoAssess.service

Expected behavior
Assessment runs to completion. .status flips from Transitioning to success.

Actual behavior
SIGTERM at exactly T+90s, MainPID=0, Result: failed (timeout). Status stays Transitioning indefinitely:

Jun 04 15:17:01 lab.opsmgr.net sudo[120526]: root : TTY=pts/1 ; PWD=/opt/lpe-repro/runtime/config ; USER=root ; COMMAND=/bin/systemctl start MsftLi nuxPatchAutoAssess.service
Jun 04 15:17:01 lab.opsmgr.net sudo[120526]: pam_unix(sudo:session): session opened for user root by root(uid=0)
Jun 04 15:17:01 lab.opsmgr.net systemd[1]: Starting Microsoft Azure Linux Patch Extension - Auto Assessment...
Jun 04 15:17:16 lab.opsmgr.net sudo[120562]: root : PWD=/opt/lpe-repro ; USER=root ; COMMAND=/bin/id
Jun 04 15:17:16 lab.opsmgr.net systemd[1]: Started Session c128 of user root.
Jun 04 15:17:16 lab.opsmgr.net sudo[120562]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 04 15:17:16 lab.opsmgr.net sudo[120562]: pam_unix(sudo:session): session closed for user root
Jun 04 15:17:16 lab.opsmgr.net systemd[1]: session-c128.scope: Succeeded.
Jun 04 15:17:16 lab.opsmgr.net sudo[120566]: root : PWD=/opt/lpe-repro ; USER=root ; COMMAND=/bin/yum -q check-update
Jun 04 15:17:16 lab.opsmgr.net systemd[1]: Started Session c129 of user root.
Jun 04 15:17:16 lab.opsmgr.net sudo[120566]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 04 15:18:01 lab.opsmgr.net systemd[1]: Started Session 3800 of user root.
Jun 04 15:18:01 lab.opsmgr.net systemd[1]: session-3800.scope: Succeeded.
Jun 04 15:18:31 lab.opsmgr.net systemd[1]: MsftLinuxPatchAutoAssess.service: start operation timed out. Terminating.
Jun 04 15:18:31 lab.opsmgr.net systemd[1]: MsftLinuxPatchAutoAssess.service: Failed with result 'timeout'.
Jun 04 15:18:31 lab.opsmgr.net sudo[120526]: pam_unix(sudo:session): session closed for user root
Jun 04 15:18:31 lab.opsmgr.net systemd[1]: Failed to start Microsoft Azure Linux Patch Extension - Auto Assessment.

Screenshots
N/A.

Desktop (please complete the following information):
• OS: RHEL 8.4 (reproduced); latent on all supported distros, same template is written unconditionally
• Cloud type: Arc-enabled (same code path as Azure)
• systemd: 239+ (default TimeoutStartSec=90s)
• Affected file: ServiceManager.py → create_service_unit_file()

Smartphone (please complete the following information):
N/A.

Additional context
Root cause
Type=forking requires the parent to fork a daemon child and exit before TimeoutStartSec. The script does neither. systemd's contract is violated → cgroup killed.

Proposed fix
In ServiceManager.create_service_unit_file():
• Type=forking → Type=simple (matches what the script actually does)
• Add TimeoutStartSec=0 (no meaningful start-gate for simple)
• Add RuntimeMaxSec=1800 (30-min safety cap against runaway hangs)

Validated on repro box after fix (~10 min execution to complete with success)

Jun 04 23:44:56 lab.opsmgr.net systemd[1]: Started Microsoft Azure Linux Patch Extension - Auto Assessment.
Jun 04 23:45:11 lab.opsmgr.net sudo[3486]:     root : PWD=/opt/lpe-repro ; USER=root ; COMMAND=/bin/id
Jun 04 23:45:11 lab.opsmgr.net sudo[3486]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 04 23:45:11 lab.opsmgr.net sudo[3486]: pam_unix(sudo:session): session closed for user root
Jun 04 23:45:12 lab.opsmgr.net sudo[3490]:     root : PWD=/opt/lpe-repro ; USER=root ; COMMAND=/bin/yum -q check-update
Jun 04 23:45:12 lab.opsmgr.net sudo[3490]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 04 23:47:13 lab.opsmgr.net sudo[3564]:     root : PWD=/opt/lpe-repro ; USER=root ; COMMAND=/bin/yum -q --security check-update
Jun 04 23:47:14 lab.opsmgr.net sudo[3564]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 04 23:49:15 lab.opsmgr.net sudo[3608]:     root : PWD=/opt/lpe-repro ; USER=root ; COMMAND=/bin/yum -y install yum-utils
Jun 04 23:49:15 lab.opsmgr.net sudo[3608]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 04 23:51:17 lab.opsmgr.net sudo[3669]:     root : PWD=/opt/lpe-repro ; USER=root ; ENV=LANG=en_US.UTF8 ; COMMAND=/bin/needs-restarting -r
Jun 04 23:51:17 lab.opsmgr.net sudo[3669]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 04 23:51:18 lab.opsmgr.net sudo[3669]: pam_unix(sudo:session): session closed for user root
Jun 04 23:51:18 lab.opsmgr.net sudo[3674]:     root : PWD=/opt/lpe-repro ; USER=root ; COMMAND=/bin/yum -y install yum-plugin-ps
Jun 04 23:51:18 lab.opsmgr.net sudo[3674]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 04 23:53:20 lab.opsmgr.net sudo[3718]:     root : PWD=/opt/lpe-repro ; USER=root ; COMMAND=/bin/yum ps
Jun 04 23:53:20 lab.opsmgr.net sudo[3718]: pam_unix(sudo:session): session opened for user root by (uid=0)
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Bootstrap environment: Prod
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Building bootstrap container configuration...
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Trying to connect IMDS end point. url:http://169.254.169.254/metadata/instance/compute?api-version=2019-06-01.
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Connecting to IMDS endpoint...
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Exception from IMDS connection http request: URLError(ConnectionRefusedError(111, 'Connection refused'),)
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Failed to connect to IMDS end point. [Trial=1].
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Connecting to IMDS endpoint...
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Exception from IMDS connection http request: URLError(ConnectionRefusedError(111, 'Connection refused'),)
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Failed to connect to IMDS end point. [Trial=2].
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Connecting to IMDS endpoint...
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Exception from IMDS connection http request: URLError(ConnectionRefusedError(111, 'Connection refused'),)
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Failed to connect to IMDS end point. [Trial=3].
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Connecting to IMDS endpoint...
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Exception from IMDS connection http request: URLError(ConnectionRefusedError(111, 'Connection refused'),)
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Failed to connect to IMDS end point. [Trial=4].
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Connecting to IMDS endpoint...
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Exception from IMDS connection http request: URLError(ConnectionRefusedError(111, 'Connection refused'),)
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: - Failed to connect to IMDS end point. [Trial=5].
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Failed to connect IMDS end point after 5 retries. This is expected in Arc VMs. VMCloudType is set to Arc.
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: --------------------------------------------------------------------------------------------------------------------------------
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: ,---.                        ,---.     |         |        ,-.-.                                        |
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: |---|,---,.   .,---.,---.    |---',---.|--- ,---.|---.    | | |,---.,---.,---.,---.,---.,-.-.,---.,---.|---
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: |   | .-' |   ||    |---'    |    ,---||    |    |   |    | | |,---||   |,---||   ||---'| | ||---'|   ||
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: `   ''---'`---'`    `---'    `    `---^`---'`---'`   '    ` ' '`---^`   '`---^`---|`---'` ' '`---'`   '`---'
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]:                                                                               `---'
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: --------------------------------------------------------------------------------------------------------------------------------
Jun 04 23:55:20 lab.opsmgr.net MsftLinuxPatchAutoAssess.sh[3453]: Completed building bootstrap container configuration.
Jun 04 23:55:20 lab.opsmgr.net systemd[1]: MsftLinuxPatchAutoAssess.service: Succeeded.

Status flipped Transitioning to success; AssessmentState.json.lastStartInSecondsSinceEpoch updated correctly.

[mathias-kallberg-slu]

Seeing what looks like the same root cause, but with a different failure signature — no explicit TimeoutStartSec=90s / Failed with result 'timeout'. Posting in case the variant is useful.

Environment

• Ubuntu 22.04.5 (jammy), Arc-enabled server (on-prem, via Azure Arc)
• Microsoft.CPlat.Core.LinuxPatchExtension-1.6.62
• assessmentMode: AutomaticByPlatform, maximumAssessmentInterval: PT12H

What we see

Instead of a hard SIGTERM at T+90s, the service unit transitions to Started / Deactivated successfully within ~50s of the timer firing — well under the default TimeoutStartSec. But the actual MsftLinuxPatchCore.py process (spawned by the forking wrapper) keeps running well past that point, since it's doing a real apt-get -s dist-upgrade simulation against a large backlog (70+ packages, including kernel/docker-stack updates on this host).

Jul 14 06:02:44 <host> systemd[1]: MsftLinuxPatchAutoAssess.service: Deactivated successfully.
Jul 14 06:02:44 <host> systemd[1]: Started Microsoft Azure Linux Patch Extension - Auto Assessment.
Jul 14 06:02:44 <host> systemd[1]: MsftLinuxPatchAutoAssess.service: Consumed 27.538s CPU time.

0.aa.core.log confirms the real assessment work is still in progress well after that "Started" line — it doesn't finish apt-get -s dist-upgrade and move on to security-patch classification until minutes later:

2026-07-14T06:03:09Z> Starting patch assessment... [ActivityId: 13e1580d-...]
2026-07-14T06:03:09Z> Getting available patches...
...
2026-07-14T06:03:13Z> [APM] Refreshing local repo... [SourcePartsDir=...]

At the next hourly timer firing, the new invocation reports the previous PID as already gone, without ever seeing a completion:

2026-07-14T07:02:58Z> Processes still running from the previous request: [PIDsFound=None][PreviousPIDs=[3384734]]

The on-disk status for that assessment (status/1.status) is permanently stuck in Transitioning, with criticalAndSecurityPatchCount: 0 and 65 patches all classified as Other — consistent with PatchAssessor.start_assessment() having completed get_all_updates() / set_package_assessment_status(), but never reaching get_security_updates() or the final set_assessment_substatus_json(status=Success) a few lines later. No exception is logged; the process just disappears between one hourly cycle and the next.

systemctl cat on the generated unit confirms the same root config as #350:

[Service]
Type=forking
ExecStart=/bin/bash /var/lib/waagent/Microsoft.CPlat.Core.LinuxPatchExtension-1.6.62/MsftLinuxPatchAutoAssess.sh

No PIDFile=, and the wrapper script never daemonizes — it just execs the core process in the foreground.

Working theory for the variant

With Type=forking and no PIDFile, systemd's notion of "the service has started" is based on the initial ExecStart process tree exiting/forking, not on the actual work completing. On this host, something in the bash→python startup chain apparently satisfies systemd's fork-detection heuristic quickly (~50s) even though the real worker process is nowhere near done. systemd then considers the unit's lifecycle complete, and the still-running Python assessment continues as an effectively untracked process in a cgroup systemd believes is empty/reusable. It appears to get reaped (no dedicated log line for that death) whenever the next hourly timer invocation reuses the same unit/cgroup — journalctl for the next firing shows the previous PID already absent, with no explicit kill message tied to the old process.

Net effect: on hosts with a large enough patch backlog that the assessment genuinely takes more than roughly an hour of wall-clock (not CPU) time, auto-assessment never completes and Update Manager shows stale/no periodic assessment data indefinitely — while a manual "Check for updates" (different operation path, not -autoAssessOnly) always succeeds, since it isn't going through this service unit at all.

Given roughly half our ~100-server fleet is affected, and the affected machines are consistently the ones with the largest pending-update backlog, this seems very reproducible in the wild without needing to artificially slow down the package manager — a large dist-upgrade -s simulation plus classification is apparently enough on its own.

Happy to provide the full journalctl/aa.core.log/status sequence (hostnames redacted) if useful for triage.