policy-routes@.service loops indefinitely when DHCP lease is lost: no timeout, no active recovery

[rgoltz]

Problem

If a DHCP lease is lost during boot (e.g. another service calls networkctl reload) and the VPC DHCP server does not respond to the renewal, policy-routes@ens5 enters an endless retry loop. Each attempt blocks ~120s on systemd-networkd-wait-online (no explicit timeout), fails, restarts, and blocks again. The instance sits without network for as long as the loop runs, and since nothing actively triggers DHCP recovery, it never comes back.

We trigger the reload from a legitimate oneshot service that writes a DNS drop-in and calls networkctl reload. The reload causes systemd-networkd to drop and re-acquire the DHCP lease. On most boots this takes <1s. On some boots, the DHCP response never comes (tracked separately as a DHCP/VPC issue). When that happens, policy-routes has no way to recover.

Journal from a failing instance

Captured from root volume snapshot of a failed c6i.large (Dedicated Tenancy, AL2023, amazon-ec2-net-utils 2.7.6):

Aug 17 19:24:42 localhost systemd-networkd[1547]: ens5: DHCPv4 address 10.106.157.99/24, gateway 10.106.157.1 acquired from 10.106.157.1
Aug 17 19:24:42 localhost systemd-networkd[1547]: ens5: Link UP
Aug 17 19:24:42 localhost systemd-networkd[1547]: ens5: Gained carrier
Aug 17 19:24:43 localhost systemd-networkd[1547]: ens5: Reconfiguring with /usr/lib/systemd/network/80-ec2.network.
Aug 17 19:24:43 localhost systemd-networkd[1547]: ens5: DHCP lease lost
Aug 17 19:24:43 localhost systemd-networkd[1547]: ens5: DHCPv6 lease lost
Aug 17 19:24:43 localhost systemd-networkd[1547]: ens5: Gained IPv6LL
         (no further DHCPv4 messages - DHCP never recovers)
Aug 17 19:26:42 localhost systemd-networkd-wait-online[1527]: Timeout occurred while waiting for network connectivity.
Aug 17 19:26:42 localhost systemd[1]: policy-routes@ens5.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 19:26:42 localhost systemd[1]: Failed to start policy-routes@ens5.service - Set up policy routes for ens5.
Aug 17 19:26:44 localhost systemd[1]: policy-routes@ens5.service: Scheduled restart job, restart counter is at 1.
Aug 17 19:26:44 localhost systemd[1]: Starting policy-routes@ens5.service - Set up policy routes for ens5...
Aug 17 19:28:53 localhost systemd-networkd-wait-online[3888]: Timeout occurred while waiting for network connectivity.
Aug 17 19:28:53 localhost systemd[1]: policy-routes@ens5.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 19:28:53 localhost systemd[1]: Failed to start policy-routes@ens5.service - Set up policy routes for ens5.
Aug 17 19:28:54 localhost systemd[1]: policy-routes@ens5.service: Scheduled restart job, restart counter is at 2.
Aug 17 19:31:00 localhost systemd-networkd-wait-online[5461]: Timeout occurred while waiting for network connectivity.
Aug 17 19:31:00 localhost systemd[1]: policy-routes@ens5.service: Main process exited, code=exited, status=1/FAILURE
Aug 17 19:33:07 localhost systemd-networkd-wait-online[6993]: Timeout occurred while waiting for network connectivity.
Aug 17 19:33:07 localhost systemd[1]: policy-routes@ens5.service: Main process exited, code=exited, status=1/FAILURE

Each cycle: ~120s of useless waiting. No attempt to actively restore DHCP. The interface stays on link-local IPv6 only. Cloud-init finishes with DataSourceNone, instance is dead.

Successful boot for comparison (same instance, previous day)

Aug 16 02:10:48 localhost systemd-networkd[1594]: ens5: DHCPv4 address 10.106.156.220/24, gateway 10.106.156.1 acquired from 10.106.156.1
Aug 16 02:10:48 localhost systemd-networkd[1594]: ens5: Reconfiguring with /usr/lib/systemd/network/80-ec2.network.
Aug 16 02:10:48 localhost systemd-networkd[1594]: ens5: DHCP lease lost
Aug 16 02:10:48 localhost systemd-networkd[1594]: ens5: DHCPv6 lease lost
Aug 16 02:10:49 localhost systemd-networkd[1594]: ens5: DHCPv4 address 10.106.156.220/24, gateway 10.106.156.1 acquired from 10.106.156.1
Aug 16 02:10:49 localhost systemd[1]: Finished policy-routes@ens5.service - Set up policy routes for ens5.

Same AMI, same config, same reload. Only difference: DHCP came back.

What could be better

1. systemd-networkd-wait-online has no explicit --timeout. It uses the built-in default (~120s). A shorter explicit timeout (e.g. 60s) would reduce the useless wait and give more retries in the same time window.

2. No mechanism to actively kick DHCP. When wait-online fails, the service just retries the same passive wait. A networkctl renew or networkctl reconfigure before retrying could actually fix the situation.

3. After=systemd-networkd.service is missing. On some boots, policy-routes can start before networkd has loaded .network files (we measured only 750ms margin). Adding After= is cheap defense-in-depth.

Proposed fix

Unit file:

 [Unit]
 Description=Set up policy routes for %I
-StartLimitIntervalSec=10
+After=systemd-networkd.service
+Wants=systemd-networkd.service
+StartLimitIntervalSec=300
 StartLimitBurst=5

Script (bin/setup-policy-routes.sh):

-    /lib/systemd/systemd-networkd-wait-online -i "$iface"
+    /lib/systemd/systemd-networkd-wait-online -i "$iface" --timeout=60

StartLimitIntervalSec=300 (was 10): prevents the edge case where wait-online fails fast (transient interface state) and 5 rapid failures permanently kill the service + timer (BindsTo).

--timeout=60: bounded wait, faster retry cycle, more chances for DHCP to recover within the same overall window.

Not addressed here

The root cause (DHCP not recovering after networkctl reload on EC2/VPC) is a separate problem tracked in amazonlinux/amazon-linux-2023#1127.

Reproduction

Intermittent, timing-dependent. ~50% failure rate on our Dedicated Tenancy ECS cluster with AL2023 ECS-optimized AMI. Logs from rescue-mounted root volume snapshot.

Contribution

I'm preparing a PR with the proposed fix. Will link here once submitted.

Related

• [Bug] - systemd-networkd fails to re-acquire DHCP lease after networkctl reload amazon-linux-2023#1127 (DHCP recovery failure after networkctl reload)