Skip to content

fix(linux): stop chmod'ing resolv.conf to 0444 in direct-write path (EACCES crash loop on container restart) - #36

Merged
ssrlive merged 2 commits into
tun2proxy:masterfrom
xz-dev:fix/resolv-conf-0444-restart-eaccess
Sep 21, 2026
Merged

ssrlive merged 2 commits into
tun2proxy:masterfrom
xz-dev:fix/resolv-conf-0444-restart-eaccess

Conversation

@xz-dev

@xz-dev xz-dev commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes a restart crash loop in containers: the direct-write fallback in setup_resolv_conf chmod'ed /etc/resolv.conf to 0444 via write_nameserver, which poisons Docker's bind-mounted host-side resolv.conf file. Every subsequent container restart fails with EACCES unless the process holds CAP_DAC_OVERRIDE.

  • Remove the fchmod(fd, 0444) from write_nameserver.
  • On the direct-write path, if open() fails with EACCES, restore mode 0644 and reopen — heals deployments already poisoned by older versions.

Why the 0444 was wrong

The comment says it protects resolv.conf from NetworkManager, but NM runs as root with CAP_DAC_OVERRIDE and can overwrite a 0444 file anyway. The real protection on the mount path is the MS_REMOUNT|MS_RDONLY remount, not the file mode. On the direct-write path the chmod protects nothing — it only guarantees the next startup fails.

Failure mechanism (Docker)

  1. Fresh docker run: resolv.conf is 0644 root:root (Docker-owned host file, bind-mounted) → tun2proxy opens, writes, chmods to 0444 → healthy.
  2. docker restart (also daemon restart / boot restore): same file, still 0444. open(O_WRONLY) → owner has no write bit → needs CAP_DAC_OVERRIDE → cap_drop: ALL containers fail with EACCES → with --exit-on-fatal-error, permanent crash loop. Only docker rm + re-create fixes it.

Observed in production (Docker Engine 29.1.3, ghcr.io/tun2proxy/tun2proxy:v0.8.3): host-side resolv.conf of the tun2proxy container was 444, all other containers on the host were 644.

Log signature:

WARN  tproxy_config::linux] failed to bind mount custom resolv.conf onto /etc/resolv.conf, resorting to direct write
ERROR tproxy_config::linux] tproxy removal failed: Permission denied (os error 13)
ERROR tproxy_config::common] Failed to set up TProxy: EACCES: Permission denied

Testing

Integration test added (test-restart-eacces.sh), run against Docker Engine 29.1.3, container flags --cap-drop ALL --cap-add NET_ADMIN --device /dev/net/tun:

scenario stock v0.8.3 this PR
fresh create healthy yes yes
host resolv.conf mode after first run 0444 (poisoned) 0644 (kept)
docker restart healthy no (EACCES crash loop) yes
restart with pre-poisoned 0444 file (upgrades) n/a heals to 0644, healthy

cargo check / cargo test pass (crate has no other tests).

Notes

  • Downstream tun2proxy consumes this crate via crates.io (tproxy-config = "^7.0.7"); it will need a version bump + release for users on official images.
  • I plan to file a matching issue in tun2proxy/tun2proxy linking here.

@xz-dev

xz-dev commented Sep 10, 2026

Copy link
Copy Markdown
Contributor Author

Why fresh creation works but every restart fails (mechanism walkthrough)

Adding a detailed explanation of the failure mode here, since it's the non-obvious part of this bug and the reason it ships silently.

Linux DAC check. The process runs as root (uid 0), and the file is root:root. When uid matches the owner, the kernel checks the owner permission bits. After the first run the file is r--r--r-- (0444), so the owner has no write bit. Open would fail — unless the second check passes: CAP_DAC_OVERRIDE, which bypasses file permission bits entirely. A container with cap_drop: ALL and only NET_ADMIN added does not hold it.

docker run (create path). Docker generates a fresh host-side resolv.conf, mode 0644 root:root, and bind-mounts it into the container as /etc/resolv.conf. tun2proxy takes the direct-write fallback (no CAP_SYS_ADMIN for the bind-mount path), opens the file — the owner write bit is there, the open succeeds — writes its nameserver line, and chmods the file to 0444. Everything is healthy. The poison is now planted, with zero symptoms.

docker restart (restart path). Same container, same host-side file is reused — still 0444. This time open(O_WRONLY | O_TRUNC) fails with EACCES: the owner has no write bit, and there is no CAP_DAC_OVERRIDE to fall back on. Setup fails; with --exit-on-fatal-error the process exits; the restart policy brings it back up; it fails again — permanent crash loop. The only unlock is docker rm + re-create, which makes Docker write a fresh 0644 file.

The trap window. A deployment that only ever creates (first install, image upgrade) never sees the bug. It fires exactly on the restart paths — docker restart, dockerd restart, host reboot restore — i.e. the paths production setups rely on. In our case the stack ran fine for days after first deploy, then the first host reboot took it down.

Bonus: the cleanup path fails too. When setup fails, _tproxy_setup runs _tproxy_remove to roll back. The rollback writes the saved original resolv.conf content back with fs::write — to the same 0444 file, with the same missing capability. That's why the log signature is three lines that look like three separate problems:

WARN  failed to bind mount custom resolv.conf ... resorting to direct write   ← normal in containers, not an error
ERROR tproxy removal failed: Permission denied                                 ← cleanup hits the same 0444
ERROR Failed to set up TProxy: EACCES: Permission denied                       ← the actual failure

Three log lines, one root cause.

Why the fix has two parts:

  • Dropping the fchmod(0444) stops new deployments from being poisoned. The chmod is also ineffective for its stated purpose even on hosts: NetworkManager runs as root with CAP_DAC_OVERRIDE and can overwrite a 0444 file anyway; the real protection on the mount path is the MS_REMOUNT|MS_RDONLY remount, which not even root can write through without another remount.
  • The EACCES-recovery reopen lets deployments already poisoned by an older version heal themselves on their first restart after upgrading, instead of crash-looping until someone recreates the container. Verified in the integration test: a pre-poisoned 0444 file comes back as 0644 and the container starts healthy.

@xz-dev
xz-dev marked this pull request as ready for review September 10, 2026 02:27
The direct-write fallback in setup_resolv_conf (used when bind-mounting
is unavailable, e.g. inside Docker without CAP_SYS_ADMIN) called
fchmod(fd, 0444) on /etc/resolv.conf via write_nameserver. In
containers, /etc/resolv.conf is Docker's bind-mounted host file
(/var/lib/docker/containers/<id>/resolv.conf), so the mode change
persists across restarts. On the next start, open(O_WRONLY) on the
now-0444 file fails with EACCES unless the process holds
CAP_DAC_OVERRIDE - which capability-dropped containers don't. With
--exit-on-fatal-error this becomes a permanent crash loop that only
recreating the container can fix.

The 0444 does not even serve its stated purpose on this path: it is
meant to keep NetworkManager from overwriting resolv.conf, but NM runs
as root with CAP_DAC_OVERRIDE and can overwrite a 0444 file anyway.
The real protection is the MS_RDONLY bind-mount remount on the
mount path. On the direct-write path the chmod protects nothing and
only guarantees the next start fails.

Two changes:
- write_nameserver: drop the fchmod(0444). The tempfile on the mount
  path doesn't need it either (it is bind-mounted read-only).
- direct-write branch: on EACCES, restore mode 0644 and reopen, so
  deployments already poisoned by an older version heal themselves
  on the first restart instead of crash-looping.

Verified with an integration test (test-restart-eacces.sh) against
Docker Engine 29.1.3 with --cap-drop ALL --cap-add NET_ADMIN:
- stock v0.8.3: host resolv.conf becomes 0444 after first run
- patched: mode stays 0644, restart healthy, and a pre-poisoned
  0444 file is healed back to 0644 with no EACCES
@xz-dev
xz-dev force-pushed the fix/resolv-conf-0444-restart-eaccess branch from 4b60b9b to 49d9c0e Compare September 10, 2026 02:33
@ssrlive
ssrlive requested a lite review from Copilot September 21, 2026 04:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The integration test has portability, recovery-coverage, log-capture, and startup-timeout issues.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 Medium severity

Open (2)
What changed in this PR

Fixes Linux resolv.conf permission poisoning and container restart failures.

Changes:

  • Removes the direct-write 0444 chmod.
  • Restores 0644 and retries for poisoned files.
  • Adds Docker restart integration coverage.
File Summary
src/​linux.rs Updates direct-write permissions and recovery handling.
test-restart-eacces.sh Adds restart and permission validation tests.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread test-restart-eacces.sh Outdated
Comment thread test-restart-eacces.sh Outdated
- Resolve the host resolv.conf path via `docker info --format
  '{{.DockerRootDir}}'` instead of hardcoding /var/lib/docker, so the
  test works with custom data-root and rootless Docker.
- Pre-poison the host file to 0444 before restart, so the EACCES
  heal-to-0644 branch in setup_resolv_conf is actually exercised
  (upgrade scenario), and assert the mode is restored.
- Drop unused CIDFILE/mktemp leftover.

Addresses Copilot review feedback on PR tun2proxy#36.
@ssrlive
ssrlive merged commit bf397f3 into tun2proxy:master Sep 21, 2026
6 checks passed
@ssrlive

ssrlive commented Sep 21, 2026

Copy link
Copy Markdown
Member

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants