Skip to content

Sandbox: bubblewrap starts where a protected path is missing - #319

Merged
shoom1 merged 11 commits into
developfrom
fix/bwrap-missing-protected-files
Nov 17, 2026
Merged

shoom1 merged 11 commits into
developfrom
fix/bwrap-missing-protected-files

Conversation

@shoom1

@shoom1 shoom1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Owner

On Linux with bubblewrap installed, the OS sandbox never started: every execute_python call was refused ("it did not start (bubblewrap, exit status 1): bwrap: Can't create file at ~/.bashrc: Read-only file system"), and with os_sandbox_strict off the code ran unsandboxed. CI's Linux runner now ships bwrap (ubuntu-26.04), so TestLiveSandbox ran there for the first time; on ubuntu-24.04 it was skipped. Running it also showed that a timed-out execute_python kept running, on Linux and on macOS.

Cause: for a protected path that does not exist (~/.bashrc, ~/.zshrc, and on any Linux root /System and /Library), the wrap passed --ro-bind /dev/null <path>. bwrap must create that mount point below the root it has already mounted read-only, and dies with EROFS. Where the parent was writable (a project's .git when it is not a repository, .{app_name}), bwrap would have left an empty 0444 file behind, which breaks git and the app's own folder.

The sandbox (tools/os_sandbox/bubblewrap.py, base.py):

  • A missing protected path the code cannot create (under the read-only root, or under a protected path bound read-only) gets no mount.
  • One the code could create (its first missing component lies below a writable bind) is held by a placeholder: a folder holding a hold file per run, bound read-only.
    • A run removes only a placeholder it made, or one it joined that held only placeholder holds. A real folder, even an empty one, is never removed.
    • Holds and staging go through folders opened without following symlinks.
    • A placeholder that cannot be made, or a hold that cannot be written (other than EACCES/EPERM/EROFS), fails the wrap.
  • The code runs as pid 1 of its sandbox (--as-pid-1), so a run that ends leaves no process behind. OSSandboxResult.release() (new, a no-op by default) drops the run's holds only after a run that ended on its own (status 0-127). After a timeout or a signal, the hold stays and the placeholder stays protected.

The executor (tools/executor.py), on every platform:

  • Each run starts in a session of its own.
  • On a timeout, the executor SIGSTOPs the run's group, SIGKILLs bubblewrap's pid 1 by its host pid (found through /proc), then SIGKILLs the group, always in a finally. Ubuntu's AppArmor profile for bwrap clears pid 1's parent-death signal at exec, so killing bwrap alone left the code running.
  • A run that ends on its own also kills what it left in its group.
  • On Windows (no process groups), a timeout kills the process, as before.
  • BaseCLIApp.run() installs SIGHUP and SIGTERM handlers, unless the signal is ignored. Each kills the live runs, then chains to the previous handler, so the app still ends as before. A normal exit waits for a run, as before. Outside a pid namespace, code that calls setsid() can outlive a run.

Tests: the new offline tests on the argv, the placeholders, the holds, the executor's kill paths and the signal handlers were each seen failing first. The live tests (12 on ubuntu-26.04, all passing) cover a missing shell start-up file, a project that is no repository, the code as pid 1, forks ending with the run, and a timeout stopping the code and what it forked.

execute_python never ran inside bubblewrap: every call failed with "bwrap:
Can't create file at ~/.bash_profile: Read-only file system" (or another
protected path the host lacks; /System and /Library, macOS paths in the same
list, are on no Linux host). The backend bound /dev/null over each missing
deny-write path whose parent existed, and bwrap makes a missing mount point
itself, which below --ro-bind / / fails with EROFS before the code runs. The
logic is the same in bwrap 0.9.0 (Ubuntu 24.04) and 0.11.1 (26.04); CI saw it
only now because GitHub's ubuntu-26.04 image is the first with bwrap on PATH,
so TestLiveSandbox had always skipped.

A missing protected path the read-only root covers now gets no mount: the
code cannot create it there. One the code could create (in a folder it may
write: a project's .git before git init, its .{app_name} before the app makes
it) would have been made by bwrap on the host as an empty file left behind,
and an empty .git file breaks git. The wrap now holds a placeholder there
instead: an empty folder, made with a hold file inside and renamed into
place, bound read-only like any existing protected path. Each run holds it
with its own file, in this process or another, and OSSandboxResult.release
drops the hold and removes the folder when no hold is left (rmdir: whatever
was put in it meanwhile stays). The executor releases after every call. A
placeholder that cannot be made fails the wrap, since the code could still
create the path; a missing folder above a protected path takes the
placeholder, so neither can be made.

The live test of the project's .env now checks that the key never reaches
the code: bubblewrap shows /dev/null in its place, which reads empty, where
seatbelt refuses the open.
The code now runs as the sandbox's pid 1 (--as-pid-1). bwrap's reaper used
to report the command's exit as soon as it came, while a process the code
forked could still run: the executor then released the placeholders, an
rmdir that detaches the sandbox's mount of .git, and the survivor could put
a repository there. As pid 1 the code's exit completes only once the kernel
has killed and reaped every process in the namespace, and bwrap waits for it.
A run killed at its timeout may still leave them alive a moment, so the
executor releases only after a run that ended on its own (or code the
checks refused); a timed-out run keeps its holds, and the placeholder stays,
empty and protected, for later runs to hold.

The hold protocol:
- An empty folder at a creation point below a writable bind is joined like
  a placeholder: between a release's unlink of the last hold and its rmdir
  the folder is empty, and a run wrapping then bound it as a real folder that
  the rmdir removed from under it. O_EXCL and rmdir's emptiness check now
  order the two.
- The hold step is the plan: it returns the points it holds, and the argv
  builder binds the protected paths that exist as themselves plus those, no
  missing path and no placeholder another run holds.
- A placeholder that cannot be made fails the wrap only where the code could
  create the path; a read-only file system, or a folder the user may not
  write and does not own, needs none.
- A wrap that fails after holding releases; the release is safe from two
  threads; holds and staging folders are made, and dropped, in folders
  opened without following a symlink below the writable bind (a folder
  swapped for one fails the wrap).
- OSSandboxResult.release is left out of equality.
The changelog bullet and CLAUDE.md's os_sandbox line say the code is the
sandbox's pid 1 and that a timed-out run keeps its placeholder. The README
and OSSandboxPolicy.writable_paths say that a writable path holding the home
directory gets placeholder folders at the start-up files it lacks, and that
a folder at ~/.bash_profile makes a login bash skip ~/.profile. The package
docstring's example releases only after the command ended on its own.
Round one joined any empty folder at a creation point as a placeholder and
removed it at release. That deleted a project's empty .git (CI's
test_git_dir_cannot_be_moved_aside), an empty .{app_name} or memory folder
and, with a writable home, an empty ~/.ssh or ~/.aws; and an empty folder
above a protected path (an empty working directory under a writable home,
an empty ~/.config) became the placeholder, bound read-only and removed.

- Only the protected path itself is joined while empty: that closes the
  mid-release race. A folder above it is a creation point only when it
  holds a hold, our own placeholder.
- A hold records whether a run made the folder. Joining a folder that holds
  no hold, or a kept one, writes a kept hold, and its release only unlinks
  it: a run removes no folder it did not make, whoever joins after it. The
  cost is an empty folder after the mid-release race.
- A protected folder that is there but takes no hold (one the user may not
  write) is bound as it is, rather than judged by its parent's owner and
  left unbound.
- ENOENT on the way to a creation point, or for the staging folder, means
  the place changed: looked at again, not refused.
- The executor keeps the holds when a status says a signal ended the run
  (negative, or 128 and above from a shell): a signal from outside that
  kills the shell or bwrap's monitor returns before the kernel has emptied
  the sandbox.

The timeout's live test let its child write at 1 s, before the 2 s timeout;
the child now waits 5 s, the file is checked at the timeout and after the
5 s, and the test runs both as the executor runs the command and behind a
shell that stays, so the kill reaches the shell alone.
The changelog bullet and CLAUDE.md's os_sandbox line say an empty protected
folder that was there is left as it was, and that a run a signal ended is
not released. The package docstring's example checks result.success before
it runs the command (a failed wrap's command is the bare one, unsandboxed)
and releases only after a status no signal set. The README sandbox
paragraph is wrapped again.
CI showed a timed-out run's code still running 4 s later, on Linux as on
macOS. subprocess.run kills only the process it started: the shell, or
bwrap's monitor. bwrap sets PR_SET_PDEATHSIG in the sandbox's pid 1 just
before execve, but its init outlived the monitor on the runner, and it is
in a session of its own (--new-session); on macOS a process the code forks
outlives sandbox-exec.

The executor now runs its command through _run_command: Popen in a session
of its own, communicate with the input and the timeout, and, on a timeout or
any interruption while it waits, SIGKILL first for each child of the
command's process group that left the group (read from /proc on Linux:
bwrap's sandbox init, whose death ends every process in its namespace),
then for the whole group. The result, the timeout's dict and the
environment are as before; the tests that recorded subprocess.run record
_run_command. On macOS, code that calls setsid() itself escapes the group.

The timeout's live test runs through _run_command on both backends and,
when a forked writer survives, shows the shells, sandboxes and Pythons
still running.

A hold that cannot be written into a folder another run holds now fails the
wrap unless the cause is one a run that made the folder would meet too
(EACCES, EPERM, EROFS): bound without a hold on a full disk, the folder
could be removed by that run while this one runs. A hold whose write fails
is removed. The staging folder's ENOENT branch has a test.
The changelog gets a Fixed bullet for the timeout, and its bubblewrap bullet
says that after a timeout or a signal the run's hold file stays in a folder
that was there. The README's sandbox paragraph says a placeholder is kept
after a signal too, holding the run's hidden hold file, and that a timeout
stops every process the code started. CLAUDE.md's executor line names
_run_command.
…walk

A run in a session of its own no longer got the hangup a closing terminal
sends the app's process group, nor a SIGTERM sent to the app: the app died
and the code ran on. The executor keeps the live runs, kills them all as a
timeout does (_kill_live_runs), at exit through atexit, and BaseCLIApp.run()
installs SIGHUP and SIGTERM handlers (_stop_runs_on_signals, once, from the
main thread) that kill them, then call the handler they replaced, or
restore the default and send the signal again, so the app still dies of it.
A library user who never runs the CLI gets the atexit cleanup alone.

- Windows has no os.killpg nor signal.SIGKILL: a timeout raised
  AttributeError and then waited forever. There a run gets no session and a
  timeout kills the process alone, as subprocess.run did; everywhere the
  process the run started is killed in a finally, whatever the walk did.
- The group is stopped (SIGSTOP) before the walk for children that left it:
  nothing in it can fork, leave or reap meanwhile, so a child found keeps
  its pid until it is killed. A walk that fails is logged; the group is
  killed all the same.
- A run that ends on its own kills its own group: what the code left in it
  ends with it, as bubblewrap's namespace ends with the code.
- The comment on why bubblewrap's init is killed by its host pid names the
  cause: an exec into another AppArmor label clears the parent-death signal
  (apparmor_bprm_committing_creds; Ubuntu's bwrap-userns-restrict execs the
  code with pix).
- A kept placeholder holds the run's hold file: the docstrings no longer
  call it empty. The os_sandbox example says how to run the command, in
  words, without subprocess.run's timeout, which leaves the sandbox running.
- The live tests run their command through _run_command, so the fork test
  runs on seatbelt too.
… namespace

The changelog's timeout bullet and the README's sandbox paragraph say that a
timeout, a KeyboardInterrupt in a main-thread caller, or the app's end stops
the code and the processes it started, and that code that calls setsid()
and forks again can escape outside bubblewrap's pid namespace (macOS, and
Linux without the sandbox). The bubblewrap bullet no longer calls a kept
placeholder empty. CLAUDE.md's executor and run() lines name the live runs
and the handlers.
…killed

- No handler is installed for a SIGHUP or SIGTERM the app ignores (under
  nohup): the hangup killed the running call while the app went on. The
  app and its runs now go on, each run to its timeout.
- The handler runs the chain, or the default's re-raise, whatever killing
  the runs does (try/finally).
- _kill_session kills the group in a finally: interrupted between its
  SIGSTOP and its SIGKILL (a KeyboardInterrupt, a second signal), a group
  left stopped, orphaned in a session of its own, would never be continued.
- A run is listed inside the try that unlists it, and _kill_live_runs skips
  a run already reaped.
- The setsid() wording says what escapes: outside a pid namespace, code that
  calls setsid() can outlive the run (on macOS a fork that calls it once;
  on Linux the walk only catches a direct child of a live group member).
  The executor's docstring says a normal exit waits for a running call, as
  before, and atexit reaches only runs on daemon threads.
The changelog's timeout bullet says a hangup or SIGTERM kills a live run,
an ignored one changes nothing, and a normal exit waits for the run, as
before, after which nothing it started is left. The README, the changelog
and CLAUDE.md say that outside a pid namespace code that calls setsid() can
outlive the run (on Linux the walk only catches a direct child of a live
group member).
@shoom1
shoom1 merged commit c53d2c6 into develop Nov 17, 2026
6 checks passed
@shoom1
shoom1 deleted the fix/bwrap-missing-protected-files branch November 17, 2026 23:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant