geckodriver killed with SIGABRT (broken-pipe panic) on browser teardown — stdout logged via FIFO

[vringar]

Summary

geckodriver is killed with SIGABRT (exit 134) and dumps core every time a browser is torn down or recycled. The crash is a Broken pipe panic caused by OpenWPM logging geckodriver's stdout through a named FIFO whose reader (FirefoxLogInterceptor, a daemon thread) can close before geckodriver stops writing. Each recycle of N parallel browsers produces N coredumps.

It does not corrupt crawl data — geckodriver was being shut down anyway — but it spams the system coredump store and muddies process-exit bookkeeping (134 instead of a clean exit).

Environment

• OpenWPM v0.35.0 (Firefox 152)
• geckodriver 0.37.0 (conda-forge build)
• Linux, 3 parallel browsers

Evidence

Nine coredumps collected over one crawl, all byte-for-byte the same failure (same geckodriver version, same rustc build hash). Strings recovered from each core:

thread 'webdriver dispatcher' panicked at library/core/src/panicking.rs:233:5:
failed printing to stdout: Broken pipe (os error 32)
panic in a destructor during cleanup

Crashes arrive in bursts of 3 (= the 3 parallel browsers), once per recycle interval.

Root cause

deploy_firefox.py routes geckodriver's stdout into a FIFO created by FirefoxLogInterceptor:

# openwpm/deploy_browsers/deploy_firefox.py
webdriver_interceptor = FirefoxLogInterceptor(browser_params.browser_id)
webdriver_interceptor.start()
...
driver = webdriver.Firefox(
    options=fo,
    service=Service(
        executable_path=geckodriver_path,
        log_output=open(webdriver_interceptor.fifo, "w"),   # <-- FIFO write end
    ),
)

FirefoxLogInterceptor opens the read end of that FIFO on a daemon thread (openwpm/deploy_browsers/selenium_firefox.py):

class FirefoxLogInterceptor(threading.Thread):
    def __init__(self, browser_id):
        ...
        self.fifo = mktempfifo(suffix=".log", prefix="owpm_driver_")
        self.daemon = True          # <-- killed abruptly at interpreter/process teardown
    def run(self):
        with open(self.fifo, "rt") as f:
            for line in f:
                self.logger.debug(...)

At browser teardown/recycle, the BrowserManager process (and with it the daemon interceptor thread) exits, closing the read end of the FIFO. If geckodriver is still alive and emits even one more log line, its write() to stdout returns EPIPE.

geckodriver (Rust) turns that into a panic when the stdio print macro fails — and because a Drop handler run during unwinding also writes to stdout, it double-panics (panic in a destructor during cleanup), which calls abort() unconditionally → SIGABRT + coredump.

In other words: a FIFO makes geckodriver's writes depend on the reader staying alive, and the reader (a daemon thread) is not guaranteed to outlive the writer.

Suggested fix

Back the interceptor with a regular file instead of a FIFO. Writes to a regular file never raise EPIPE, so geckodriver can no longer be aborted by reader teardown — regardless of thread/process exit ordering. The interceptor tails the file and stops on request.

--- a/openwpm/deploy_browsers/selenium_firefox.py
+++ b/openwpm/deploy_browsers/selenium_firefox.py
@@
-import errno
 import logging
 import os
 import tempfile
 import threading
@@
 class FirefoxLogInterceptor(threading.Thread):
     """
-    Intercept logs from Selenium and/or geckodriver, using a named pipe
-    and a detached thread, and feed them to the primary logger for this
-    instance.
+    Intercept logs from Selenium and/or geckodriver, using a regular log
+    file and a detached thread, and feed them to the primary logger for
+    this instance.

     A regular file (rather than a FIFO) is deliberate: geckodriver writes
     its log to this path's write end, and a FIFO whose read end has closed
     would make those writes fail with EPIPE -- which geckodriver escalates
     to a panic and SIGABRT. A regular file never raises EPIPE, so the
     driver is unaffected by interceptor/teardown ordering.
     """

     def __init__(self, browser_id: BrowserId) -> None:
         threading.Thread.__init__(self, name=f"log-interceptor-{browser_id}")
         self.browser_id = browser_id
-        self.fifo = mktempfifo(suffix=".log", prefix="owpm_driver_")
+        fd, self.logfile = tempfile.mkstemp(suffix=".log", prefix="owpm_driver_")
+        os.close(fd)
         self.daemon = True
+        self._stop = threading.Event()
         self.logger = logging.getLogger("openwpm")
-        assert self.fifo is not None

     def run(self) -> None:
-        assert self.fifo is not None
         try:
-            with open(self.fifo, "rt") as f:
-                for line in f:
-                    self.logger.debug(
-                        "BROWSER %i: driver: %s" % (self.browser_id, line.strip())
-                    )
-                    if self.fifo is not None:
-                        os.unlink(self.fifo)
-                        self.fifo = None
+            with open(self.logfile, "rt") as f:
+                while not self._stop.is_set():
+                    line = f.readline()
+                    if line:
+                        self.logger.debug(
+                            "BROWSER %i: driver: %s"
+                            % (self.browser_id, line.strip())
+                        )
+                    else:
+                        self._stop.wait(0.1)
         except Exception:
             self.logger.error("Error in LogInterceptor", exc_info=True)
         finally:
-            if self.fifo is not None:
-                os.unlink(self.fifo)
-                self.fifo = None
+            try:
+                os.unlink(self.logfile)
+            except OSError:
+                pass
+
+    def stop(self) -> None:
+        self._stop.set()

--- a/openwpm/deploy_browsers/deploy_firefox.py
+++ b/openwpm/deploy_browsers/deploy_firefox.py
@@
         service=Service(
             executable_path=geckodriver_path,
-            log_output=open(webdriver_interceptor.fifo, "w"),
+            log_output=open(webdriver_interceptor.logfile, "w"),
         ),

mktempfifo can then be deleted if it has no other users.

For the tidiest shutdown, call webdriver_interceptor.stop() from the browser teardown path (browser_manager.py) after geckodriver is killed, so the final lines are drained and the temp file removed promptly; the daemon flag still guarantees the thread never blocks process exit.

Alternative (smaller, less complete)

Keep the FIFO but guarantee the reader outlives the writer: make the interceptor non-daemon and ensure geckodriver is terminated (so the FIFO gets EOF) before the process exits. This narrows the race but does not close it — geckodriver can still log during its own signal handling after the reader is gone. The regular-file approach removes the failure mode entirely.

Upstream note

The deeper cause is in geckodriver/Rust: a broken pipe on stdout should be a clean exit, not a panic-to-abort. That is worth reporting to Mozilla separately, but OpenWPM can fully avoid triggering it today with the change above.