Skip to content

Node never starts on an eCryptfs home: silent infinite write loop, RPC never binds, fills the disk #151

Description

@Wired4ncer

Reported by: operator running BitWindow + Thunder on the eCash alphanet, 2026-09-05.

What you see as a user

You open Thunder in BitWindow and it sits on "Initialising" forever. There is no error
message, no failed-to-start dialog, nothing in the UI to act on. It looks like a slow sync.

It is not syncing. The Thunder node process is in an infinite loop writing to its own
database, and it will consume every byte of free space on the drive — roughly 79 MB/s,
about 4.7 GB per minute — until the disk fills or the process dies. On this machine it wrote
45 GB in under twenty minutes before eCryptfs returned a write error and killed it.

The trigger is that Thunder's data directory sits on an eCryptfs-encrypted home, which is
the default for "encrypt my home folder" on Ubuntu/Mint and its derivatives. Nothing about
the setup is exotic and nothing warns you.

Environment

Thunder thunder_app 0.17.5, binary 41,178,832 bytes, 2026-09-03
BitWindow AppImage 235,377,856 bytes, 2026-09-05
bip300301-enforcer bundled with the above BitWindow
Network alphanet
OS Linux Mint 22.3, kernel 7.0.0-31-generic, x86_64, page size 4096
Datadir ~/.local/share/thunder — on eCryptfs
eCryptfs mount ecryptfs_cipher=aes, ecryptfs_key_bytes=16, ecryptfs_unlink_sigs

Mainchain and enforcer were healthy the whole time and are not implicated:
bitcoind at height 996,845, verificationprogress: 1, initialblockdownload: false;
enforcer answering on :50051, with Thunder's gRPC connection ESTABLISHED to it throughout.

Reproduction

  1. Put Thunder's datadir on an eCryptfs-encrypted home (Ubuntu/Mint "encrypt home folder").
  2. Start Thunder against a synced mainchain + enforcer.
  3. Thunder never binds its RPC port. Any client polling it — including BitWindow — hangs on
    "Initialising" indefinitely.

The decisive test — same binary, same arguments, only the filesystem differs

thunder --datadir=<DIR> --headless --mainchain-grpc-url=http://localhost:50051 \
        --network=alphanet --rpc-addr=127.0.0.1:<PORT> --net-addr=0.0.0.0:<PORT>
ext4 eCryptfs
RPC bound yes, at t+10s never
written in first 60 s 102 KB, then flat 5.0 GB, climbing linearly
data.mdb blocks allocated 136, flat 9,840,856, climbing
last log line connecting to peers, syncing Instantiating node struct

Where it hangs

The last thing Thunder ever logs is:

thunder_app::app: app/app.rs:278: Instantiating node struct

Two independent signals show this is a synchronous loop on the main thread, before the
async runtime starts
:

  • Over 152 s the main thread burned 149 s of CPU while all 24 tokio-rt-worker threads
    sat at exactly 0 ticks
    . The runtime never runs, so the RPC server is never created.
  • I/O counters over the same window: write_bytes +12.0 GB, read_bytes +0. It writes
    continuously and never reads back what it wrote.

On an earlier run the process exited with exit_code=-1 at the same log line, which
BitWindow's orchestrator reported as
reset restart failed error="wait for thunder: thunder exited with code -1".

Mechanism

Thunder's LMDB environment declares a 128 GiB map size (mm_mapsize = 0x2000000000,
read from the meta page). LMDB creates the file and relies on a sparse allocation to the map
size, then writes through the mmap.

On ext4 that works: the file reports size=137438953472 with only blocks=136 actually
allocated — a 128 GiB sparse file holding 68 KB of data.

On eCryptfs it does not. The file's apparent size stays at 8192 bytes (two meta pages,
i.e. a freshly created env) while allocated blocks grow without bound:

stat:  Size: 8192      Blocks: 51012408     <- 8 KB apparent, ~26 GB allocated
       Size: 8192      Blocks: 92960520     <- minutes later, ~47 GB allocated

So every write goes somewhere that never becomes part of the file; the env never takes
shape, initialisation never completes, and the loop never terminates. The kernel confirms
the write path is failing, logged at the moment the process died:

ecryptfs_encrypt_page: Error attempting to write lower page; rc = [-4]
ecryptfs_write: Error encrypting page; rc = [-4]

(-4 is -EINTR.) eCryptfs is known to have incomplete mmap/writeback and sparse-file
semantics; LMDB is entirely mmap-based, so the two are fundamentally incompatible.

Impact

  1. Data loss risk from disk exhaustion. This is the serious one. A silent 79 MB/s write
    loop fills any drive. It is especially dangerous because the UI says "Initialising", so a
    user will reasonably leave it running and walk away.
  2. No diagnosable error. Nothing surfaces to the user, the log, or BitWindow. The one
    log line that exists (Instantiating node struct) is DEBUG level, and the default file
    log level is WARN — so by default nothing is written to the log file at all.
  3. Affects a common desktop configuration (encrypted home) that Thunder's own default
    datadir lands inside.

Workaround (verified)

Move only the node DB to a non-eCryptfs filesystem, keeping the datadir — and therefore
wallet.mdb and its seed — encrypted:

ln -s /path/on/ext4/thunder-data ~/.local/share/thunder/data.mdb

Verified under BitWindow's own orchestration: starting target binary → connection established in one second, RPC answering, writes flat at ~196 KB, node DB correctly
sparse on ext4.

⛔ Do not relocate the whole datadir as a workaround — that puts wallet.mdb and the
Thunder seed on unencrypted disk.

Suggested fixes, roughly in order of value

  1. Bound the loop. Whatever retries during node instantiation must fail after N attempts
    or when the env is not growing, rather than spinning forever. Nothing should be able to
    write 45 GB without making progress.
  2. Check the write actually landed. After creating/growing the env, verify the file's
    apparent size matches expectation and error out clearly if not. This catches eCryptfs and
    any other filesystem with broken sparse/mmap semantics, without special-casing any of them.
  3. Detect and warn at startup. Read the datadir's filesystem type; if it is ecryptfs
    (or another known-incompatible mmap target), refuse to start with a message naming the
    problem and suggesting a different datadir.
  4. Surface failures to the frontend. BitWindow should be able to show "Thunder failed to
    start: " instead of an indefinite "Initialising".
  5. Log the datadir and its filesystem at INFO on startup, so this is a one-line diagnosis
    next time rather than a day of bisecting.

Items 1 and 2 are the ones that matter — they fix the whole class, not just eCryptfs.

Happy to help

I still have the failing configuration available and can run further diagnostics on request
— strace, a debug build, MDB_* env-var experiments, or a test against any patch.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions