Offer: postage-batch lifecycle helpers (apply-detection, erasure-aware + exact sizing, bucket occupancy) — would a PR be welcome?

[petfold]

Context

I'm a downstream user: bee-py provides feeds/SOC and key handling for two of my
projects (recordstore,
ucomm), and I maintain
swarmfs (fsspec backend) and
swarmlite (SQLite over Swarm) alongside it.
I opened #1 and #2 earlier this month.

Over the last few days I worked through the postage-batch lifecycle — renewing a
published root rather than buying a batch for it — and ended up implementing a layer
on top of exactly the endpoints bee.postage and bee.storage already wrap. Along
the way I verified a set of behaviours against a live Bee 2.8.1 and against bee's Go
source, several of which surprised me.

Since bee-py already covers the endpoints and the (size, duration) helpers, this
knowledge arguably belongs here rather than in a fourth Python implementation of it.
Would a PR be welcome, and in what shape? Details below so the offer is
checkable rather than vague — everything is either a citation into bee's source or a
live measurement.

1. "Submitted" is not "applied" (the one I'd fix first)

top_up_batch, dilute_batch and storage.extend_storage_duration return None
as soon as the PATCH returns. The node needs time to index the chain event: I
measured 41.8 s, 52.5 s and ~40 s across four real topups. During that window the
wallet is already debited while GET /stamps/{id} still reports the old values — so
a caller who reads straight afterwards sees no change and cannot distinguish "not
indexed yet" from "silently failed". My first implementation had exactly that bug.

An optional wait=True (mirroring bee-js's waitForUsable on purchase) or a
wait_for_topup(batch_id, before) helper would remove a footgun that every caller
otherwise re-discovers.

2. A batch's amount is not a reliable ledger

This is the part I'd most like to save someone else the trouble of finding.

GET /stamps reports the local stamp issuer's BatchAmount
(pkg/api/postage.go
uses v.Amount()), and HandleTopUp increments that in memory without persisting
it (pkg/postage/service.go:186).
Observed live: hours after two confirmed topups, amount had reverted to the
creation value (32954342400) while batchTTL still reflected both (40.1 days). A
third topup then accumulated from the reverted base, exactly as that mechanism
predicts.

Two consequences:

• Never derive remaining life from amount. Separately, it counts from
  blockNumber, so amount / currentPrice * 5 is total lifetime with the elapsed
  part already spent — 27.78 d implied on a 3.79-d-old batch whose reported TTL was
  24.0 d (the three agree to 0.06%). batchTTL (from the batchstore, via
  estimateBatchTTLFromID) is the authoritative field.
• Detection must not watch amount alone, or a paid-for topup can hang until
  timeout. Accepting either an amount increase or a batchTTL jump works.

3. Sizing depends on the erasure level (and encryption more than I expected)

A batch is filled by stamped chunks — leaves plus per-level parity plus
intermediates plus dispersed root replicas — not by payload bytes. From bee's own
appendix-F tables
(pkg/file/redundancy/level.go),
the inflation is:

level	plain	encrypted
MEDIUM	1.08	1.12
STRONG	1.20	1.30
INSANE	1.32	1.52
PARANOID	3.45	5.88

The encrypted column is easy to get wrong: redundancy.New() takes maxShards from
the encrypted table but maxParity from the plain one
(redundancy.go:47-55).
I had this as "encryption adds ~1%" for a while; it is up to 1.7×.

EFFECTIVE_SIZE_BREAKPOINTS answers a different and also-useful question (payload
capacity for encrypted, medium-erasure batches, mirroring bee-js), so this would be
complementary rather than a replacement.

And sizing need not be probabilistic at all. A chunk's bucket is
BigEndian.Uint32(addr[:4]) >> (32 - bucket_depth)
(stampissuer.go:384),
so when the chunk addresses are known — computable offline — the exact bucket
histogram gives a 0 %-risk depth. Measured on a real batch: estimating from bytes put
depth 18 at a 34 % overflow risk for content whose true histogram fit depth 18
exactly. Only node-generated parity stays irreducibly probabilistic.

4. 402 "batch is overissued" is recoverable, and reads like it isn't

On an immutable batch a chunk hashing into a full bucket is refused
(ErrBucketFull → 402, stampissuer.go:186),
but the batch is not destroyed and everything already stamped stays paid for.
Dilution preserves the bucket counters, so depth + 1 doubles every bucket and the
same upload then succeeds with the same root. On a mutable batch the counter
instead resets and the stamp index is reused, silently invalidating whatever was
stamped there before — so immutability converts quiet data loss into a loud 402.

An error message that says this turns a dead end into a two-step recovery.

5. GET /stamps/{id}/buckets

Not currently wrapped, as far as I can see. It returns all 65 536 counters and is the
ground truth for how much a batch can still take; utilizationRatio only summarises
the fullest one. Useful before adding to an existing batch, where no estimate is
needed.

What I'm offering

The above is implemented, documented and tested in swarmfs 0.4.0 (with the numbers
above pinned as regression tests, including the live measurements). I'd be glad to
port any subset here as small, reviewable PRs — my guess at the useful order:

1. wait=/poll helpers for topup and dilute, plus the amount-is-not-a-ledger note
2. pure sizing helpers (erasure-aware stamped_chunks, bucket histogram, exact depth)
3. stamps/{id}/buckets wrapper
4. the recoverable-402 message

swarmfs is BSD-3; I'm the copyright holder and would relicense the contribution MIT
to match this project. Happy to follow whatever style/typing conventions you prefer —
the existing code is noticeably careful, which is why I'd rather add to it than fork
the knowledge.

The practical question

#1 and #2 have had no reply, and the last push here was in May, so I don't want to
assume bandwidth. Whichever of these is true is genuinely fine — I'd just like to
know which:

• Contributions welcome → tell me the preferred shape and I'll open PRs.
• Low bandwidth right now → I can leave this issue as documentation; the citations
  above stand on their own for whoever gets there next.
• Not maintained → say so and I'll factor the lifecycle logic into a small
  zero-dependency module instead, designed so bee-py (or anything else) can use it
  without taking on a transport dependency. I'd rather do that with your blessing than
  in silence.

Either way, thanks for bee-py — the BMT/SOC and key handling have been solid against
a live 2.8.1, and it saved me writing them.