Updating to the next core-sdk release
Five core-sdk changes. Only two need code changes in bee-js; the rest change behaviour or hashes.
Status on core-sdk main: 1, 3 and 4 are merged. 2 is on
fix/unencrypted-mantaray-nodes-keep-an-all-zero-obfuscation-key. 5 is not committed yet.
1. Bytes.toJSON() returns hex; JSON parsing moved to parseJson()
JSON.stringify(bytes) now produces the hex string, and console.log(bytes) prints hex
(nodejs.util.inspect.custom). The old "parse my UTF-8 payload as JSON" behaviour is now
parseJson().
- Replace every
.toJSON() call that expected parsed content with .parseJson().
No occurrences in bee-js src/, so this is a docs + downstream-user break.
- README line 72 lists
toJSON among the shared primitive methods — update it.
2. MantarayNode generates a random obfuscation key per node
Unencrypted nodes used to keep an all-zero key, so structurally identical nodes shared one address —
every metadata-only '/' node and every empty manifest was the same chunk
(0cc878d32c96126d47f63fbe391114ee1438cd521146fc975dea1546d302b6c0), so client-side stamping burned
a single postage bucket until it filled while the batch was ~0% used.
Required fix in src/manifest/manifest.ts, loadRecursively: it copies targetAddress,
forks, path and parent from the downloaded node, but not the key. Add:
fork.node.obfuscationKey = loaded.obfuscationKey
Without it every loaded node re-marshals under a fresh key: a new address for every node in the tree
on every save, manifests stop deduping against their previous version, and re-saving an unchanged
manifest no longer returns the same root reference. This hits swarm-cli's manifest sync / add /
remove / merge, which all go through loadRecursively + saveRecursively.
Also: manifest references are no longer deterministic for newly created nodes, and the constant
empty-manifest reference above is gone. Anything pinning it breaks.
3. ChunkSplitter no longer drops chunks
finalize() failed to emit the children of a promoted node, so uploads of files just past a multiple
of 64 MiB (16 MiB encrypted) silently omitted up to the last 128 data chunks while still producing
the correct root hash. Those chunks are now emitted.
No API change. Uploads at those sizes now stamp and upload more chunks than before, and anything
uploaded at those sizes by the old code is incomplete on the network.
4. ChunkSplitter only-child rule fixed for redundancy
The promotion test used the node's physical ref capacity (128, or 64 encrypted) as its modulus, while
batches actually seal at getMaxShards(level, encrypted) — 119/107/97/39 unencrypted. Both directions
of the mismatch broke reads: a corrupted tail (the joiner read an intermediate node as a leaf) or a
silently dropped last chunk.
No API change, and level 0 is provably unchanged (identical root hashes across 22 sizes including
partial leaves and three-level trees). But with redundancy, files whose leaf count lands on a
maxShards boundary now produce a different — correct — root hash. Content uploaded at those sizes by
the old code is corrupt on the network and must be re-uploaded; pinned expected hashes for redundant
uploads must be regenerated.
bee-js has no code action here: it only builds new ChunkSplitter(onBatch) / ChunkSplitter.NOOP in
src/utils/chunk-stream.ts and .browser.ts, with no maxShards and no makeErasureBatch anywhere.
5. ChunkSplitter promotion no longer leaks per-level state
A promoted leaf was announced through onIntermediateChunk as if it were an intermediate chunk, and a
promoted intermediate chunk lost its parity flag, so makeIntermediateChunkHandler never wrote the
redundancy marker (span[7] = level | 0x80) on it — Bee and ChunkJoiner then read its parity refs as
data refs, losing RS recovery for that node and diverging from Bee's root hash.
onIntermediateChunk now fires fewer times (promoted leaves are not reported) and passes
hasParity = true where it previously passed false. A callback that counts calls or marks chunks
unconditionally will see different numbers. bee-js passes no callback, so it is unaffected.
Separate: swarm-cli bug, independent of this bump
src/command/manifest/sync.ts:50 constructs the splitter with maxShards left undefined while
redundancy is on:
const splitter = new ChunkSplitter(onBatch, undefined, false, makeIntermediateChunkHandler(level))
// must be getMaxShards(level, false)
undefined means 128 refs per node, but each node also takes 9 parity refs, so nodes overflow: a
130-leaf file round-trips as 16 KB out of 532 KB. Every redundant file over ~524 KB therefore gets an
expectedReference() unrelated to Bee's hash, is reported CHANGED, and is re-uploaded on every sync.
Fixing the core-sdk modulus does not fix this — the caller has to pass getMaxShards(level, false).
Updating to the next core-sdk release
Five core-sdk changes. Only two need code changes in bee-js; the rest change behaviour or hashes.
Status on
core-sdkmain: 1, 3 and 4 are merged. 2 is onfix/unencrypted-mantaray-nodes-keep-an-all-zero-obfuscation-key. 5 is not committed yet.1.
Bytes.toJSON()returns hex; JSON parsing moved toparseJson()JSON.stringify(bytes)now produces the hex string, andconsole.log(bytes)prints hex(
nodejs.util.inspect.custom). The old "parse my UTF-8 payload as JSON" behaviour is nowparseJson()..toJSON()call that expected parsed content with.parseJson().No occurrences in bee-js
src/, so this is a docs + downstream-user break.toJSONamong the shared primitive methods — update it.2.
MantarayNodegenerates a random obfuscation key per nodeUnencrypted nodes used to keep an all-zero key, so structurally identical nodes shared one address —
every metadata-only
'/'node and every empty manifest was the same chunk(
0cc878d32c96126d47f63fbe391114ee1438cd521146fc975dea1546d302b6c0), so client-side stamping burneda single postage bucket until it filled while the batch was ~0% used.
Required fix in
src/manifest/manifest.ts,loadRecursively: it copiestargetAddress,forks,pathandparentfrom the downloaded node, but not the key. Add:Without it every loaded node re-marshals under a fresh key: a new address for every node in the tree
on every save, manifests stop deduping against their previous version, and re-saving an unchanged
manifest no longer returns the same root reference. This hits swarm-cli's
manifest sync/add/remove/merge, which all go throughloadRecursively+saveRecursively.Also: manifest references are no longer deterministic for newly created nodes, and the constant
empty-manifest reference above is gone. Anything pinning it breaks.
3.
ChunkSplitterno longer drops chunksfinalize()failed to emit the children of a promoted node, so uploads of files just past a multipleof 64 MiB (16 MiB encrypted) silently omitted up to the last 128 data chunks while still producing
the correct root hash. Those chunks are now emitted.
No API change. Uploads at those sizes now stamp and upload more chunks than before, and anything
uploaded at those sizes by the old code is incomplete on the network.
4.
ChunkSplitteronly-child rule fixed for redundancyThe promotion test used the node's physical ref capacity (128, or 64 encrypted) as its modulus, while
batches actually seal at
getMaxShards(level, encrypted)— 119/107/97/39 unencrypted. Both directionsof the mismatch broke reads: a corrupted tail (the joiner read an intermediate node as a leaf) or a
silently dropped last chunk.
No API change, and level 0 is provably unchanged (identical root hashes across 22 sizes including
partial leaves and three-level trees). But with redundancy, files whose leaf count lands on a
maxShardsboundary now produce a different — correct — root hash. Content uploaded at those sizes bythe old code is corrupt on the network and must be re-uploaded; pinned expected hashes for redundant
uploads must be regenerated.
bee-js has no code action here: it only builds
new ChunkSplitter(onBatch)/ChunkSplitter.NOOPinsrc/utils/chunk-stream.tsand.browser.ts, with nomaxShardsand nomakeErasureBatchanywhere.5.
ChunkSplitterpromotion no longer leaks per-level stateA promoted leaf was announced through
onIntermediateChunkas if it were an intermediate chunk, and apromoted intermediate chunk lost its parity flag, so
makeIntermediateChunkHandlernever wrote theredundancy marker (
span[7] = level | 0x80) on it — Bee andChunkJoinerthen read its parity refs asdata refs, losing RS recovery for that node and diverging from Bee's root hash.
onIntermediateChunknow fires fewer times (promoted leaves are not reported) and passeshasParity = truewhere it previously passedfalse. A callback that counts calls or marks chunksunconditionally will see different numbers. bee-js passes no callback, so it is unaffected.
Separate: swarm-cli bug, independent of this bump
src/command/manifest/sync.ts:50constructs the splitter withmaxShardsleftundefinedwhileredundancy is on:
undefinedmeans 128 refs per node, but each node also takes 9 parity refs, so nodes overflow: a130-leaf file round-trips as 16 KB out of 532 KB. Every redundant file over ~524 KB therefore gets an
expectedReference()unrelated to Bee's hash, is reported CHANGED, and is re-uploaded on every sync.Fixing the core-sdk modulus does not fix this — the caller has to pass
getMaxShards(level, false).