GET /bzz/<ref>/ can 404 off-node, and the 404 is indistinguishable from a missing index document

[crtahlin]

GET /bzz/<ref>/ on a single-file upload can 404 on a node that did not upload it, while
every other route for the same reference answers 200. The 404 is indistinguishable from
"no index document set", so there is no way to tell from the outside what went wrong.

Measured on a 5-node local cluster, bee v2.8.2, uploaded via the queen:

queen  /bzz/<ref>/               200
worker /bzz/<ref>/               404   {"message":"address not found or incorrect","code":404}
worker /bzz/<ref>/manifest.json  200
worker /bytes/<ref>              200
worker /chunks/<ref>             200

The data is entirely present on the worker. Only the root form fails.

Why

GET /bzz/<ref>/ is the only retrieval path that has to load the manifest's / metadata
node — pkg/api/bzz.go:624-643 at v2.8.2:

if pathVar == "" {
    if indexDocumentSuffixKey, ok := manifestMetadataLoad(ctx, m, manifest.RootPath, manifest.WebsiteIndexDocumentSuffixKey); ok {
        ...
    }
    jsonhttp.NotFound(w, "address not found or incorrect")
    return
}

For a single-file upload that node is created at bzz.go:266-269 with
swarm.ZeroAddress and only the index-document metadata:

rootMetadata := map[string]string{
    manifest.WebsiteIndexDocumentSuffixKey: queries.FileName,
}
err = m.Add(ctx, manifest.RootPath, manifest.NewEntry(swarm.ZeroAddress, rootMetadata))

The metadata is serialised into the parent's fork record, not into this node's own
chunk, so the chunk itself carries no information — and with a zero obfuscation key it
serialises identically for every unencrypted single-file upload. LookupNode fetches it
anyway and then discards it, because the recursion terminates immediately on an empty
remaining path.

So the root redirect for every single-file POST /bzz on the network depends on one
shared, content-free chunk. We believe its address is
0cc878d32c96126d47f63fbe391114ee1438cd521146fc975dea1546d302b6c0 — derived from
marshal.go rather than instrumented, but GET /chunks/<that address> does return 200 on
our cluster, which is consistent.

That also explains why the failure looks intermittent: once any upload has pushed that
chunk successfully, every later upload on the same cluster inherits it. On a
re-measurement the worker returned 200 for the root form.

The part that is squarely a bug

manifestMetadataLoad (bzz.go:820-836) returns ("", false) for both "the node could
not be fetched" and "the key is not set":

me, err := manifest.Lookup(ctx, path)
if err != nil {
    return "", false
}

Both paths then reach the same jsonhttp.NotFound(w, "address not found or incorrect")
with no distinguishing log line beyond bzz download: address not found or incorrect. A
retrieval failure and a manifest that legitimately has no index document are reported
identically. That cost us a day of looking in the wrong place.

Suggested fix: propagate the lookup error and answer 500 (or a 404 with a distinct
message) when the / node cannot be loaded, keeping the current 404 for a manifest that
genuinely has no index document.

Two smaller observations

1. There is no "manifest has exactly one entry, serve it" fallback in the empty-path
   branch. Given a single-file upload always has its index document set to the filename,
   the only way this branch 404s in practice is the retrieval failure above.
2. Swarm-Index-Document and Swarm-Error-Document are in the CORS allowed-headers list
   (pkg/api/api.go:605) but are only read in dirs.go, inside the directory handler. The
   single-file upload header struct (bzz.go:70-81) does not include them, so on a
   single-file POST /bzz they are accepted by preflight and silently ignored. Rejecting
   them, or documenting that they are directory-only, would save someone the experiment.

Workaround, for anyone who finds this

Request the entry by name — GET /bzz/<ref>/<filename>. It never loads the / node.

Note that GET /bytes/<ref> is not a workaround for a /bzz upload: it returns the
manifest node's own bytes rather than the file, so it answers 200 with the wrong content
and the failure surfaces much later.

---

Filed with AI assistance.