Skip to content

Store local cache outputs by output ID instead of action ID - #26

Merged
richardartoul merged 2 commits into
mainfrom
store-local-outputs-by-output-id
Oct 27, 2026
Merged

richardartoul merged 2 commits into
mainfrom
store-local-outputs-by-output-id

Conversation

@richardartoul

@richardartoul richardartoul commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #25. localcache stored each output in a file named after its action ID, with its metadata (output ID and size) in a separate .meta file. #25 made each file's write atomic, but data and metadata are still two separate renames. Two unserialized writers of one action ID with different outputs (for example LOCK_TYPE=memory across processes with a non-reproducible action) can interleave like this:

A: rename data(A)
B: rename data(B)
B: rename meta(B)
A: rename meta(A)   -> data is B's, metadata says A's output ID and size

When the sizes differ, Go's size check turns this into a miss. When they match, the build silently uses the wrong bytes. This matters more once #23 turns a lock timeout into "proceed without the lock".

Change

  • Outputs are stored under their output ID (<subdir>/v2<outputID>-d). Go computes the output ID as the SHA-256 of the body, so concurrent writers of one output file write identical bytes. The .meta file under the action ID is now the pointer to that file, as in Go's own disk cache.
  • The data is written before the metadata, so whichever metadata wins references a complete file holding exactly its output. write() also skips rewriting an output file that already exists with the right size.
  • Metadata files use a .meta2 suffix so they never mix with earlier versions' .meta files (see Compatibility).
  • check() now stats the output file and treats a missing or wrongly sized file as a miss (one extra stat per lookup).
  • handlePut used to return the existing entry's path whenever the action ID was already cached. cmd/go records the returned DiskPath as the file for the output ID it sent (noteOutputFile(out, res.DiskPath) in cmd/go/internal/cache/prog.go), so for a different output ID that mapped the new output ID to the old bytes. It now short-circuits only when the output IDs match.
  • writeWithMetadata rejects an empty output ID instead of building a path from it.

Compatibility

fileFormatVersion is not bumped, because it is also part of the backend key and bumping it would invalidate every remote cache. Instead, metadata now lives in .meta2 files, so this layout and earlier versions never read each other's local entries. Versions sharing one cache directory, or a downgrade, each keep using their own entries. Earlier versions' .meta files and action-ID data files are left untouched.

Testing

  • TestLocalCacheConcurrentWritesOfOneActionIDWithDifferentOutputs: 16 writers put one action ID with distinct, same-size outputs while 4 readers keep checking that every hit's metadata matches the bytes in its file. On main it fails 5 of 5 runs, both from readers and in the final entry. Here it passes 30 of 30 runs with -race.
  • TestLocalCacheIgnoresEntriesOfEarlierVersions: an entry in the old layout is a miss, and writing the entry again leaves the old .meta and data files unchanged (fails with the .meta suffix).
  • TestLocalCacheIgnoresEntriesWithoutOutputFile: metadata whose output file is missing is a miss.
  • TestPut_ExistingEntryWithDifferentOutput: a second PUT of one action ID with a different output returns a DiskPath that holds the new output (fails on main).
  • go vet . and go test -race . ./pkg/... pass. The cloud-backend integration tests were not run locally.

The local cache wrote each output to a file named after its action ID, with
the metadata (output ID and size) in a separate file next to it. Two
unserialized writers of one action ID with different outputs could
interleave their renames so the metadata of one pointed at the data of the
other. When the outputs had the same size, Go's size check could not catch
it and the build used the wrong bytes.

Outputs are now stored under their output ID, the SHA-256 of their content,
and the metadata under the action ID acts as the pointer, as in Go's own
disk cache. The data is written before the metadata, so whichever metadata
wins references a complete file holding exactly its output.

check() now also stats the output file and treats a missing or wrongly
sized file as a miss. Entries written by earlier versions therefore fall
through to the backend and are rewritten, without bumping fileFormatVersion,
which would also invalidate the remote cache.

handlePut no longer returns the path of an existing entry holding a
different output ID: Go records the returned path as the file for the
output ID it sent.
Earlier versions store data under the action ID and point to it from
".meta" files. When versions share one cache directory, or after a
downgrade, an earlier version reading metadata written by this layout
returned a data file that was missing or held a different output.

Metadata now lives in ".meta2" files, so each layout only reads its own
entries and earlier versions keep working on theirs.
@richardartoul
richardartoul merged commit e6c2bfe into main Oct 27, 2026
1 check passed
richardartoul added a commit that referenced this pull request Oct 27, 2026
FSLockGroup waited at most one second for a lock and then failed the
request. The holder is usually doing a backend GET, which regularly takes
longer, so concurrent go processes requesting one action ID failed their
builds with "failed to acquire lock: context deadline exceeded" (#23).

Since #25 and #26, local cache writes are atomic and isolated, so the lock
only deduplicates work. When it can't be acquired, the request now runs
without it: a GET typically finds the entry the holder just cached, and a
PUT writes it again. Other lock errors fall back the same way. Fallbacks
are logged and counted in the exit stats.

DoWithLock wraps acquisition failures in locking.ErrLockNotAcquired so the
server can tell them apart from errors returned by the function. The wait
is configurable with -lock-timeout / LOCK_TIMEOUT, and the default goes
from 1s to 5s, since waiting a little longer usually ends in a local hit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant