Add unity-sync, a mirroring CLI for the Unity Asset Store

.github/workflows/ci.yml

@@ -0,0 +1,45 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+  # Called by release.yml as its gate, so the bar for tagging is exactly the bar for
+  # merging rather than a second copy that drifts.
+  workflow_call:
+
+permissions:
+  contents: read
+
+# Superseded runs on the same ref are pointless once a newer commit exists.
+concurrency:
+  group: ci-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+
+      - run: go build ./...
+
+      - name: gofmt
+        run: |
+          unformatted=$(gofmt -l .)
+          if [ -n "$unformatted" ]; then
+            echo "unformatted files:"
+            echo "$unformatted"
+            exit 1
+          fi
+
+      - run: go vet ./...
+
+      # The syncer fans out downloads under a semaphore and persists the lockfile from
+      # each goroutine, so the suite is worth running under the detector. No network and
+      # no session are needed: everything runs against httptest servers and the
+      # committed, scrubbed fixtures.
+      - run: go test -race ./...

.github/workflows/release.yml

@@ -0,0 +1,74 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - 'v*'
+
+permissions:
+  contents: write
+
+jobs:
+  check-branch:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+      - name: Verify tag is on main
+        run: |
+          git fetch origin main
+          git merge-base --is-ancestor ${{ github.sha }} origin/main || {
+            echo "Error: tag must point to a commit on main"
+            exit 1
+          }
+
+  test:
+    needs: check-branch
+    uses: ./.github/workflows/ci.yml
+
+  release:
+    needs: test
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+
+      - name: Get version from tag
+        id: version
+        run: echo "VERSION=${GITHUB_REF_NAME#v}" >> "$GITHUB_OUTPUT"
+
+      - name: Build release artifacts
+        env:
+          VERSION: ${{ steps.version.outputs.VERSION }}
+        run: |
+          mkdir -p dist
+
+          platforms=(
+            "darwin/amd64/mac-intel"
+            "darwin/arm64/mac-apple"
+            "linux/amd64/linux-intel"
+            "linux/arm64/linux-arm64"
+            "windows/amd64/win"
+          )
+
+          for p in "${platforms[@]}"; do
+            IFS='/' read -r goos goarch label <<< "$p"
+            bin="unity-sync"
+            if [ "$goos" = "windows" ]; then bin="unity-sync.exe"; fi
+
+            CGO_ENABLED=0 GOOS=$goos GOARCH=$goarch go build \
+              -ldflags "-X main.version=${VERSION}" \
+              -o "dist/$bin" .
+            (cd dist && zip "unity-sync-${VERSION}-${label}.zip" "$bin" && rm "$bin")
+          done
+
+      - name: Create GitHub release
+        uses: softprops/action-gh-release@v2
+        with:
+          generate_release_notes: true
+          files: dist/*.zip

CLAUDE.md

@@ -0,0 +1,82 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this is
+
+`unity-sync` is a Go CLI that mirrors the assets owned on the Unity Asset Store into a
+local library, downloading only what changed since the last run. See `README.md`
+(user-facing) and `docs/design.md` (the authoritative design doc: measured store
+behaviour, identity rules, the failure model). Read `docs/design.md` before changing
+enumeration, the lockfile, the cache layout, or the download guards.
+
+Browsing the mirrored library is a separate tool,
+[quarry](https://github.com/curbol/quarry). This repo acquires files; quarry reads them.
+
+## Build & test
+
+```bash
+go build -o unity-sync .        # requires Go 1.26+, no cgo
+go test ./...                   # full suite, fully offline
+go test -race ./...             # what CI runs
+go test ./internal/syncer/ -run TestClassify -v
+go vet ./...
+gofmt -l .
+```
+
+No Makefile or task runner; use the `go` toolchain directly. The suite needs no network
+and no session: everything runs against `httptest` servers and the committed, scrubbed
+fixtures in `testdata/store/`.
+
+## Architecture
+
+`main.go` `run()` parses flags and dispatches `select`, `status`, `sync`, `list`, `update`
+and `version`, returning an exit code alongside its error. Layered `internal/` packages,
+each with a package doc comment stating its contract:
+
+- `model` — domain types and the identity rules. Carries `id` (the store product id) and
+  deliberately not `productId`, which is a different value no endpoint accepts.
+- `config` — user settings by precedence: defaults → `config.toml` → env → flags. There is
+  no browser session default, because a browser session cannot work here.
+- `session` — builds the Cookie header from a pasted curl file or a `cookies.txt`, and
+  asserts the `LS` cookie is present before any request.
+- `retry` — backoff policy. `retry.Permanent` lets a caller stop on a body-based verdict
+  that the status code alone would have retried.
+- `unitypackage` — reads the store descriptor from a package's gzip FEXTRA field.
+- `store` — the Asset Store client and the response-level download guards.
+- `cache` — the local mirror. Two-phase writes (`Store` → `Commit`/`Discard`), adopt by
+  scan, relocate on rename, temp sweep, root confinement.
+- `lockfile` — `unity-sync.lock.json`, advertised fields kept apart from resolution fields.
+- `manifest` — `unity-sync.toml`, the committed allowlist keyed by asset id.
+- `syncer` — orchestration, the pure `classify`, and the semantic download guards.
+- `web` — the `select` page.
+- `selfupdate` — the `update` subcommand.
+- `fixtures` + `cmd/scrubfixtures` — regenerate PII-free `testdata/` from raw captures.
+
+### Key invariants (don't break these)
+
+- **`LS` is the credential.** Not the NextAuth session token, which neither endpoint
+  consults. Its absence is reported before any request, because the store answers a
+  missing `LS` with an opaque 500.
+- **No store client follows a redirect.** An unauthenticated download 302s to Unity's
+  OAuth page. `selfupdate` is the deliberate exception: it talks to GitHub, whose asset
+  API 302s to a signed CDN URL by design.
+- **Downloads ask for `Accept-Encoding: identity`.** The endpoint honours gzip by
+  gzipping the already-gzipped package, and Go will not decode an encoding the caller
+  requested.
+- **`resolvedVersionId` is the diff key**, not the advertised `version.id`. The advertised
+  value refreshes every run; pairing a refreshed id with an unresolved entry's file would
+  mark it current forever.
+- **Nothing unverified reaches a real cache path.** `cache.Store` does not rename;
+  `Commit` does, after the syncer's guards pass.
+- **A failed download fails its asset, not the run**, and a pulled asset does not make the
+  run exit non-zero.
+- **Only `select` writes the manifest.** `status` and `sync` read it.
+- **No account data in the repo.** Sessions and raw captures stay out; the
+  `internal/fixtures` guard test fails the build if any reaches `testdata/`.
+
+## Editing testdata
+
+Don't hand-edit `testdata/store/*.json`. They are generated by
+`go run ./cmd/scrubfixtures` from git-excluded raw captures. Regenerate rather than patch,
+and keep the guard test green.

README.md

@@ -0,0 +1,167 @@
+# unity-sync
+
+A Go CLI that mirrors the assets you own on the Unity Asset Store into a local library,
+downloading only what changed since the last run. It is the download manager the Asset
+Store does not give you outside the Editor. Design: `docs/design.md`.
+
+## Install
+
+Grab the latest release into `~/.local/bin` (private repo, so it uses your `gh` login or
+`GITHUB_TOKEN`):
+
+```bash
+gh api repos/curbol/unity-sync/contents/install.sh --jq .content | base64 -d | bash
+```
+
+Then update in place:
+
+```bash
+unity-sync update           # latest release
+unity-sync update 0.2.0     # a specific version
+unity-sync version          # what is installed
+```
+
+Releases are cut by pushing a `v*` tag; a workflow builds the cross-platform binaries and
+publishes them.
+
+## Build from source
+
+```bash
+go build -o unity-sync .
+```
+
+Requires Go 1.26+. No cgo. To stamp a version into a local build:
+`go build -ldflags "-X main.version=0.2.0" -o unity-sync .`
+
+## One-time setup
+
+Two kinds of state, kept apart:
+
+- **User config** (session, machine defaults) lives *outside* any project, resolved as
+  `--config <dir>` › `$UNITY_SYNC_CONFIG_DIR` › `$XDG_CONFIG_HOME/unity-sync` ›
+  `~/.config/unity-sync`.
+
+  ```bash
+  mkdir -p ~/.config/unity-sync
+  cp config.example.toml ~/.config/unity-sync/config.toml
+  ```
+
+- **Project manifest** (`unity-sync.toml`: which assets this project draws from) lives
+  *in* the project that consumes the assets, committed to its repo. unity-sync finds it by
+  walking up from the working directory, or point at it with `--manifest`. Its lockfile
+  (`unity-sync.lock.json`) is written beside it. It carries no account identity. See
+  `unity-sync.example.toml`.
+
+Packages are cached at `$XDG_DATA_HOME/unity-sync` (`~/.local/share/unity-sync`) by
+default; override with `library_path`, `UNITY_SYNC_LIBRARY`, or `--library`.
+
+## Session
+
+The store gates everything behind your signed-in session, and the cookie it actually
+checks — `LS` — is a session cookie that lives only in your browser's memory. No browser
+cookie database has it, so unity-sync cannot read your session automatically the way a
+tool for some other store might. You paste one instead:
+
+In DevTools → Network, right-click any `assetstore.unity.com` request → Copy → Copy as
+cURL, and save it:
+
+```bash
+$EDITOR ~/.config/unity-sync/session.curl     # paste, save
+unity-sync status
+```
+
+A Netscape `cookies.txt` export works too, as long as your exporter keeps HttpOnly rows
+(they are written with a `#HttpOnly_` prefix). Point at either file with `--session`, with
+`session_source` in `config.toml`, or just save it as `session.curl` or `cookies.txt` in
+the config dir, where unity-sync looks by default.
+
+If the file is missing the `LS` cookie, unity-sync says so before making any request,
+because the store's own answer in that case is an HTTP 500 that reads like a server fault.
+
+A pasted session expires. When it does, re-copy it.
+
+## Commands
+
+```bash
+unity-sync select   # pick which assets to mirror (opens a local page)
+unity-sync status   # what a sync would change; downloads nothing, changes nothing
+unity-sync sync     # download the delta and update the lockfile
+unity-sync list     # print the current lockfile
+```
+
+Useful flags: `--manifest <path>`, `--only <asset-slug-glob>`, `--library <dir>`,
+`--concurrency <n>`, `--verify`, `--dry-run` (makes `sync` behave like `status`),
+`--config <dir>`, `--session <file>`, `--addr <host:port>` (the `select` page's address).
+
+## Selecting assets
+
+Selection is opt-in: an asset is mirrored only once you enable it. This matters more here
+than it might sound — a typical Asset Store account owns hundreds of packages and tens of
+gigabytes, and individual packages reach 23 GB.
+
+`unity-sync select` lists every owned asset with its thumbnail and a checkbox and writes
+the `[[asset]]` entries into `unity-sync.toml`:
+
+```toml
+[[asset]]
+  id = "115488"
+  name = "Quick Outline"
+  enabled = true
+```
+
+Entries key on the asset id, so a publisher renaming their asset cannot silently deselect
+it. Newly-bought assets appear disabled on the next `select`, so buying something never
+downloads it behind your back. Hand-editing the file is fine.
+
+`select` is the only command that writes the manifest. `status` and `sync` only read it.
+
+## What it does
+
+- Lists every owned asset with its current version inline, so a run that changes nothing
+  costs only the enumeration: one bootstrap, a page request per 100 owned assets plus the
+  empty page that ends the walk, and no package bytes at all.
+- Downloads only what is new, changed, or missing from the cache, into
+  `<library>/<publisher>/<asset>/<asset>.unitypackage`.
+- Records everything in `unity-sync.lock.json` beside the manifest: what is owned, at what
+  version, what is mirrored, and its checksum. Commit it for a changelog.
+- Reports assets your manifest lists that the account does not own, and assets the store
+  has delisted.
+
+## Verifying the cache
+
+Every run checks cached files cheaply: the file exists, its size is exactly what was
+recorded, and — for packages that carry a version stamp — that stamp still matches. That
+catches a truncated or replaced file without reading tens of gigabytes. A package with no
+stamp is checked on size alone, so it does not re-download on every run.
+
+`--verify` re-hashes instead, which is the only way to catch corruption in the middle of a
+file. It is opt-in for the obvious reason.
+
+## Cache
+
+The library is local and expendable: current versions are re-downloadable, and `sync`
+re-fetches anything missing or failing its check. Deleting the cache and re-syncing
+rebuilds it. Durability of the assets you actually ship belongs in the consuming project,
+not here.
+
+When an asset leaves your account, unity-sync does not delete anything: its entry drops
+out of the lockfile and the run tells you which file is now unreferenced, so you can
+decide. A run removes a file only when it is replacing that same asset's own copy: the
+superseded build after its path changed, whether the new copy was downloaded or adopted,
+and a copy that failed its check when a good copy of the same package is adopted over it.
+
+## Browsing what you have
+
+Searching and previewing the mirrored library lives in a separate tool,
+[quarry](https://github.com/curbol/quarry): it indexes an asset tree, reads inside
+archives, and previews models. The cache layout here is three levels deep
+(`publisher/asset/file`) precisely so quarry's vendor and pack filters work against it.
+Point its `root` at your library path.
+
+## Notes
+
+- Test fixtures are generated: `cmd/scrubfixtures` regenerates the PII-free
+  `testdata/store/` from git-excluded raw captures, and a guard test fails the build if
+  account data reaches them.
+- The tool is polite: bounded concurrent downloads (2 by default), backoff on rate limits,
+  and it identifies itself with a User-Agent.