RustFS replaces MinIO; snapshot.sh finds the database the services actually use

.github/workflows/ci.yml

@@ -175,34 +175,35 @@ jobs:
       - name: Build
         run: pnpm run build
 
-      # Started as a step rather than as a `services:` container: the official
-      # image needs `server /data` as its command, and service containers can
-      # only supply an image plus options, never arguments. `bitnami/minio`,
-      # which does start a server unattended, no longer publishes public tags.
+      # Started as a step rather than as a `services:` container: the image takes
+      # its data directory as a command argument, and service containers can only
+      # supply an image plus options, never arguments.
+      #
+      # RustFS, not MinIO. MinIO stopped publishing images: Docker Hub first
+      # (2026-09-16), then Quay (2026-09-25, `unauthorized` to an anonymous pull),
+      # and its binary archive answers 410. RustFS 1.0.0 passes the whole
+      # object-storage suite; SeaweedFS and Garage failed the signed browser
+      # upload and LocalStack served a private object without a signature.
       #
       # Pinned to a release rather than `latest` for the same reason Node and
       # ClickHouse are pinned — a floating tag makes CI results depend on when
       # they ran (D-203).
-      # Quay, not Docker Hub: minio/minio stopped resolving there on 2026-09-16.
-      - name: Start MinIO
+      - name: Start RustFS
         run: |
-          docker run -d --name minio -p 9000:9000 \
-            -e MINIO_ROOT_USER=minioadmin \
-            -e MINIO_ROOT_PASSWORD=minioadmin \
-            quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z server /data
+          docker run -d --name rustfs -p 9000:9000             -e RUSTFS_ACCESS_KEY=rustfsadmin             -e RUSTFS_SECRET_KEY=rustfsadmin             rustfs/rustfs:1.0.0 /data
 
-      - name: Wait for MinIO
+      - name: Wait for RustFS
         run: |
           for attempt in $(seq 1 60); do
-            if curl -fsS http://localhost:9000/minio/health/live > /dev/null; then
-              echo "minio ready after ${attempt} attempt(s)"
+            if curl -fsS http://localhost:9000/health > /dev/null; then
+              echo "rustfs ready after ${attempt} attempt(s)"
               exit 0
             fi
             sleep 2
           done
-          echo "minio did not become live within 120s; last attempt:"
-          curl -vsS http://localhost:9000/minio/health/live || true
-          docker logs minio || true
+          echo "rustfs did not become live within 120s; last attempt:"
+          curl -vsS http://localhost:9000/health || true
+          docker logs rustfs || true
           exit 1
 
       # `noeviction` is D-205's requirement for the queue instance and the one
@@ -228,9 +229,9 @@ jobs:
           exit 1
 
       - name: Integration tests
-        # MinIO stands in for the object storage provider G-001 has not chosen.
-        # The suites needing Fly infrastructure skip themselves; the MinIO and
-        # Valkey ones run for real.
+        # RustFS stands in for the S3-compatible provider (Hetzner Object Storage
+        # in production). The suites needing Fly infrastructure skip themselves;
+        # the object-storage and Valkey ones run for real.
         #
         # `redis://` rather than `rediss://`: the connection factory requires TLS
         # and AUTH for the collector hop because it crosses the public internet
@@ -239,8 +240,8 @@ jobs:
         # instance.
         env:
           TEST_S3_ENDPOINT: http://localhost:9000
-          TEST_S3_ACCESS_KEY: minioadmin
-          TEST_S3_SECRET_KEY: minioadmin
+          TEST_S3_ACCESS_KEY: rustfsadmin
+          TEST_S3_SECRET_KEY: rustfsadmin
           TEST_VALKEY_URL: redis://localhost:6379
           TEST_POSTGRES_URL: postgres://openanalytics:openanalytics@localhost:5432/openanalytics_test
           TEST_CLICKHOUSE_URL: http://localhost:8123

CHANGELOG.md

@@ -8,6 +8,33 @@ taking.
 Releases before 0.6.0 have their notes on the
 [GitHub releases page](https://github.com/OpenLabs-so/openanalytics/releases).
 
+## [Unreleased]
+
+**Upgrade notes: only if you enabled the `object-storage` profile.** Create
+`env/rustfs.env` from `env/rustfs.env.example`, start the profile, and create
+the bucket and its CORS rule as `SELF-HOSTING.md`, "Object storage", shows.
+Everyone else has nothing to do.
+
+### Added
+
+- **Postgres on Neon.** `infra/selfhost/NEON.md` walks a new install, and
+  moving an existing one, onto [Neon](https://neon.com);
+  `docker-compose.neon.yml` takes the bundled `postgres` service out of the
+  stack. `snapshot.sh` and `rollback.sh` now handle a Postgres that is not on
+  the host: the snapshot holds ClickHouse only and records the instant the
+  stack stopped, and a restore stops with the stack down so Postgres can be
+  restored to that instant first.
+
+### Changed
+
+- **The optional object storage is RustFS, not MinIO.** MinIO stopped
+  publishing images — Docker Hub on 2026-09-16, then Quay on 2026-09-25 —
+  so `docker compose --profile object-storage up` could no longer pull it.
+  RustFS (`rustfs/rustfs:1.0.0`) passes the same object-storage suite CI runs.
+  Unlike MinIO it does not open CORS to every origin, which is why the setup
+  now includes a bucket CORS rule. The old `minio-data` volume is left in
+  place and nothing reads it.
+
 ## [0.8.0] - 2026-09-19
 
 **Upgrade notes: nothing by hand — and going back is a restore.**

SELF-HOSTING.md

@@ -594,18 +594,44 @@ deletions, and drop the assertion.
 
 ### Object storage
 
-Data import and export need an S3-compatible bucket. Any provider works; MinIO
-ships here so you do not need one:
+Data import and export need an S3-compatible bucket. Any provider works;
+[RustFS](https://rustfs.com) ships here so you do not need one:
 
 ```sh
 docker compose --profile object-storage up -d
 ```
 
-Create a bucket and credentials in the MinIO console, then fill in the five
-`OBJECT_STORAGE_*` variables in **both** `env/api.env` and `env/worker.env` — the
-api mints signed URLs and the worker moves the bytes. All five or none: a
-partial block is treated as "not configured" and the import surface is simply
-not mounted.
+Then, once, create the bucket and let the dashboard upload into it. The browser
+PUTs an import archive straight to the bucket, so the bucket needs a CORS rule
+naming your dashboard's origin — without it the upload fails in the browser and
+nowhere else:
+
+```sh
+S3_KEY="$(grep ^RUSTFS_SECRET_KEY env/rustfs.env | cut -d= -f2)"
+aws_cli() {
+  docker run --rm --network openanalytics_oa -v "$PWD:/w" -w /w     -e AWS_ACCESS_KEY_ID=openanalytics -e AWS_SECRET_ACCESS_KEY="$S3_KEY"     -e AWS_DEFAULT_REGION=us-east-1 amazon/aws-cli --endpoint-url http://rustfs:9000 "$@"
+}
+aws_cli s3 mb s3://openanalytics
+cat > cors.json <<'JSON'
+{"CORSRules":[{"AllowedOrigins":["https://app.example.com"],"AllowedMethods":["PUT","GET"],"AllowedHeaders":["*"],"ExposeHeaders":["ETag"],"MaxAgeSeconds":600}]}
+JSON
+aws_cli s3api put-bucket-cors --bucket openanalytics --cors-configuration file://cors.json
+```
+
+Replace `app.example.com` with your dashboard's name. Then fill in the five
+`OBJECT_STORAGE_*` variables in **both** `env/api.env` and `env/worker.env` —
+the api mints signed URLs and the worker moves the bytes. The access key is
+`openanalytics` and the secret is `RUSTFS_SECRET_KEY` from `env/rustfs.env`. All
+five or none: a partial block is treated as "not configured" and the import
+surface is simply not mounted.
+
+**Upgrading from a release that shipped MinIO.** MinIO stopped publishing
+images, so the `minio` service is gone. Only installs that enabled the
+`object-storage` profile are affected: create `env/rustfs.env` from
+`env/rustfs.env.example` with a secret of your own, start the profile, and run
+the two commands above. The old `minio-data` volume is left where it was and
+nothing reads it; import archives and exports are transient by design, so
+nothing durable lived there.
 
 ---
 

infra/selfhost/docker-compose.yml

@@ -73,7 +73,7 @@
 #
 # ---------------------------------------------------------------------------
 # Optional pieces, off by default, enabled with `--profile`:
-#   --profile object-storage   MinIO, for data import/export and CSV exports
+#   --profile object-storage   RustFS, for data import/export and CSV exports
 #
 # The tracker (`oa.js`) is built by a one-shot container into a shared volume
 # and served by Caddy. See the `tracker-build` service and the Caddyfile.
@@ -548,20 +548,22 @@ services:
   # S3-compatible object storage, for data import/export. Any S3-compatible
   # provider works; enable this only if you do not already have one.
   #   docker compose --profile object-storage up -d
-  # Pulled from Quay: MinIO's Docker Hub repository stopped resolving on
-  # 2026-09-16 ("pull access denied"); the same tag and digest live on Quay.
-  minio:
+  # RustFS rather than MinIO: MinIO stopped publishing images (Docker Hub on
+  # 2026-09-16, Quay on 2026-09-25). RustFS passes the same object-storage
+  # suite CI runs. The bucket and its CORS rule are one-time setup —
+  # SELF-HOSTING.md, "Object storage".
+  rustfs:
     logging: *logging
-    image: quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z
-    container_name: oa-minio
+    image: rustfs/rustfs:1.0.0
+    container_name: oa-rustfs
     restart: unless-stopped
     profiles: ['object-storage']
-    command: ['server', '/data', '--console-address', ':9001']
-    env_file: env/minio.env
+    command: ['/data']
+    env_file: env/rustfs.env
     volumes:
-      - minio-data:/data
+      - rustfs-data:/data
     healthcheck:
-      test: ['CMD', 'mc', 'ready', 'local']
+      test: ['CMD', 'curl', '-fsS', 'http://127.0.0.1:9000/health']
       interval: 15s
       timeout: 5s
       retries: 10
@@ -584,4 +586,4 @@ volumes:
   tracker:
   caddy-data:
   caddy-config:
-  minio-data:
+  rustfs-data:

infra/selfhost/env/api.env.example

@@ -140,7 +140,7 @@ AUTH_PASSWORD_SIGNIN=enabled
 # All five or none: a signature needs the endpoint, region, bucket and key pair,
 # so a partial block is treated as "not configured" and the import surface is
 # simply not mounted.
-# OBJECT_STORAGE_ENDPOINT=http://minio:9000
+# OBJECT_STORAGE_ENDPOINT=http://rustfs:9000
 # OBJECT_STORAGE_REGION=us-east-1
 # OBJECT_STORAGE_BUCKET=openanalytics
 # OBJECT_STORAGE_ACCESS_KEY_ID=

infra/selfhost/env/rustfs.env.example

@@ -0,0 +1,10 @@
+# RustFS — optional S3-compatible object storage, for data import and export.
+# Enabled only with `docker compose --profile object-storage up -d`.
+#
+# Any S3-compatible provider works; this is here so a self-hosted install does
+# not need one. After the first start, create the bucket and its CORS rule
+# (SELF-HOSTING.md, "Object storage"), then fill in the five OBJECT_STORAGE_*
+# variables in api.env and worker.env (all five or none — a partial block is
+# treated as "not configured"). This pair IS the access key the services use.
+RUSTFS_ACCESS_KEY=openanalytics
+RUSTFS_SECRET_KEY={{RUSTFS_SECRET_KEY}}

infra/selfhost/env/worker.env.example

@@ -107,7 +107,7 @@ EMAIL_FROM=Open Analytics <analytics@analytics.example>
 # STRIPE_SECRET_KEY=
 
 # Object storage for data import and export. All five or none.
-# OBJECT_STORAGE_ENDPOINT=http://minio:9000
+# OBJECT_STORAGE_ENDPOINT=http://rustfs:9000
 # OBJECT_STORAGE_REGION=us-east-1
 # OBJECT_STORAGE_BUCKET=openanalytics
 # OBJECT_STORAGE_ACCESS_KEY_ID=

infra/selfhost/generate-secrets.sh

@@ -141,7 +141,7 @@ export OA_SUB_CLICKHOUSE_MAINTENANCE_PASSWORD="$(hex32)"
 export OA_SUB_CLICKHOUSE_MIGRATION_PASSWORD="$(hex32)"
 export OA_SUB_VALKEY_QUEUE_PASSWORD="$(hex32)"
 export OA_SUB_VALKEY_REALTIME_PASSWORD="$(hex32)"
-export OA_SUB_MINIO_ROOT_PASSWORD="$(hex32)"
+export OA_SUB_RUSTFS_SECRET_KEY="$(hex32)"
 
 # Four independent secrets, none derived from another. They protect different
 # things, and a derivation would make rotating one force rotating the other.

infra/selfhost/snapshot.sh

@@ -124,11 +124,45 @@ volume_at() {
 	echo "$name"
 }
 
-# True when this compose project runs its own Postgres. Asked of the resolved
-# configuration, so an override that moves `postgres` behind a profile — which
-# is what `docker-compose.neon.yml` does — counts as not running it.
+# True when this compose project defines a `postgres` service. Asked of the
+# resolved configuration, so an override that moves `postgres` behind a profile
+# — which is what `docker-compose.neon.yml` does — counts as not defining it.
+#
+# The list is captured before it is searched, never piped into `grep -q`: under
+# `pipefail`, grep exiting at its first match can SIGPIPE compose while it is
+# still writing, and the pipeline then reads as "no postgres" — which would
+# silently leave a bundled database out of the snapshot. For the same reason a
+# configuration compose cannot resolve is an error here, not an answer.
+postgres_service_defined() {
+	local services
+	services="$(compose config --services)" || die "docker compose could not resolve this stack's configuration, so it cannot tell whether Postgres is part of it"
+	grep -qx postgres <<<"$services"
+}
+
+# The host the api's DATABASE_URL names, or empty when env/api.env gives it
+# through DATABASE_URL_FILE instead — that path is inside the container, so the
+# host cannot read it. Host only: the credentials never leave this function.
+database_host() {
+	local url
+	url="$(grep -E '^DATABASE_URL=' env/api.env 2>/dev/null | tail -1 || true)"
+	url="${url#DATABASE_URL=}"
+	[ -n "$url" ] || return 0
+	url="${url#*://}"
+	url="${url##*@}"
+	echo "${url%%[:/?]*}"
+}
+
+# True when the database the services actually use is the bundled one: the
+# service is defined AND the api's URL points at it. A `postgres` container that
+# is still defined while DATABASE_URL names another host — NEON.md's step 5
+# skipped, or any other managed Postgres — holds nothing the services write, so
+# archiving it would be a snapshot of the wrong database. When the URL cannot be
+# read (DATABASE_URL_FILE), the service alone decides, as it did before.
 postgres_in_stack() {
-	compose config --services 2>/dev/null | grep -qx postgres
+	postgres_service_defined || return 1
+	local host
+	host="$(database_host)"
+	[ -z "$host" ] || [ "$host" = postgres ]
 }
 
 # --- helpers that run inside the helper image -------------------------------
@@ -211,6 +245,10 @@ do_create() {
 	else
 		pg_mode=external
 		pg_volume=external
+		if postgres_service_defined; then
+			echo "snapshot: note — a postgres service is defined, but env/api.env points DATABASE_URL at"
+			echo "          $(database_host). That is the database in use; the local container is not archived."
+		fi
 	fi
 	ch_volume="$(volume_at clickhouse /var/lib/clickhouse)"
 
@@ -328,7 +366,7 @@ do_restore() {
 	[ "$pg_mode" = external ] || [ -f "$from/pg-data.tar.gz" ] || die "$from/pg-data.tar.gz is missing"
 	[ -f "$from/ch-data.tar.gz" ] || die "$from/ch-data.tar.gz is missing"
 	if [ "$pg_mode" = volume ] && ! postgres_in_stack; then
-		die "$from holds a Postgres volume, but this stack runs no postgres service — restore it into a stack that does, or restore the database by hand"
+		die "$from holds a Postgres volume, but this stack does not use a bundled postgres (no such service, or DATABASE_URL names another host) — restore it into a stack that does, or restore the database by hand"
 	fi
 
 	local pg_volume="" ch_volume