Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 16 additions & 14 deletions .github/workflows/commit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
- uses: actions/checkout@v6
- uses: actions/setup-go@v6
with:
go-version: "1.26.1"
go-version: "1.26.4"

- run: go mod download
- run: go mod verify
Expand All @@ -29,31 +29,33 @@ jobs:
tests:
name: tests-${{ matrix.os }}
runs-on: ${{ matrix.os }}
# Check on all supported GitHub Actions OS
# https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories
# https://github.com/actions/runner-images
# https://github.com/actions/partner-runner-images
# One runner owns the strongest behavior gate. The remaining runners are
# portability sentinels for each supported OS family and Linux arm64.
strategy:
fail-fast: false
matrix:
os:
- ubuntu-22.04
- ubuntu-24.04
- ubuntu-22.04-arm
- ubuntu-24.04-arm
- windows-2022
- windows-2025
- macos-14
- macos-15
- macos-26
steps:
- uses: actions/checkout@v6
- uses: actions/setup-go@v6
with:
go-version: "1.26.1"
go-version: "1.26.4"

- run: go mod download
- run: go test -v ./... -coverprofile=./coverage.txt -covermode=atomic -coverpkg=./...

- name: Run behavior tests with race detection
if: matrix.os == 'ubuntu-24.04'
run: >-
go test -race -shuffle=on -vet=all -failfast ./...
-coverprofile=./coverage.txt -covermode=atomic -coverpkg=./...

- name: Run portability tests
if: matrix.os != 'ubuntu-24.04'
run: go test -shuffle=on -vet=all -failfast ./...

- name: Archive code coverage results
if: matrix.os == 'ubuntu-24.04'
Expand Down Expand Up @@ -82,7 +84,7 @@ jobs:

- uses: actions/setup-go@v6
with:
go-version: "1.26.1"
go-version: "1.26.4"

- name: Set up QEMU for cross-compilation of Docker images
uses: docker/setup-qemu-action@v3
Expand Down Expand Up @@ -204,7 +206,7 @@ jobs:
- uses: actions/checkout@v6
- uses: actions/setup-go@v6
with:
go-version: "1.26.1"
go-version: "1.26.4"

- run: go mod download

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:

- uses: actions/setup-go@v6
with:
go-version: "1.26.1"
go-version: "1.26.4"

- name: Set up QEMU for cross-compilation of Docker images
uses: docker/setup-qemu-action@v3
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/version-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,11 @@ jobs:
fi

echo "📦 Update available: $CURRENT_TIMESTAMP → $LATEST_TIMESTAMP"
echo "needs_update=true" >> $GITHUB_ENV
echo "latest_ts=$LATEST_TIMESTAMP" >> $GITHUB_ENV
echo "current_ts=$CURRENT_TIMESTAMP" >> $GITHUB_ENV
{
echo "needs_update=true"
echo "latest_ts=$LATEST_TIMESTAMP"
echo "current_ts=$CURRENT_TIMESTAMP"
} >> "$GITHUB_ENV"

# Store versions for issue body
echo "$API_JSON" | jq -r ".\"$LATEST_TIMESTAMP\"" > new_versions.json
Expand Down
147 changes: 108 additions & 39 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,122 @@
## Overview
# any-sync-bundle repository guide

- `any-sync-bundle` wraps the Anytype coordinator, consensus, filenode, and sync services into one binary (`lightnode/anynodes.go`).
- All services share the coordinator's network stack: TCP 33010, QUIC/UDP 33020, one PeerID, one DRPC mux.
- The filenode supports two storage backends (auto-selected based on configuration):
- **BadgerDB** (default): Local embedded storage via `lightcmp/lightfilenodestore`
- **S3** (optional): Cloud storage via upstream `s3store` implementation
- External dependencies: MongoDB for coordinator/consensus, Redis for filenode cache. Sync node persists to AnyStore on disk.
## Mission and ownership

Config bootstrap (cmd/start.go):
`any-sync-bundle` is a light process wrapper around the upstream Anytype
coordinator, consensus, filenode, and sync applications.

1. Load existing bundle YAML if present.
2. Otherwise create one via `config.CreateWrite`, injecting values from env/flags.
3. Always write the client config (`YamlClientConfig`) to the target path.
This repository owns orchestration, configuration conversion, shared-network
wiring, filenode store selection, and the local BadgerDB adapter. Preserve
upstream application behavior and lifecycle contracts. Do not copy or replace
an upstream service when a narrow adapter or upstream change is enough.

## Architecture Notes
MinIO is an S3-compatible integration dependency, not another storage backend.
Configured S3 storage uses upstream `s3store`; local storage uses BadgerDB.

- Coordinator starts first, then consensus, filenode, sync (`runBundleServices`).
- `extractSharedNetwork` copies network components from the coordinator into other apps.
- DRPC routes by method prefix (`/CoordinatorService`, `/ConsensusService`, `/FileService`, `/SpaceSyncService`).
- Data layout (default `./data`):
- `bundle-config.yml` – persisted configuration (credentials, keys)
- `client-config.yml` – generated client config (regenerated on start)
- `storage/` – local storage directory:
- `network-store/` – network configuration
- `storage-sync/` – sync node persistence (AnyStore)
- `storage-file/` – filenode data (BadgerDB, when not using S3)
## Repository map

## Development
- `main.go` owns process signals and the final shutdown watchdog.
- `cmd/` owns the CLI, bundle lifecycle, embedded-process supervision, and
MongoDB replica-set initialization.
- `config/` owns YAML boundaries and conversion to upstream node configs.
- `lightnode/anynodes.go` composes the four upstream applications, their shared
network components, and the filenode store.
- `lightcmp/lightfilenodestore/` implements the local BadgerDB store.
- `integration/` and `compose.*.yml` prove and document Docker-backed system
boundaries.
- `README.md` and `CONTRIBUTING.md` are the operator and developer workflow
sources; keep their commands aligned with CI.

### Compose files
## Runtime invariants

- `compose.dev.yml` – development dependencies (MongoDB replica set + Redis Stack).
- `compose.aio.yml` – bundle image with embedded MongoDB/Redis.
- `compose.external.yml` – bundle image plus external MongoDB and Redis containers.
- `compose.s3.yml` – bundle with MinIO for S3 storage testing.
- `compose.traefik.yml` – Traefik reverse proxy example.
- All services share the coordinator's PeerID, network stack, and DRPC mux on
TCP 33010 and QUIC/UDP 33020.
- Service order is coordinator, consensus, filenode, then sync. Initialize every
application before running any application so the network cannot read the
shared DRPC mux while handlers are still being registered. Shut down in
reverse order.
- The root context represents process lifetime. A process signal, startup
failure, service run failure, or unexpected embedded-process exit cancels it.
Preserve independent startup, runtime, and cleanup errors; an operator stop is
successful only when cleanup adds no failure.
- All-in-one mode owns MongoDB and Redis for their full lifetime. An unexpected
exit fails the bundle. Intentional shutdown sends SIGTERM to every child
before waiting, then forces and reaps deadline survivors.
- `cmd.ShutdownTimeout` is the application-owned aggregate shutdown bound.
Service, infrastructure, integration, watchdog, and bundle Compose timing must
derive from or be checked against that policy rather than duplicate it.

## Configuration and data

Configuration bootstrap has one order:

1. Load an existing bundle YAML when present.
2. Otherwise create it with `config.CreateWrite` from flags and environment.
3. Regenerate the client configuration on every start.

Validate persisted configuration at this boundary without rewriting
operator-owned values. Validate MongoDB URIs with the driver, not only
`net/url`. When creating a config, preserve the coordinator URI and add a path
separator only to the derived consensus URI when the bundle adds query options.

With default flags, durable paths are `./data/bundle-config.yml`,
`./data/storage/network-store/`, `./data/storage/storage-sync/`, and
`./data/storage/storage-file/`. The generated client config is
`./data/client-config.yml`. All-in-one infrastructure uses `/data/mongo` and
`/data/redis`. Treat the bundle config as sensitive because it contains
credentials and keys.

Complete MongoDB and Redis URI logging is an explicit current project decision.
Do not change it as incidental cleanup; revisit it only for an explicit
security or privacy requirement.

## Change discipline

- Read the active diff before editing. Preserve unrelated staged, unstaged, and
untracked work.
- Keep changes inside the repository-owned boundaries. Keep lifecycle order and
resource ownership visible instead of introducing a generic framework around
upstream applications.
- Test repository-owned contracts, not dependency internals. Prefer a few
boundary and failure-path tests over broad cross-products.
- Treat `go.mod` as the Go-version source of truth. When changing it, keep the
Dockerfile and GitHub workflows aligned and verify that Nix provides the same
toolchain.

## Verification

Use focused tests while iterating. Run `gofmt -w` on each changed Go file.
Before presenting a Go change as ready, run these commands from the repository
root:

```bash
go build -o any-sync-bundle .
golangci-lint run --fix
go test -race -shuffle=on -vet=all -failfast ./...
go test -tags=integration ./integration/... # requires Docker
golangci-lint run ./...
go test -count=1 -race -shuffle=on -vet=all -failfast ./...
go build -o /tmp/any-sync-bundle .
git diff --check
```

### Integration Tests
Do not use `golangci-lint --fix` as a verification command because it mutates
the reviewed source.

For configuration, startup, shutdown, storage, Docker, or integration changes,
also run the Docker-dependent integration suite:

```bash
go test -count=1 -tags=integration -timeout=10m ./integration/...
```

For Go toolchain, Nix, dependency, or release-build changes, also run:

```bash
nix flake check --print-build-logs
nix build -L .#default
```

Uses `testcontainers-go` to spin up MongoDB, Redis, and MinIO containers.
Documentation-only changes need `git diff --check` plus a direct review of every
changed command, path, link, and behavioral claim. Run broader gates when the
documentation changes an executable contract.

Test files:
- `integration/containers.go` – container lifecycle helpers
- `integration/bundle.go` – bundle process manager
- `integration/integration_test.go` – test cases
A change is ready only when the narrow regression proof and every applicable
broader gate pass from the final source state. Report the commands actually run
and any verification that could not be completed.
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Thanks for your interest in improving any-sync-bundle! This document explains ho

### Prerequisites

- Go 1.26.1 or later
- Go 1.26.4 or later
- Docker (optional, for testing with containers)
- golangci-lint (for linting)

Expand Down Expand Up @@ -82,7 +82,7 @@ release-facing files.

```sh
# Set variables (fish shell)
set VERSION v1.4.3
set VERSION v1.5.0
set ANYTYPE_UNIX_TIMESTAMP <current timestamp from https://puppetdoc.anytype.io/api/v1/prod-any-sync-compatible-versions/>
# The compatibility date suffix is always derived in UTC.
set ANYTYPE_FORMATTED (env TZ=UTC date -r $ANYTYPE_UNIX_TIMESTAMP +'%Y-%m-%d')
Expand All @@ -97,7 +97,7 @@ git push origin tag $FINAL_VERSION

`v[bundle-version]-[anytype-compatibility-date]`

- `v1.4.3` – Bundle's semantic version (SemVer)
- `v1.5.0` – Bundle's semantic version (SemVer)
- `YYYY-MM-DD` – Date derived in UTC from the current Anytype compatibility timestamp:
https://puppetdoc.anytype.io/api/v1/prod-any-sync-compatible-versions/

Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
#
# Stage: Initial bin build
#
FROM --platform=$BUILDPLATFORM golang:1.26.1-alpine AS stage-bin
FROM --platform=$BUILDPLATFORM golang:1.26.4-alpine AS stage-bin
WORKDIR /app

# Use mount cache for dependencies
Expand Down
Loading
Loading