____                  ___                       ____
/\  _`\    __         /\_ \                     /\  _`\
\ \ \L\ \ /\_\    ___ \//\ \     ___      __    \ \,\L\_\     __   _ __   __  __     __   _ __
 \ \  _ <'\/\ \ /' _ `\ \ \ \   / __`\  /'_ `\   \/_\__ \   /'__`\/\`'__\/\ \/\ \  /'__`\/\`'__\
  \ \ \L\ \\ \ \/\ \/\ \ \_\ \_/\ \L\ \/\ \L\ \    /\ \L\ \/\  __/\ \ \/ \ \ \_/ |/\  __/\ \ \/
   \ \____/ \ \_\ \_\ \_\/\____\ \____/\ \____ \   \ `\____\ \____\\ \_\  \ \___/ \ \____\\ \_\
    \/___/   \/_/\/_/\/_/\/____/\/___/  \/___L\ \   \/_____/\/____/ \/_/   \/__/   \/____/ \/_/
                                          /\____/
                                          \_/__/
    

BinlogServer

Binlog Server is a service for pulling MySQL binlog into durable local files, persisting checkpoint state, and managing replication tasks through an API-driven control plane with UI, optional S3-compatible upload, and cluster coordination support.

If this is your first time visiting the repository, start with Quick Start. If you are evaluating whether the project fits your use case, read the positioning section below first.

What Problem Does It Solve?

It turns binlog pulling, local persistence, checkpoint tracking, and task lifecycle management into an operational service instead of a collection of scripts, cron jobs, and ad hoc metadata.

Screenshots

Console overview

Task detail

Swagger API

Good fit for:

• Teams that want binlog pulling, local durability, and task state management as a service
• Environments that need LATEST, FILE_POS, or GTID start modes
• Workflows that want local persistence with optional S3-compatible upload
• Operators who want API, UI, and observability instead of a one-off script

Probably not a fit for:

• One-time exports instead of continuous replication
• Very small single-node scripts without task orchestration needs
• Teams looking for a full CDC platform replacement

Why Use It?

• Clear checkpoint semantics: checkpoint advances only after local fsync succeeds
• Supports LATEST, FILE_POS, and GTID start modes
• Built-in API, UI, Swagger, metrics, and optional tracing
• Supports metadata-backed coordination, lease management, and S3-compatible upload
• Includes repository E2E scenarios to validate core regression paths

Install / Download

For tagged public releases, download the archive that matches your platform from GitHub Releases:

• binlog-server_<version>_darwin_amd64.tar.gz
• binlog-server_<version>_darwin_arm64.tar.gz
• binlog-server_<version>_linux_amd64.tar.gz
• binlog-server_<version>_linux_arm64.tar.gz

The real asset name is binlog-server_<ver>_<os>_<arch>.tar.gz (for example binlog-server_0.5.5_linux_amd64.tar.gz). The release page also provides checksums.txt; verify it before extracting. The tarball contains a versioned subdirectory:

binlog-server_0.5.5_linux_amd64/
  binlog-server
  migrate
  migrations/
  README.md
  README_ZH.md
  CHANGELOG.md
  LICENSE
  config.example.yaml
  config.production.example.yaml

UI assets are embedded into the binary, so you do not need a separate frontend package.

Quick Start

This is the shortest path for release operators. You do not need Go installed.

Prerequisites

• A release tarball for your platform from GitHub Releases
• A reachable MySQL or MariaDB instance with log_bin enabled

Metadata isolation is mandatory: when meta_dsn is configured, its MySQL instance must be separate from every replication source and must never be added to the backup task set. The server rejects an exact matching TCP host:port and treats localhost plus explicit loopback literals (127/8 and ::1, including bracketed IPv6) as the same endpoint class; other aliases and proxies still require operator isolation.

1. Download, verify, extract, run

VER=0.5.5
OS=linux          # linux | darwin
ARCH=amd64        # amd64 | arm64

curl -fsSL -O "https://github.com/Fanduzi/BinlogServer/releases/download/v${VER}/binlog-server_${VER}_${OS}_${ARCH}.tar.gz"
curl -fsSL -O "https://github.com/Fanduzi/BinlogServer/releases/download/v${VER}/checksums.txt"
sha256sum -c checksums.txt --ignore-missing

tar -xzf "binlog-server_${VER}_${OS}_${ARCH}.tar.gz"
cd "binlog-server_${VER}_${OS}_${ARCH}"

export BINLOG_SERVER_LISTEN_ADDR=127.0.0.1:8080
export BINLOG_SERVER_DATA_DIR=./data
./binlog-server

The default listen address is :8080 if you omit BINLOG_SERVER_LISTEN_ADDR. Non-loopback binds require API auth; the example above uses loopback so a local demo can start without a token.

2. Verify /healthz

curl -fsS http://127.0.0.1:8080/healthz

Expected response:

ok

3. Create your first task

Replace the MySQL connection details with your own instance:

curl -fsS -X POST http://127.0.0.1:8080/api/tasks \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "quickstart-task",
    "cluster_key": "quickstart-task",
    "source": {
      "host": "127.0.0.1",
      "port": 3306,
      "user": "repl",
      "password": "secret",
      "flavor": "mysql"
    },
    "start": {
      "mode": "LATEST"
    },
    "storage": {
      "retention_days": 7
    }
  }'

4. Start the task

Replace <task-id> with the id returned from the previous step:

curl -i -X POST http://127.0.0.1:8080/api/tasks/<task-id>/start

5. Check task state

curl -fsS http://127.0.0.1:8080/api/tasks

6. Open the UI or Swagger

• UI: http://127.0.0.1:8080/ui/
• Swagger: http://127.0.0.1:8080/swagger/index.html

For more operational and usage guidance, continue from docs/guide/README.md.

What You Should See

• /healthz returns ok
• /api/tasks lists the task you just created
• /ui/ opens the management interface
• /swagger/index.html lets you inspect and try the API
• After the task starts, checkpoint and metrics begin reflecting runtime state

⚠️ Security Warning

By default, API authentication is DISABLED only for loopback demo binds (127.0.0.1/localhost/::1). Non-loopback listen_addr (including the default :8080 and 0.0.0.0:8080) fail-closes at startup unless api.auth.enabled, protect_api, and protect_metrics are all true. PRODUCTION=true still requires the same independently.

For production deployments, you MUST:

1. Set api.auth.enabled: true (or BINLOG_SERVER_API_AUTH_ENABLED=true)
2. Configure your authentication method (Bearer Token or API Key)
3. Protect /api/* and /metrics (these default true when auth is enabled and the flags are unset)

See SECURITY.md and docs/security.md for security guidance.

Minimal Production Notes

Development defaults are intentionally permissive. Do not carry them unchanged into production.

• Auth: disabled by default only on loopback; non-loopback listen and PRODUCTION=true require enabled auth plus protected /api/* and /metrics
• Meta DB: if meta_dsn is configured, use an independent MySQL instance that is never a replication source, and run migrations first; the service does not auto-create or auto-upgrade schema
• Standalone without meta_dsn keeps task metadata, checkpoints, and file records in memory only. kill -9 / process restart drops the control plane. Binlog bytes already written under {data_dir}/{task_id}/ remain as orphan files. With meta_dsn, persisted active tasks resume automatically from their checkpoint after restart, and GET /files includes the current OPEN segment. storage.dir is ignored; files always go to {data_dir}/{task_id}/.
• Upload: S3-compatible upload is optional, but required fields must be complete when enabled
• Tracing: disabled by default; validate exporter configuration and sampling before rollout

Minimum production baseline:

export BINLOG_SERVER_API_AUTH_ENABLED=true
export BINLOG_SERVER_API_AUTH_MODE=bearer
export BINLOG_SERVER_API_AUTH_BEARER_TOKEN="$(openssl rand -hex 32)"
export BINLOG_SERVER_API_AUTH_PROTECT_API=true
export BINLOG_SERVER_API_AUTH_PROTECT_METRICS=true

Start from config.production.example.yaml; it contains no credential, needs only the protected bearer-token environment variable, and refuses to load until that placeholder is resolved.

If you use the metadata database:

export META_DSN='meta:replace_me@tcp(127.0.0.1:3306)/binlog_meta?parseTime=true'
./migrate up --dsn "$META_DSN" --path ./migrations
export BINLOG_SERVER_META_DSN="$META_DSN"

If you enable upload, provide at least:

• BINLOG_SERVER_UPLOAD_ENDPOINT
• BINLOG_SERVER_UPLOAD_BUCKET
• BINLOG_SERVER_UPLOAD_ACCESS_KEY
• BINLOG_SERVER_UPLOAD_SECRET_KEY

FAQ / Common Pitfalls

make e2e-quick fails locally

• Make sure Docker Desktop or another Docker daemon is running first.
• E2E scenarios start MySQL / Percona containers and depend on local Docker availability.
• See scripts/e2e/README.md for more details.

Tasks fail after configuring meta_dsn

• A common cause is that the metadata schema has not been migrated yet.
• Run make migrate-up META_DSN=... before starting the service.
• Migration command details are documented in cmd/migrate/README.md.

Auth was left disabled in production

• This is the most important development default to override.
• Loopback binds (127.0.0.1/localhost/::1) may stay unauthenticated; non-loopback listen and PRODUCTION=true require auth.
• See SECURITY.md and docs/security.md for concrete guidance.

Upload is configured but files do not upload

• endpoint, bucket, access_key, and secret_key must all be present.
• region and prefix are optional and not part of initialization minimums.
• The current implementation targets S3-compatible APIs.

/metrics or tracing looks empty

• /metrics exposes core metric families even before tasks start; some values may still be placeholders.
• Tracing is off by default, so the absence of spans is often expected rather than a fault.

Upgrade / Release Entry

Review CHANGELOG.md before upgrading.

Pay particular attention to these change types:

• schema / migration changes
• new, deprecated, or default-shifted config keys
• sqlc workflow changes
• observability contract changes that affect dashboards or alerts

This repository does not apply migrations or migrate configuration for you, so upgrades should be handled as an operator change, not just a binary swap.

Architecture

BinlogServer is organized as a control-plane-oriented service with clear boundaries between HTTP/API handling, task orchestration, replication execution, metadata persistence, upload integration, and UI delivery.

Modules

Module	Responsibility	Entry
cmd	Top-level service startup and migration commands	cmd/README.md
internal/api	HTTP routes, request validation, Swagger, metrics, tracing hooks	internal/api/README.md
internal/app	Runtime assembly and role lifecycle orchestration	internal/app/README.md
internal/binlog	Local binlog file writing and checkpoint persistence helpers	internal/binlog/README.md
internal/config	YAML and environment-based configuration loading	internal/config/README.md
internal/logging	Logger setup and log output rotation	internal/logging/README.md
internal/meta	Metadata storage, schema checks, lease-backed coordination data	internal/meta/README.md
internal/replication	MySQL replication pull loop and durable local write path	internal/replication/README.md
internal/tasks	Task state machine, scheduling, and execution orchestration	internal/tasks/README.md
internal/ui	Embedded UI asset serving	internal/ui/README.md
internal/upload	S3-compatible upload integration	internal/upload/README.md
scripts	Local build helpers, release asset packaging, and E2E entrypoints	scripts/README.md
frontend	Frontend source and build pipeline for the embedded UI	frontend/README.md

Repository Map

Once Quick Start works, continue from these entry points:

Topic	Entry
Usage and operations guide	docs/guide/README.md
Security policy	SECURITY.md
Version history	CHANGELOG.md
Service startup command	cmd/binlog-server/README.md
Database migration	cmd/migrate/README.md
API module	internal/api/README.md
Replication pipeline	internal/replication/README.md
Upload module	internal/upload/README.md
E2E suite	scripts/e2e/README.md

Development

Source build is the path when you are changing the code, not installing a release.

Prerequisites: Go 1.26.7+. Docker is needed only for E2E.

make build

# Linux deployment binaries should keep CGO disabled to avoid host glibc coupling.
make build-linux

# Run from source
go run ./cmd/binlog-server
BINLOG_SERVER_LISTEN_ADDR=127.0.0.1:18080 go run ./cmd/binlog-server

# Prepare a local set of release assets
make release-assets VERSION=v0.5.5

Development Validation

Common verification commands:

go test ./...
go vet ./...
make e2e-quick

If you want frontend-only development:

cd frontend
npm install
npm run dev