ceph-volsync-plugin

A VolSync external mover plugin for Ceph storage, enabling efficient snapshot-based volume replication for CephFS and RBD volumes.

Overview

ceph-volsync-plugin extends VolSync with a Ceph-native data mover. Instead of copying entire volumes or relying on generic file-level rsync, it uses Ceph's snapshot diff APIs to identify and transfer only the changed data between replications.

It registers as an external mover with VolSync and handles ReplicationSource and ReplicationDestination custom resources for PVCs backed by supported Ceph CSI drivers.

Supported CSI providers:

Provider	Mover
`cephfs.csi.ceph.com`	CephFS (snapshot diff + rsync fallback)
`nfs.csi.ceph.com`	CephFS
`rbd.csi.ceph.com`	RBD (block diff)
`nvmeof.csi.ceph.com`	RBD

Features

Incremental sync — uses CephFS snapshot diff and RBD block diff APIs to transfer only changed data
Concurrent pipeline — parallel gRPC data pipeline with configurable concurrency
Direct TLS — data transfer uses ephemeral mTLS certificates exchanged at startup; stunnel handles only the initial handshake and rsync tunneling
Zero-block optimization — zeroed and unchanged ranges are detected early and never transferred
CephFS + RBD — CephFS routes non-empty regular files through the block pipeline and uses rsync for special files, zero-length files, and metadata; RBD operates directly on raw block devices
OpenShift ready — automatically creates the required SecurityContextConstraints on OpenShift clusters

Architecture

  Source Cluster                         Destination Cluster
  ------------------------------         --------------------------------
  ReplicationSource CR                   ReplicationDestination CR
        |                                        |
  ReplicationSource                       ReplicationDestination
    Controller                               Controller
        |                                        |
  Ceph Mover Builder                      Ceph Mover Builder
        |                                        |
  K8s Job                                 K8s Job
  (Source Worker)                         (Destination Worker)
        |                                        |
        |<-- stunnel (handshake) + direct TLS -->|

Data flow:

Controllers watch ReplicationSource/ReplicationDestination CRs
The Ceph mover builder detects the CSI provider from spec.external.provider and creates a Mover
The mover orchestrates Kubernetes Jobs, PVCs, Services, and Secrets
Source worker reads snapshot diffs and streams changed blocks via gRPC
Destination worker receives and writes blocks, then commits

CephFS path: SnapshotDiffer categorises changes into non-empty regular files (block diff pipeline), special/zero-length files (rsync), deletions (gRPC Delete), and directory metadata (convergence rsync) -- all run concurrently.

RBD path: RBD diff iterator yields changed block ranges; these flow through a concurrent pipeline to the destination block device. Supports both full (no prior snapshot) and incremental (snapshot-to-snapshot) modes.

Prerequisites

Kubernetes ≥ 1.28
VolSync installed in the cluster
Ceph cluster with ceph-csi drivers deployed
VolumeSnapshot CRDs and external snapshot controller installed
A StorageClass and VolumeSnapshotClass referencing a supported Ceph CSI driver

Quick Start

Install the operator

# Install CRDs (VolSync CRDs must already be present)
make install

# Deploy the operator using the published image
make deploy IMG=quay.io/ramendr/ceph-volsync-plugin-operator:latest \
  MOVER_IMG=quay.io/ramendr/ceph-volsync-plugin-mover:latest

Configure replication

Ceph credentials are automatically obtained from the CSI provisioner secret referenced in the ceph-csi-config ConfigMap — no manual secret creation required.

Create a ReplicationSource on the source cluster:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: my-source
  namespace: my-namespace
spec:
  sourcePVC: my-pvc
  trigger:
    schedule: "*/30 * * * *"  # every 30 minutes
  external:
    provider: cephfs.csi.ceph.com
    parameters:
      storageClassName: cephfs-sc
      volumeSnapshotClassName: cephfs-snapclass
      copyMethod: Snapshot
      address: <copy from destination status.rsyncTLS.address>

Create a ReplicationDestination on the destination cluster:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
  name: my-dest
  namespace: my-namespace
spec:
  trigger:
    manual: first-sync
  external:
    provider: cephfs.csi.ceph.com
    parameters:
      storageClassName: cephfs-sc
      volumeSnapshotClassName: cephfs-snapclass
      copyMethod: Snapshot
      destinationPVC: my-dest-pvc

See docs/user-guide.md for a complete walkthrough including RBD volumes and incremental sync.

Configuration

CR external parameters

Set these in spec.external.parameters on both ReplicationSource and ReplicationDestination.

Parameter	Source	Dest	Description
`storageClassName`	✓	✓	StorageClass for the snapshot PVC
`volumeSnapshotClassName`	✓	✓	VolumeSnapshotClass for creating snapshots
`copyMethod`	✓	✓	`Snapshot` (recommended) or `Direct`; defaults to `Direct` if omitted
`address`	✓		Destination service address (hostname or IP)
`keySecret`	✓	✓	Stunnel PSK Secret (`psk.txt`). Destination auto-generates one; source must reference it via `status.rsyncTLS.keySecret`. For cross-cluster, copy the Secret data to the source namespace.
`destinationPVC`		✓ (required)	Pre-existing destination PVC name
`volumeName`	✓		PVC name to resolve CSI volume handle (source + Direct only)
`baseSnapshotName`	✓		Base VolumeSnapshot name for incremental diff (source + Direct only)
`targetSnapshotName`	✓		Target VolumeSnapshot name (source + Direct only)

volumeName, baseSnapshotName, and targetSnapshotName are used only on the source with copyMethod: Direct. Snapshot mode tracks incremental state via snapshot status labels.

Operator environment variables

Variable	Default	Description
`MOVER_IMAGE`	`quay.io/ramendr/ceph-volsync-plugin-mover:latest`	Override the mover container image
`CEPH_CSI_CONFIG_NAME`	`ceph-csi-config`	ConfigMap name for Ceph CSI cluster config
`CEPH_CSI_CONFIG_NAMESPACE`	`rook-ceph`	Namespace of the Ceph CSI config ConfigMap

See docs/configuration.md for the full reference including mover environment variables and manager CLI flags.

Building

Protobuf generation is required before the first build:

make proto-generate        # generate gRPC/protobuf code (required first)
make generate              # generate deepcopy methods
make manifests             # generate CRDs and RBAC
make build                 # build manager binary

Container images:

make docker-build IMG=<registry>/ceph-volsync-plugin-operator:<tag>
make docker-build-mover MOVER_IMG=<registry>/ceph-volsync-plugin-mover:<tag>

# Multi-arch builds
make docker-buildx IMG=<registry>/ceph-volsync-plugin-operator:<tag>
make docker-buildx-mover MOVER_IMG=<registry>/ceph-volsync-plugin-mover:<tag>

Pinned versions: Go 1.25.7 · Ceph tentacle (quay.io/ceph/ceph:v20) · rsync 3.2.5 · stunnel 5.71

Testing

make test          # unit tests (uses envtest)
make test-e2e      # end-to-end tests (Kind cluster with Rook Ceph)
make lint          # golangci-lint
make lint-fix      # auto-fix linting issues

Local e2e tests use Kind (make test-e2e). CI uses Minikube. Both deploy Rook Ceph, install the snapshot controller, build and deploy the operator, then run the Ginkgo test suite. Requires Docker.

Project Structure

ceph-volsync-plugin/
├── cmd/
│   ├── manager/        # Kubernetes operator entry point
│   └── mover/          # Mover pod entry point (worker factory)
├── internal/
│   ├── ceph/           # go-ceph wrappers (CephFS diff, RBD diff, config)
│   ├── controller/     # ReplicationSource and ReplicationDestination controllers
│   ├── mover/ceph/     # Mover builder and K8s Job orchestration
│   ├── proto/          # gRPC/protobuf definitions (SyncService, VersionService)
│   └── worker/         # Data transfer workers
│       ├── cephfs/     # CephFS source and destination workers
│       ├── rbd/        # RBD source and destination workers
│       ├── pipeline/   # Concurrent Read->SendData pipeline
│       ├── tunnel/     # stunnel + rsync daemon management
│       ├── common/     # Shared worker interfaces and gRPC infrastructure
│       └── constant/   # Env var names, ports, paths
├── config/             # Kustomize: CRDs, RBAC, manager deployment
├── build/              # Containerfiles and build.env (pinned versions)
└── test/               # E2E tests and deployment scripts

Documentation

Architecture — component diagram, data flow, pipeline internals
User Guide — deployment, CephFS and RBD walkthroughs, troubleshooting
Configuration Reference — all parameters, env vars, and CLI flags
Contributing — dev setup, code generation workflow, PR process

License

Apache License 2.0 — see LICENSE.

Name		Name	Last commit message	Last commit date
.devcontainer		.devcontainer
.github/workflows		.github/workflows
build		build
cmd		cmd
config		config
docs		docs
hack		hack
internal		internal
test		test
.dockerignore		.dockerignore
.gitignore		.gitignore
.golangci.yml		.golangci.yml
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
Makefile		Makefile
PROJECT		PROJECT
README.md		README.md
go.mod		go.mod
go.sum		go.sum

Navigation Menu

Uh oh!

Repository files navigation

ceph-volsync-plugin

Overview

Features

Architecture

Prerequisites

Quick Start

Install the operator

Configure replication

Configuration

CR external parameters

Operator environment variables

Building

Testing

Project Structure

Documentation

License

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

ceph-volsync-plugin

Overview

Features

Architecture

Prerequisites

Quick Start

Install the operator

Configure replication

Configuration

CR external parameters

Operator environment variables

Building

Testing

Project Structure

Documentation

License

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages