diff --git a/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md new file mode 100644 index 000000000..cbd1cb248 --- /dev/null +++ b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md @@ -0,0 +1,161 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515008 +--- + +# How to Deploy CloudBeaver for PostgreSQL Management + +## Issue + +You need a web-based SQL client for browsing and querying PostgreSQL instances managed by Alauda Application Services, without installing a desktop tool on every operator's workstation. CloudBeaver is the open-source web edition of DBeaver and runs as a single-pod Deployment on Kubernetes. This how-to deploys CloudBeaver and connects it to a PostgreSQL cluster managed by the PostgreSQL Operator. + +## Environment + +- Any Kubernetes cluster reachable to operators (NodePort exposure is used below; Ingress / Route works equally well) +- A `StorageClass` capable of provisioning ReadWriteOnce PVCs — used to persist CloudBeaver workspace state across pod restarts +- Network reachability from the CloudBeaver pod to the target PostgreSQL Service (`` on port 5432) + +## Resolution + +### 1. Prepare the manifest + +Save the following to `cloudbeaver.yaml`. Adjust `storageClassName`, image registry, and resource requests to match your environment: + +```yaml +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cloudbeaver +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: sc-topolvm # replace with any RWO StorageClass available in the cluster + volumeMode: Filesystem +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cloudbeaver +spec: + replicas: 1 + selector: + matchLabels: + app: cloudbeaver + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 1 + template: + metadata: + labels: + app: cloudbeaver + spec: + containers: + - name: cloudbeaver + image: docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest + imagePullPolicy: Always + ports: + - name: web + containerPort: 8978 + protocol: TCP + resources: + limits: + cpu: "1" + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi + volumeMounts: + - name: cloudbeaver-data + mountPath: /opt/cloudbeaver/workspace + volumes: + - name: cloudbeaver-data + persistentVolumeClaim: + claimName: cloudbeaver +--- +apiVersion: v1 +kind: Service +metadata: + name: cloudbeaver +spec: + type: NodePort + selector: + app: cloudbeaver + ports: + - name: web + port: 8978 + targetPort: 8978 + protocol: TCP +``` + +> CloudBeaver stores its administrator password, saved connections, and query history under `/opt/cloudbeaver/workspace`. Without a persistent volume, everything is lost on every pod restart. Confirm the chosen `storageClassName` exists in the target cluster before applying. + +> **Air-gapped / IPv6-only clusters:** the public `docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest` image may be unreachable from the cluster. Mirror it into the cluster's own registry first and reference that path in the Deployment, for example `skopeo copy docker://docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest docker:///dbeaver/cloudbeaver:latest`. + +### 2. Deploy + +```bash +kubectl -n apply -f cloudbeaver.yaml +kubectl -n rollout status deploy/cloudbeaver +``` + +### 3. Discover the access URL + +The Service uses NodePort, so any node IP plus the allocated NodePort exposes the UI: + +```bash +namespace= +HOST=$(kubectl -n "$namespace" get pod -l app=cloudbeaver \ + -o jsonpath='{.items[0].status.hostIP}') +PORT=$(kubectl -n "$namespace" get svc cloudbeaver \ + -o jsonpath='{.spec.ports[0].nodePort}') +echo "http://$HOST:$PORT" +``` + +Open the URL in a browser. + +### 4. Initial setup + +1. On first launch CloudBeaver prompts you to set the administrator password. Choose a strong password and store it safely — this account governs all subsequent server-side configuration. +2. Log in with the administrator user. +3. (Optional) Switch the UI language under the user menu in the top-right. + +### 5. Connect to a PostgreSQL instance + +1. Click **New Connection** and choose **PostgreSQL**. +2. Fill **Host** with the cluster Service name and **Port** `5432`. From inside the cluster the host is `.` (the read-write Service); the `-repl` Service points at replicas. From outside the cluster, retrieve the NodePort or LoadBalancer address: + + ```bash + kubectl -n get svc + ``` + +3. Enter the database user and password. The Operator stores each role's credentials in a Secret named `..credentials.postgresql.acid.zalan.do`. Retrieve the `postgres` superuser password from its Secret: + + ```bash + kubectl get secret postgres..credentials.postgresql.acid.zalan.do \ + -n -o jsonpath='{.data.password}' | base64 -d; echo + ``` + + The same Secret also carries `username`, `host` and `port` keys. + +4. (Optional) Set **Database** to the target database; otherwise CloudBeaver connects to the default `postgres` database. +5. Under **Access Management**, grant the current CloudBeaver user permission to use the new connection. +6. Save and test the connection. SQL editors can now be opened from the browser and queries executed against the PostgreSQL cluster. + +### 6. Uninstall + +```bash +kubectl -n delete -f cloudbeaver.yaml +``` + +The PVC is removed alongside the rest of the manifest. To preserve CloudBeaver state for a future redeploy, delete only the Deployment and Service and re-attach the existing PVC during the next install. diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md new file mode 100644 index 000000000..db2fb7e8b --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md @@ -0,0 +1,364 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.3 +id: KB260720001 +--- + +# PostgreSQL Instance Cross-Cluster Migration Guide (Operator v4.3.3) + +## Background + +### The Challenge + +Sometimes a running PostgreSQL instance must move to a different Kubernetes cluster — cluster decommissioning, hardware refresh, moving a workload between ACP platforms, or consolidating environments. Dump-and-restore migrations require downtime proportional to database size and lose writes made after the dump. + +### The Solution + +This guide migrates a PostgreSQL instance across clusters using the operator's hot standby (cross-cluster replication) feature: the target instance is created as a *standby cluster* that bootstraps directly from the source via streaming replication, stays continuously in sync, and is then promoted in a controlled two-phase switchover. Downtime is limited to the switchover itself (seconds to a couple of minutes), and data integrity is verified with checksums before and after cutover. + +This procedure was validated end-to-end with PostgreSQL Operator v4.3.3 on two separate ACP platforms (source and target in different platforms, connected via NodePort). It builds on the [PostgreSQL Hot Standby Cluster Configuration Guide](./How_to_Use_PostgreSQL_Hot_Standby_Cluster.md) (KB251000009); read that first for concept background. + +## Environment Information + +- PostgreSQL Operator: v4.3.3 on **both** source and target clusters (see [Important Limitations](#important-limitations)) +- ACP: any 4.x cluster able to run the v4.3.3 operator; source and target may be in different ACP platforms +- PostgreSQL: same major version on both sides (this guide uses 16) + +## Important Limitations + +- **The operator version must match on both sides.** Cross-version pairing (e.g. a v4.2+/v4.3 standby against a primary managed by v4.1.x) fails with `pq: column "external_ip" does not exist` and — dangerously — leaves the "standby" running as an empty independent primary (tracked as ECO-703). See [Troubleshooting](#troubleshooting). +- Source and target must run the same PostgreSQL major version. +- The standby cluster must initially be created with `numberOfInstances: 1`; scale it up after promotion. +- `replSvcType` must be identical on both clusters. +- The target cluster must be able to reach the source cluster's node IP + NodePort (standby pulls from primary). Verify this before starting — see Step 2. If there is **no network path between the clusters**, the streaming approach cannot work — use the [workstation-relayed logical migration](#alternative-migration-without-inter-cluster-connectivity) instead. +- On mixed-architecture target clusters, pin the instance (and ideally the operator) to nodes matching the source architecture. Streaming replication copies the data directory bit-for-bit; PostgreSQL does not support mixed-architecture replication. + +## Migration Overview + +``` +Step 1 Prepare source: enable clusterReplication, record NodePort, baseline checksums +Step 2 Preflight: network reachability, version/arch checks +Step 3 Create standby on target (bootstraps from source, stays streaming) +Step 4 Verify sync: identity, lag, checksums +Step 5 Cutover: two-phase switchover (demote source, promote target) +Step 6 Post-migration: repoint clients; keep reverse standby or dismantle +``` + +## Step 1: Prepare the Source Instance + +If the source instance does not yet have cluster replication enabled, patch it (this is an online change; the operator creates the replication metadata and exposes the master service): + +```bash +SRC_NS="pg-migrate" +SRC_CLUSTER="acid-mig" + +kubectl -n $SRC_NS patch postgresql $SRC_CLUSTER --type merge -p '{ + "spec": { + "clusterReplication": {"enabled": true, "replSvcType": "NodePort"}, + "postgresql": {"parameters": {"max_slot_wal_keep_size": "10GB"}} + } +}' +``` + +> `max_slot_wal_keep_size` bounds WAL retained for the replication slot if the standby disconnects. Size it to your volume: it must fit in the instance's free disk space, and it defines how long a standby outage you can tolerate before a re-bootstrap is needed. + +Record the connection coordinates the standby will use: + +```bash +# NodePort of the source master service +kubectl -n $SRC_NS get svc $SRC_CLUSTER -o jsonpath='{.spec.ports[0].nodePort}{"\n"}' + +# A node IP hosting the instance (any node IP of the cluster works for NodePort) +kubectl -n $SRC_NS get pod -l cluster-name=$SRC_CLUSTER -o jsonpath='{range .items[*]}{.status.hostIP}{"\n"}{end}' +``` + +Confirm the source registered itself in the replication metadata (role `primary`, correct `node_port`): + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -x \ + -c "SELECT * FROM sys_operator.multi_cluster_info;" +``` + +Take a data integrity baseline. Adapt the checksum query to your schema — the point is a number you can compare after migration: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -d -tA -c " +SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY relname;" +# Example per-table checksum: +# SELECT count(*), sum(hashtext(id::text || payload)) FROM your_table; +``` + +Finally, force a checkpoint so the standby's basebackup starts from a consistent point: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -c "CHECKPOINT;" +``` + +## Step 2: Preflight Checks on the Target + +**Network reachability** — the standby (operator pod *and* PostgreSQL pods) must reach the source NodePort. Test from a pod on the overlay network of the target cluster, on the node(s) where the instance will run: + +```bash +# From any pod with bash on the target cluster: +kubectl exec -- bash -c 'timeout 4 bash -c "echo > /dev/tcp//" && echo OPEN || echo CLOSED' +``` + +If this prints `CLOSED`, stop and fix connectivity first. Note that reachability can differ per node (broken egress on individual nodes has been observed); test from the nodes you will schedule onto. + +**Version check** — both operators must be v4.3.3 (or at least the same version): + +```bash +kubectl -n operators get csv | grep postgres-operator +``` + +**Architecture** — on mixed-architecture targets, decide the node set now and pin with `nodeAffinity` (shown in Step 3). + +## Step 3: Create the Standby on the Target Cluster + +Create the namespace and a bootstrap secret holding the **source** cluster's admin credentials: + +```bash +TGT_NS="pg-migrate" + +# Read the admin password from the source cluster: +kubectl --context -n $SRC_NS get secret \ + postgres.${SRC_CLUSTER}.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d +``` + +```yaml +kind: Secret +apiVersion: v1 +metadata: + name: standby-bootstrap-secret + namespace: pg-migrate +type: kubernetes.io/basic-auth +stringData: + username: postgres + password: "" +``` + +Create the standby instance. Keep PostgreSQL version, parameters, and volume size aligned with the source; start with a single replica: + +```yaml +apiVersion: acid.zalan.do/v1 +kind: postgresql +metadata: + name: acid-mig # name may differ from the source + namespace: pg-migrate +spec: + teamId: acid + numberOfInstances: 1 # required initially for standby clusters + postgresql: + version: "16" # must match the source major version + parameters: + max_slot_wal_keep_size: '10GB' + volume: + size: 5Gi # same capacity as source + storageClass: + # Only needed on mixed-architecture clusters — match the source arch: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/arch + operator: In + values: ["arm64"] + clusterReplication: + enabled: true + isReplica: true + peerHost: "" + peerPort: + replSvcType: NodePort + bootstrapSecret: standby-bootstrap-secret +``` + +What happens on creation: the operator connects to the source, copies the database user credential secrets into the target namespace, creates a local `-xcr` service whose endpoints point at the source nodes, and bootstraps the pod with `pg_basebackup` from the source. `patronictl list` shows `creating replica` during the basebackup (duration depends on database size and link bandwidth), then `Standby Leader | streaming`. + +## Step 4: Verify Synchronization + +All three checks below must pass before cutover. + +**1. Streaming state and shared identity** — the cluster identifier printed by `patronictl list` must be *identical* on source and target (it is the PostgreSQL system identifier). If the target shows a different identifier, it bootstrapped as an independent cluster and contains none of your data — see [Troubleshooting](#troubleshooting). + +```bash +# Target — expect: Standby Leader | streaming +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list + +# Source — expect the standby connected, plus the slot active: +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -c \ + "SELECT application_name, client_addr, state FROM pg_stat_replication; + SELECT slot_name, active FROM pg_replication_slots WHERE slot_name='xdc_hotstandby';" +``` + +**2. Replication lag** — write on the source, confirm it appears on the target within seconds, and compare LSNs: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -tA -c "SELECT pg_current_wal_lsn();" +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -tA -c "SELECT pg_last_wal_replay_lsn();" +``` + +**3. Data checksums** — rerun the Step 1 baseline queries on the target; every value must match. + +## Step 5: Cutover (Two-Phase Switchover) + +Perform the switchover in the documented order — demote first, then promote — so there is never a moment with two writable primaries. + +1. **Stop application writes** to the source (scale writers down, or hold traffic at the application layer). + +2. **Confirm zero lag** (Step 4, check 2 — the two LSNs must be equal once writes stop). + +3. **Phase 1 — demote the source** to a standby: + +```bash +kubectl --context -n $SRC_NS patch postgresql $SRC_CLUSTER --type merge \ + -p '{"spec":{"clusterReplication":{"isReplica":true},"numberOfInstances":1}}' +``` + +Wait until the source shows `Standby Leader` (it may pass through `stopped` briefly). The demoted source finds the target automatically through the replication metadata — no `peerHost` needs to be added to its spec. + +4. **Phase 2 — promote the target** and scale it to full size: + +```bash +kubectl --context -n $TGT_NS patch postgresql acid-mig --type merge \ + -p '{"spec":{"clusterReplication":{"isReplica":false},"numberOfInstances":2}}' +``` + +5. **Gate on the real promotion signal.** Do not rely on `pg_is_in_recovery()` alone — during promotion and scale-up the cluster status can read `Running` while Patroni is still converting roles. Wait until **all** of the following hold: + +```bash +# a) Patroni shows a Leader in state running (a timeline increase here is normal): +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list +# b) the CR reports Running: +kubectl -n $TGT_NS get postgresql acid-mig -o jsonpath='{.status.PostgresClusterStatus}{"\n"}' +# c) the new replica is streaming (after scale-up): +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -c \ + "SELECT application_name, state FROM pg_stat_replication;" +``` + +6. **Verify writes and integrity** on the new primary: insert a marker row, rerun the checksum queries, compare with the baseline. + +> A leadership change between the target's pods during promote+scale-up (with an extra timeline bump) is normal operator rolling behavior and does not indicate a problem. + +## Step 6: Post-Migration + +- **Repoint clients** to the target cluster's service (and update any external access such as NodePort/LoadBalancer/ingress used by applications). +- The demoted source is now a live **reverse DR standby** of the target: writes on the target replicate back to it. Choose one: + - **Keep it** as disaster-recovery / rollback insurance (recommended for at least a soak period). Rolling back is the same two-phase switchover in the opposite direction. + - **Dismantle it** and remove the replication setup entirely — see below. +- Scale/tune the target (replica count, resources, backup schedule, monitoring) to match what the source had. + +### Dismantling the Source and Removing the Replication Configuration + +Perform these steps **in order** — the standby must be gone before the primary's replication configuration is removed, otherwise the standby loses its upstream while still streaming. + +**1. Delete the demoted source instance** (on the source cluster): + +```bash +kubectl --context -n $SRC_NS delete postgresql $SRC_CLUSTER +# PVC retention on CR deletion depends on operator configuration — check and +# remove any leftovers to reclaim storage: +kubectl --context -n $SRC_NS get pvc -l cluster-name=$SRC_CLUSTER +kubectl --context -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found +``` + +The operator deletes the credential secrets it owns along with the CR. + +**2. Convert the target to a normal (non-replicated) instance** by removing the `clusterReplication` block: + +```bash +kubectl --context -n $TGT_NS patch postgresql acid-mig --type json \ + -p '[{"op":"remove","path":"/spec/clusterReplication"}]' +``` + +The operator handles this transition: it re-applies normal-primary Patroni configuration, drops the `sys_operator` metadata schema from the database, and removes the `xdc_hotstandby` slot from the replication configuration. (The reverse — converting an existing normal instance *into* a standby — is not a supported transition; standbys must be created as standbys.) + +**3. Clean up leftover objects** that the operator does not remove: + +```bash +# Bootstrap secret (user-created, never operator-owned): +kubectl --context -n $TGT_NS delete secret standby-bootstrap-secret + +# The -xcr service lingers after the role change (it is only removed +# with the CR itself) — delete it: +kubectl --context -n $TGT_NS delete svc acid-mig-xcr --ignore-not-found + +# The physical replication slot can also linger even though it was removed +# from the Patroni configuration — drop it if the Step 4 check finds it: +kubectl --context -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -c \ + "SELECT pg_drop_replication_slot('xdc_hotstandby') + WHERE EXISTS (SELECT 1 FROM pg_replication_slots WHERE slot_name='xdc_hotstandby' AND NOT active);" +``` + +**4. Verify the target is a clean standalone instance:** + +```bash +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -tA -c \ + "SELECT count(*) FROM pg_replication_slots WHERE slot_name='xdc_hotstandby'; + SELECT count(*) FROM pg_namespace WHERE nspname='sys_operator';" +# Both counts must be 0 +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list +# Expect a plain Leader/Replica cluster, no Standby Leader +``` + +## Alternative: Migration Without Inter-Cluster Connectivity + +When the clusters have **no network path between them** but your workstation can reach both Kubernetes API servers, the migration can be relayed through the workstation as a logical dump/restore piped between two `kubectl exec` sessions — the clusters never talk to each other. Downtime equals the full copy duration (versus seconds for the streaming switchover), but there are no operator-version or CPU-architecture constraints, and the PostgreSQL major only needs to be the same or newer on the target. + +The full validated procedure is a separate solution: [How to Migrate a PostgreSQL Instance Between Network-Isolated Clusters](./How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md) (KB260721001). + +## Troubleshooting + +### Standby fails with `pq: column "external_ip" does not exist` — and runs as an empty independent primary + +**Cause:** operator version mismatch between source and target (e.g. source primary managed by v4.1.x, target operator v4.2+/v4.3). The replication metadata table schema differs between these lines and is never migrated (ECO-703). The standby create hook aborts midway and the pod falls through to a normal bootstrap: it comes up as a *fresh, empty, writable* primary while its CR still says `isReplica: true`. + +**Detection:** compare the cluster identifier in `patronictl list` on both sides — different identifier means independent cluster, not a standby. + +**Fix:** upgrade both operators to the same version (preferred). If the source operator cannot be upgraded immediately, add the missing column on the source primary (safe for both versions — all statements reference columns by name): + +```sql +ALTER TABLE sys_operator.multi_cluster_info ADD COLUMN IF NOT EXISTS external_ip CHAR(64) DEFAULT ''; +``` + +Then recreate the standby cleanly — see the next item. + +### Recreating a failed standby: create hook fails with secret "already exists" + +The operator copies the source's credential secrets into the target namespace *before* bootstrapping, and the copy step fails hard if they already exist from a previous attempt. To retry a standby creation from scratch, delete all of: + +```bash +kubectl -n $TGT_NS delete postgresql acid-mig +kubectl -n $TGT_NS delete pvc -l cluster-name=acid-mig # data from the failed attempt must go +kubectl -n $TGT_NS delete secret \ + postgres.acid-mig.credentials.postgresql.acid.zalan.do \ + standby.acid-mig.credentials.postgresql.acid.zalan.do +``` + +then re-apply the standby manifest. + +### Standby stuck in `creating replica` + +The basebackup is either still copying (large database — check network throughput) or cannot connect. Verify Step 2 reachability *from the node the standby pod landed on*; per-node egress differences are a real failure mode. Also confirm the bootstrap secret contains the current source admin password. + +### Replication slot errors (`TypeError ... 'int' and 'NoneType'`) in standby logs + +Known Patroni issue; replication generally continues. See the [Hot Standby guide's troubleshooting section](./How_to_Use_PostgreSQL_Hot_Standby_Cluster.md#troubleshooting) — drop the `xdc_hotstandby` slot if needed. + +## Verification Checklist + +| Check | When | Pass condition | +|---|---|---| +| Network preflight | before Step 3 | target overlay pod reaches `SRC_NODE_IP:NODEPORT` | +| Operator versions | before Step 3 | identical on both clusters | +| Shared system identifier | after Step 3 | `patronictl list` identifier equal on both sides | +| Streaming | after Step 3 | target `Standby Leader / streaming`; source slot `xdc_hotstandby` active | +| Checksums (pre-cutover) | Step 4 | all values match baseline | +| LSN parity | Step 5, writes stopped | `pg_current_wal_lsn()` == `pg_last_wal_replay_lsn()` | +| Promotion | Step 5 | Patroni Leader running + CR Running + replica streaming | +| Checksums (post-cutover) | Step 5 | all values match baseline; new write on target succeeds | +| Reverse replication (if source kept) | Step 6 | target write appears on demoted source | diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md new file mode 100644 index 000000000..91f66f19e --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -0,0 +1,541 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.x +id: KB260721001 +--- + +# How to Migrate a PostgreSQL Instance Between Network-Isolated Clusters + +## Background + +### The Challenge + +The streaming-replication migration described in the [PostgreSQL Instance Cross-Cluster Migration Guide](./How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md) requires the target cluster to reach the source cluster's network. In many real deployments the clusters are network-isolated — separate platforms, firewalled sites, disconnected security zones — and no such path exists or can be opened. + +### The Solution + +When an administrator workstation can reach **both** Kubernetes API servers, the migration can be relayed through the workstation: `pg_dump` runs in the source pod, `pg_restore` runs in the target pod, and the data flows between them through two `kubectl exec` channels piped together on the workstation. The clusters never exchange a packet. + +The procedure is four human operations: create the target instance (Step 1), stop application writes (Step 2), run the migration script (Step 3), and cut over (Steps 4–5). The script migrates **every** application database of the instance and verifies each one. + +Because the transfer is logical (SQL-level), this method has no same-version requirement: it works across different operator versions, across CPU architectures, and from an older PostgreSQL major to the same or a newer major on the target. Migrating to an **older** PostgreSQL major (a downgrade) is not supported. + +**Trade-off:** unlike streaming replication, this is a point-in-time copy. Writes made on the source after the dump starts are not transferred — application writes must be stopped for the whole dump+restore window, so downtime equals the full copy duration. + +### What Is and Is Not Migrated + +A logical migration carries less than a byte-level copy. Know the boundary before relying on it: + +| Carried by this procedure | NOT carried — handle deliberately | +|---|---| +| Tables, data, indexes, views, functions, sequences (including positions) | `GRANT`s (`-x` skips ACLs at restore; they remain inside the dump file — see Step 4) | +| Roles and their attributes (generated into the target CR in Step 1) | `COMMENT ON` metadata (`--no-comments`; a deliberate, lossy trade for a verifiable exit code) | +| Database encoding and locale (enforced by the script) | CR spec beyond `users`/`databases`/`volume`: `resources`, `postgresql.parameters`, `patroni.pg_hba`, `connectionPooler`, sidecars, load-balancer flags — port them in the target CR yourself | +| Database- and role-in-database-level settings (`ALTER DATABASE/ROLE ... SET`) | Objects requiring superuser: event triggers, publications/subscriptions, FDW servers and user mappings | +| Extensions (schema placement and version preserved) | Tablespace layout (`--no-tablespaces` maps everything to the default) | +| Planner statistics (regenerated via `ANALYZE` in the window) | Instances using `preparedDatabases` (its `_owner/_reader/_writer` role model and `user_management` schema are out of scope for this guide) | + +Applications connecting through a source-side `connectionPooler` service need the pooler declared in the target CR, or their connection endpoint changes at cutover. + +## Environment Information + +- PostgreSQL Operator: any 4.x version on each side (versions do **not** need to match) +- PostgreSQL: target major version equal to or newer than the source +- Workstation: `bash` and `kubectl` access (kubeconfig/context) to both clusters + +## Prerequisites + +- Two kubeconfig contexts on the workstation, one per cluster; verify both work: + +```bash +SRC_CTX=""; SRC_NS=""; SRC_CLUSTER="" +TGT_CTX=""; TGT_NS=""; TGT_CLUSTER="" + +kubectl --context $SRC_CTX -n $SRC_NS get postgresql $SRC_CLUSTER +kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER 2>/dev/null || echo "target instance not created yet" +``` + +- Enough workstation bandwidth to both API servers for the database size (all data flows through the workstation). +- A maintenance window sized from a rehearsal (below) — not guessed. +- A stable window on both instances: no pending node maintenance or evictions. The script resolves each master pod once at start; a Patroni failover mid-run sends the stream to a demoted pod. For long runs, consider `patronictl pause` on both sides for the duration, and redo any database that was in flight if a failover happens anyway. +- Sufficient Kubernetes permissions in both namespaces: `get` and `list` on `postgresqls.acid.zalan.do`, `pods`, and `secrets`; `create` on `pods/exec`; on the target side additionally `create` on `postgresqls` and `secrets` (credential pre-staging) and `patch` on `secrets` (troubleshooting); on the source side, for the final cleanup, `delete` on `postgresqls` and `persistentvolumeclaims`. +- The commands assume the operator's standard Spilo image: pod labels `spilo-role`/`cluster-name`, database container named `postgres`, and client binaries under `/usr/lib/postgresql//bin/` for the majors that image generation bundles (current images bundle 13–17; the script verifies the major it needs before starting). Custom or non-Spilo images require adapting these labels and paths. + +### Rehearse and Size the Window + +The stop-write window equals the full dump+restore+verify duration — do not open a maintenance window without knowing that number. Rehearse first: run Step 1 against a throwaway target instance and run the Step 3 script **without stopping source writes**. Expect `FAIL` verdicts (`SOURCE CHANGED DURING DUMP` is normal here) — the rehearsal measures *duration*, not correctness. Then delete the rehearsal target (CR and PVCs). + +Set a go/no-go rule before the real window: if verification has not passed with an agreed margin of window time remaining, roll back — re-enable source writes, keep the source untouched, and investigate offline. The target can be dropped and recreated at any time; the source is the asset. + +## Step 1: Create the Target Instance + +List the application databases and owners the source actually holds: + +```bash +SRC_POD=$(kubectl --context $SRC_CTX -n $SRC_NS get pod \ + -l spilo-role=master,cluster-name=$SRC_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT datname, pg_get_userbyid(datdba) AS owner FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY 1;" +``` + +Every CR-managed database in this list must be declared in the target CR (`spec.databases`, with its owner in `spec.users`) — the operator creates the roles and databases and manages their credentials; do not attempt to dump global objects (roles) with `pg_dumpall` from the source. The `postgres` maintenance database is managed by the operator/image on each side and is not migrated. Databases owned by `postgres` (created outside the CR spec) do not go into the CR — the migration script creates them on the target automatically. + +The `users:`/`databases:` sections of the target CR can be generated directly from the source instead of hand-written. The `users:` generator enumerates **all** application roles from `pg_roles` — not just database owners — because read-only, monitoring, and per-service accounts that own no database must also exist on the target, and it carries each role's attributes as `userFlags` (a role generated as `some_role: []` on the target would silently lose `NOLOGIN`, `CREATEDB`, etc.): + +```bash +{ + echo " users:" + kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT ' ' || rolname || ': [' || array_to_string(ARRAY( + SELECT f FROM unnest(ARRAY[ + CASE WHEN rolsuper THEN 'superuser' END, + CASE WHEN rolcreatedb THEN 'createdb' END, + CASE WHEN rolcreaterole THEN 'createrole' END, + CASE WHEN NOT rolcanlogin THEN 'nologin' END]) f WHERE f IS NOT NULL), ',') || ']' + FROM pg_roles + WHERE rolname NOT LIKE 'pg\\_%' + AND rolname NOT IN ('postgres','standby','pooler','admin','zalandos','cron_admin','robot_zmon') + ORDER BY (rolname) COLLATE \"C\";" + echo " databases:" + kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT ' ' || datname || ': ' || pg_get_userbyid(datdba) FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' AND pg_get_userbyid(datdba) <> 'postgres' ORDER BY 1;" +} +``` + +Review the generated `users:` list before applying: the `NOT IN` filter excludes the standard operator/image system roles — if your deployment defines additional operator-managed roles, exclude those too rather than declaring them in the CR. + +### Preserve application credentials + +By default the target operator generates **new** passwords for the CR users, which would force every application to be re-configured at cutover. To keep the source passwords, copy each application user's credential secret into the target namespace **before creating the CR** — when the operator finds an existing credential secret, it adopts it and sets the role's password from it instead of generating a new one: + +```bash +SUFFIX="credentials.postgresql.acid.zalan.do" +for SEC in $(kubectl --context $SRC_CTX -n $SRC_NS get secret \ + -l application=spilo,cluster-name=$SRC_CLUSTER -o name); do + NAME=${SEC#secret/} + U=${NAME%.$SRC_CLUSTER.$SUFFIX} + [ "$U" = "$NAME" ] && continue # not a credential secret + case "$U" in postgres|standby|pooler) continue ;; esac # operator-managed system users + UB64=$(kubectl --context $SRC_CTX -n $SRC_NS get $SEC -o jsonpath='{.data.username}') + PB64=$(kubectl --context $SRC_CTX -n $SRC_NS get $SEC -o jsonpath='{.data.password}') + [ -n "$UB64" ] && [ -n "$PB64" ] \ + || { echo "WARNING: $NAME has an empty username/password field — skipping (fix on the source first)" >&2; continue; } + kubectl --context $TGT_CTX -n $TGT_NS apply -f - < +``` + +Restoring into a single instance keeps the full-restore WAL burst off streaming replication and the WAL archive; scale `numberOfInstances` up after verification (Step 4). Size `volume` with headroom — a logical restore holds data, indexes under construction, and the restore's own WAL at the same time. + +Wait until the instance is actually ready — `Running` reports pod health, while the operator creates roles, databases, and credential secrets **asynchronously after** that, so poll for the objects Step 3 depends on: + +```bash +until [ "$(kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER \ + -o jsonpath='{.status.PostgresClusterStatus}')" = "Running" ]; do sleep 5; done + +TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ + -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +# every CR-declared user must exist as a role, with its credential secret: +until [ "$(kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + psql -U postgres -tA -c "SELECT count(*) FROM pg_roles WHERE rolname = 'app_owner';")" = "1" ] \ + && kubectl --context $TGT_CTX -n $TGT_NS get secret \ + "app-owner.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do" >/dev/null 2>&1 +do sleep 5; done +``` + +Note the secret's name: the operator sanitizes role names into RFC 1123 form, so a role like `app_owner` gets the secret `app-owner..credentials...` (`_` becomes `-`). Use the sanitized form wherever a secret is fetched by name. + +## Step 2: Stop Writes + +Stop application writes on the source — the transfer is a point-in-time copy, and anything written after the dump starts is lost. + +Do not rely on "the team said the app is stopped": **enforce** it. Blocking new connections and terminating leftovers turns a would-be silent data loss into an immediate, visible failure: + +```bash +# per application database: no new non-superuser connections, kill the stragglers +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -c "ALTER DATABASE CONNECTION LIMIT 0;" +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity + WHERE datname = '' AND usename NOT IN ('postgres','standby');" +``` + +(If you abort the migration, restore access with `ALTER DATABASE CONNECTION LIMIT -1;`.) As a second line of defense, the Step 3 script re-reads the source row counts after each transfer and fails that database with `SOURCE CHANGED DURING DUMP` if anything moved. + +The script records and compares exact per-table row counts automatically. For stronger guarantees on critical tables, additionally record per-table checksums now and re-check them in Step 4. Use a construct that is stable across PostgreSQL versions and explicit about NULLs (an aggregate over an expression that can be NULL silently skips those rows): + +```bash +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -d -tA -c \ + "SELECT count(*), md5(string_agg(id::text || ':' || coalesce(payload,''), '|' ORDER BY id)) + FROM ;" +``` + +## Step 3: Run the Migration Script + +The script performs the whole migration in one run: it enumerates the source's application databases, ensures each exists on the target **with the source's encoding and locale** (recreating empty mismatched ones — this also covers CR-created databases, which the operator builds with target defaults), pre-creates missing extensions preserving their schema and version, transfers each database, carries over database-level settings, runs `ANALYZE`, and verifies row counts, an object census, and sequence positions — reporting `PASS`/`FAIL` per database and exiting non-zero if anything failed. + +Fill in the six variables at the top, then run it with `bash`. Without arguments it migrates every database; pass database names (`bash migrate.sh appdb`) to migrate only those — used to redo a single database after a failure. Every transfer goes through the same script in one of two modes: + +- **File mode** (default; override the location with `DUMP_DIR=/path`): each database is dumped to `$DUMP_DIR/.dump` first, then restored from the file. This is the default because the long-lived `kubectl exec` streams are the fragile link in this relay — API-server load balancers and kubelet idle timeouts can cut a multi-hour stream, and a file makes that cost one database instead of the whole run. By default every run takes a **fresh** dump; set `REUSE_DUMPS=1` to reuse completed dumps (resumable). Reuse is only safe **within the same stopped-write maintenance window** — a dump taken before source writes resumed would silently migrate stale data. +- **Pipe mode** (`PIPE_MODE=1 bash migrate.sh`): source streams straight into the target — no intermediate storage on the workstation. Acceptable for small instances (roughly under 5 GB) where a cut stream just means a quick rerun; for anything that takes hours, use file mode. + +```bash +#!/usr/bin/env bash +# Whole-instance PostgreSQL migration relayed through the workstation. +# Migrates every application database of $SRC_CLUSTER into $TGT_CLUSTER. +set -u -o pipefail + +SRC_CTX=""; SRC_NS=""; SRC_CLUSTER="" +TGT_CTX=""; TGT_NS=""; TGT_CLUSTER="" + +# File mode is the default (resumable; a broken exec stream costs one database, +# not the run). PIPE_MODE=1 streams directly — for small instances only. +DUMP_DIR="${DUMP_DIR:-./pg-migrate-dumps}" + +SRC_POD=$(kubectl --context "$SRC_CTX" -n "$SRC_NS" get pod \ + -l spilo-role=master,cluster-name="$SRC_CLUSTER" -o jsonpath='{.items[0].metadata.name}') \ + && [ -n "$SRC_POD" ] || { echo "ERROR: cannot find source master pod for $SRC_CLUSTER" >&2; exit 1; } +TGT_POD=$(kubectl --context "$TGT_CTX" -n "$TGT_NS" get pod \ + -l spilo-role=master,cluster-name="$TGT_CLUSTER" -o jsonpath='{.items[0].metadata.name}') \ + && [ -n "$TGT_POD" ] || { echo "ERROR: cannot find target master pod for $TGT_CLUSTER" >&2; exit 1; } + +srcsql() { local db=$1; shift; kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } +tgtsql() { local db=$1; shift; kubectl --context "$TGT_CTX" -n "$TGT_NS" exec "$TGT_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } +tgtexec() { kubectl --context "$TGT_CTX" -n "$TGT_NS" exec "$TGT_POD" -c postgres -- "$@"; } + +# Multi-hour dumps/restores must not be killed by timeouts configured on the +# instance or its roles. +PGOPT='-c statement_timeout=0 -c lock_timeout=0 -c idle_in_transaction_session_timeout=0' + +# Client binaries matching the SOURCE server major; images bundle a bounded set +# of majors, so verify before starting instead of failing mid-restore. +SRC_MAJOR=$(srcsql postgres -c "SHOW server_version;" | cut -d. -f1) +case "$SRC_MAJOR" in ''|*[!0-9]*) echo "ERROR: cannot determine source PostgreSQL major (psql via $SRC_POD failed)" >&2; exit 1 ;; esac +PG_BIN=/usr/lib/postgresql/$SRC_MAJOR/bin +tgtexec test -x "$PG_BIN/pg_restore" \ + || { echo "ERROR: $PG_BIN/pg_restore not present in the target image" >&2; exit 1; } +echo "Source PostgreSQL major: $SRC_MAJOR (client binaries: $PG_BIN)" + +# Verification queries. The explicit COLLATE "C" pins the sort order — the two +# databases can legitimately have different collations, and a text comparison +# of differently-ordered output would report false differences. The object +# census excludes extension-owned objects (extension versions differ across +# PostgreSQL majors) and counts everything else by kind. +COUNT_QUERY="SELECT schemaname||'.'||relname, (xpath('/row/c/text()', query_to_xml(format( + 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint + FROM pg_stat_user_tables + WHERE schemaname NOT IN ('metric_helpers','user_management') + ORDER BY (schemaname||'.'||relname) COLLATE \"C\";" +OBJ_QUERY="SELECT c.relkind||':'||count(*) FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname NOT IN ('pg_catalog','information_schema','pg_toast','metric_helpers','user_management') + AND NOT EXISTS (SELECT 1 FROM pg_depend dep WHERE dep.objid = c.oid AND dep.deptype = 'e') + GROUP BY c.relkind ORDER BY (c.relkind::text) COLLATE \"C\";" +SEQ_QUERY="SELECT schemaname||'.'||sequencename||'='||coalesce(last_value,0) FROM pg_sequences + WHERE schemaname NOT IN ('metric_helpers','user_management') + ORDER BY (schemaname||'.'||sequencename) COLLATE \"C\";" + +FAILED=""; ATTEMPTED=0 +while read -r DB OWNER; do + [ -z "$DB" ] && continue + # With arguments, migrate only the named databases (e.g. to redo one FAIL). + if [ "$#" -gt 0 ]; then + case " $* " in *" $DB "*) ;; *) continue ;; esac + fi + ATTEMPTED=$((ATTEMPTED+1)) + echo "=== migrating database: $DB (owner: $OWNER) ===" + + # The target database must exist with the SOURCE's encoding and locale — + # plain CREATE DATABASE inherits the target template's defaults, silently + # changing sort order and LIKE behavior (or failing the restore outright on + # an encoding mismatch). TEMPLATE template0 is required to override them. + # CR-declared databases are covered too: the operator creates those with + # target defaults, so an empty database with the wrong locale is recreated. + read -r ENC COLL CTYPE < <(srcsql postgres -F' ' -c \ + "SELECT pg_encoding_to_char(encoding), datcollate, datctype FROM pg_database WHERE datname = '$DB';") + CREATE_SQL="CREATE DATABASE \"$DB\" OWNER \"$OWNER\" TEMPLATE template0 ENCODING '$ENC' LC_COLLATE '$COLL' LC_CTYPE '$CTYPE';" + TGT_LOC=$(tgtsql postgres -c "SELECT pg_encoding_to_char(encoding)||'/'||datcollate||'/'||datctype FROM pg_database WHERE datname = '$DB';") + if [ -z "$TGT_LOC" ]; then + echo " creating database $DB on target ($ENC/$COLL/$CTYPE)" + tgtsql postgres -c "$CREATE_SQL" >/dev/null || { FAILED="$FAILED $DB(create)"; continue; } + elif [ "$TGT_LOC" != "$ENC/$COLL/$CTYPE" ]; then + if [ "$(tgtsql "$DB" -c "SELECT count(*) FROM pg_stat_user_tables WHERE schemaname NOT IN ('metric_helpers','user_management');")" = "0" ]; then + echo " recreating database $DB on target with source locale ($ENC/$COLL/$CTYPE, was $TGT_LOC)" + tgtsql postgres -c "DROP DATABASE \"$DB\";" >/dev/null \ + && tgtsql postgres -c "$CREATE_SQL" >/dev/null \ + || { FAILED="$FAILED $DB(locale)"; continue; } + else + echo " FAIL: $DB exists on target with different locale ($TGT_LOC vs $ENC/$COLL/$CTYPE) and is not empty" + FAILED="$FAILED $DB(locale)"; continue + fi + fi + + # Pre-create only the extensions the target is missing (creation needs + # superuser), preserving the source's schema placement and version — a + # relocated or version-drifted extension breaks dependent objects at restore. + while read -r EXT ESCH EVER; do + [ -z "$EXT" ] && continue + [ "$(tgtsql "$DB" -c "SELECT 1 FROM pg_extension WHERE extname = '$EXT';")" = "1" ] && continue + tgtsql "$DB" -c "CREATE SCHEMA IF NOT EXISTS \"$ESCH\";" >/dev/null 2>&1 + tgtsql "$DB" -c "CREATE EXTENSION \"$EXT\" WITH SCHEMA \"$ESCH\" VERSION '$EVER';" >/dev/null 2>&1 \ + || tgtsql "$DB" -c "CREATE EXTENSION \"$EXT\" WITH SCHEMA \"$ESCH\";" >/dev/null 2>&1 \ + || echo " WARNING: could not create extension $EXT (schema $ESCH) on target — restore may fail" + done < <(srcsql "$DB" -F' ' -c \ + "SELECT e.extname, n.nspname, e.extversion FROM pg_extension e + JOIN pg_namespace n ON n.oid = e.extnamespace WHERE e.extname <> 'plpgsql';") + + # Baseline on the source (exact counts, object census, sequence positions). + SRC_COUNTS=$(srcsql "$DB" -c "$COUNT_QUERY") + SRC_OBJS=$(srcsql "$DB" -c "$OBJ_QUERY"); SRC_SEQS=$(srcsql "$DB" -c "$SEQ_QUERY") + + # Transfer. The restore connects as postgres over the pod-local socket and + # switches to the owner with --role: objects land owned by the owner, with + # no password handling and nothing secret in the kubectl argument list. + if [ "${PIPE_MODE:-0}" != "0" ]; then + kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ + env PGOPTIONS="$PGOPT" \ + "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management "$DB" \ + | kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + env PGOPTIONS="$PGOPT" \ + "$PG_BIN/pg_restore" -U postgres --role="$OWNER" -d "$DB" --no-owner --no-tablespaces -x + RC=$? + else + install -d -m 700 "$DUMP_DIR" + DUMP_FILE="$DUMP_DIR/$DB.dump" + if [ "${REUSE_DUMPS:-0}" != "0" ] && [ -s "$DUMP_FILE" ]; then + echo " reusing existing dump $DUMP_FILE" + else + kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ + env PGOPTIONS="$PGOPT" \ + "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management "$DB" > "$DUMP_FILE.partial" \ + && mv "$DUMP_FILE.partial" "$DUMP_FILE" \ + || { rm -f "$DUMP_FILE.partial"; FAILED="$FAILED $DB(dump)"; continue; } + fi + kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + env PGOPTIONS="$PGOPT" \ + "$PG_BIN/pg_restore" -U postgres --role="$OWNER" -d "$DB" --no-owner --no-tablespaces -x < "$DUMP_FILE" + RC=$? + fi + + # Carry over database- and role-in-database-level settings + # (ALTER DATABASE ... SET / ALTER ROLE ... IN DATABASE ... SET) — pg_dump + # without --create does not emit them, so they would silently disappear. + srcsql postgres -c \ + "SELECT 'ALTER '||CASE WHEN s.setrole = 0 THEN 'DATABASE '||quote_ident(d.datname) + ELSE 'ROLE '||quote_ident(r.rolname)||' IN DATABASE '||quote_ident(d.datname) END + ||' SET '||quote_ident(split_part(cfg,'=',1))||' = '||quote_literal(substr(cfg, strpos(cfg,'=')+1))||';' + FROM pg_db_role_setting s + JOIN pg_database d ON d.oid = s.setdatabase + LEFT JOIN pg_roles r ON r.oid = s.setrole + CROSS JOIN LATERAL unnest(s.setconfig) cfg + WHERE d.datname = '$DB';" \ + | kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + psql -U postgres -v ON_ERROR_STOP=1 -f - >/dev/null \ + || echo " WARNING: could not carry over ALTER DATABASE/ROLE ... SET settings for $DB" + + # A restore ships no planner statistics — without ANALYZE the first minutes + # of production traffic run on empty stats (sequential scans everywhere). + # This belongs INSIDE the maintenance window, not after cutover. + tgtexec env PGOPTIONS="$PGOPT" "$PG_BIN/vacuumdb" -U postgres --analyze-in-stages -d "$DB" >/dev/null 2>&1 \ + || echo " WARNING: vacuumdb --analyze-in-stages failed for $DB — run ANALYZE manually before cutover" + + # Verify: rows, object census, and sequence positions must match — and the + # source must not have changed while the transfer ran (writes not stopped). + TGT_COUNTS=$(tgtsql "$DB" -c "$COUNT_QUERY") + TGT_OBJS=$(tgtsql "$DB" -c "$OBJ_QUERY"); TGT_SEQS=$(tgtsql "$DB" -c "$SEQ_QUERY") + SRC_RECHECK=$(srcsql "$DB" -c "$COUNT_QUERY") + if [ "$SRC_RECHECK" != "$SRC_COUNTS" ]; then + echo " FAIL: $DB (SOURCE CHANGED DURING DUMP — enforce the write stop per Step 2, then redo this database)" + FAILED="$FAILED $DB" + elif [ "$RC" -eq 0 ] && [ "$SRC_COUNTS" = "$TGT_COUNTS" ] && [ "$SRC_OBJS" = "$TGT_OBJS" ] && [ "$SRC_SEQS" = "$TGT_SEQS" ]; then + echo " PASS: $DB (rows, objects, and sequences identical)" + else + echo " FAIL: $DB (transfer exit $RC; rows $( [ "$SRC_COUNTS" = "$TGT_COUNTS" ] && echo match || echo DIFFER ); objects $( [ "$SRC_OBJS" = "$TGT_OBJS" ] && echo match || echo DIFFER ); sequences $( [ "$SRC_SEQS" = "$TGT_SEQS" ] && echo match || echo DIFFER ))" + FAILED="$FAILED $DB" + fi +done < <(srcsql postgres -F' ' -c \ + "SELECT datname, pg_get_userbyid(datdba) FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY (datname) COLLATE \"C\";") + +echo +if [ "$ATTEMPTED" -eq 0 ]; then + echo "ERROR: no databases migrated — source enumeration failed, or no database matched the arguments" >&2 + exit 1 +fi +if [ -n "$FAILED" ]; then echo "MIGRATION INCOMPLETE — failed:$FAILED"; exit 1; fi +echo "MIGRATION COMPLETE — $ATTEMPTED database(s) verified." +``` + +If a database reports `FAIL`, see [Troubleshooting](#troubleshooting); after fixing the cause, redo just that database by passing its name to the script (in file mode, add `REUSE_DUMPS=1` to skip re-dumping if writes have stayed stopped). + +### Security notes + +- This relay deliberately makes the administrator workstation a data path between two otherwise-isolated environments — exactly what an isolation policy exists to control. Confirm the data path is approved by your security/compliance owner before migrating; "each hop is TLS" is a transport property, not an authorization. +- The migration script itself handles no database passwords: dumps and restores connect as `postgres` over the pod-local socket (the restore switches to the owner with `--role`). The only password-bearing command in this guide is the Step 4 login test — see the caveat there. +- In file mode (the default), `$DUMP_DIR` holds full plaintext logical copies of the databases. The script creates it with mode `700`; if you point `DUMP_DIR` at an existing directory, restrict it yourself, encrypt at rest if your policy requires, and delete the dumps once the migration is verified. +- The data traverses the two TLS-protected `kubectl exec` channels; nothing is exposed on the network beyond the two Kubernetes API connections. + +## Step 4: Verify Application Access + +The script has already verified per-table row counts, an object census, and sequence positions, and has run `ANALYZE` on each database. Confirm in addition that each application user can actually query its data with the **target**-managed credentials (this catches ownership problems immediately), and re-check any Step 2 checksums. If you pre-staged the credential secrets in Step 1, this password is identical to the source one — applications keep their existing credentials and only the connection endpoint changes at cutover. Note the secret name uses the RFC 1123-sanitized role name (`app_owner` → `app-owner`): + +```bash +TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ + -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ + .$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d) + +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + env PGPASSWORD="$APP_PW" psql -U -h localhost -d -tA -c \ + "SELECT current_user, count(*) FROM ;" +``` + +**Caveat:** `kubectl exec` places its command — including this `PGPASSWORD` value — in the API request URI, which Kubernetes audit logging records at `Metadata` level and above, and it is also visible in local `ps` output while the command runs. This is a deliberate, one-off login *test*; if your audit policy treats recorded credentials as exposed, rotate this password after cutover (or verify from an application pod instead). + +The restore skips ACL statements (`pg_restore -x`) because they reference roles managed by the source operator. The `GRANT`s themselves are still **inside the dump file** — if additional (non-owner) users need privileges on the migrated data, either re-grant manually now, or list the dump's ACL entries (`"$PG_BIN/pg_restore" -l .dump | grep ' ACL '`) and replay just those with `pg_restore -L` once the roles exist on the target. + +## Step 5: Cut Over and Clean Up + +- Scale the target back out (`numberOfInstances: 2` or your HA baseline) and wait for the replica to catch up. +- Repoint applications to the target instance's service and re-enable writes. +- Retire the source in stages rather than deleting it immediately — it is your only rollback for data problems discovered after cutover. Keep it read-only and scaled to zero for an observation period first: + +```bash +# freeze the source but keep its data (rollback insurance) +kubectl --context $SRC_CTX -n $SRC_NS patch postgresql $SRC_CLUSTER \ + --type merge -p '{"spec":{"numberOfInstances":0}}' +``` + +- After the observation period (days, per your policy), delete it. There is no replication configuration to tear down. If your operator is configured with delete-protection annotations (`delete_annotation_date_key`/`delete_annotation_name_key`), set those annotations on the CR first or the operator ignores the deletion: + +```bash +kubectl --context $SRC_CTX -n $SRC_NS delete postgresql $SRC_CLUSTER +# PVC retention depends on operator configuration — remove leftovers: +kubectl --context $SRC_CTX -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found +``` + +## Troubleshooting + +### Application user gets `permission denied for table ...` after a successful restore + +The restore ran without switching to the application user, so all objects are owned by `postgres` (the script connects as `postgres` but switches with `--role=`; this typically comes from a hand-run restore that omitted `--role`). Either redo the database — drop and recreate it, then rerun the script with the database name — or transfer ownership in place: + +```sql +DO $$ +DECLARE r record; +BEGIN + FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'public' LOOP + EXECUTE format('ALTER TABLE public.%I OWNER TO app_owner', r.tablename); + END LOOP; + FOR r IN SELECT sequencename FROM pg_sequences WHERE schemaname = 'public' LOOP + EXECUTE format('ALTER SEQUENCE public.%I OWNER TO app_owner', r.sequencename); + END LOOP; +END $$; +``` + +This block covers tables and sequences in the `public` schema only. Views, functions, types, and objects in other schemas need analogous `ALTER ... OWNER TO` statements (enumerate them via `pg_views`, `pg_proc`, `pg_type`, or `\dn`/`\df` in psql). + +Related scope note: the migration's `--no-owner` model assumes the operator's standard layout — one database wholly owned by one CR-defined user. If a database has multiple owning roles, `SECURITY DEFINER` functions, or objects in non-`public` schemas with distinct owners, ownership collapses onto the restoring user; plan an explicit post-restore ownership and grant pass for those objects. + +### Restore reports `permission denied to create extension ...` + +The source database uses an extension that is not pre-installed on the target, and the application user cannot create it. The script warns about this (`WARNING: could not create extension`). Create the extension as `postgres` on the target database, then rerun the restore. If `CREATE EXTENSION` itself fails with a missing-file error, the extension's packages are not present in the target image at all — it must be added to the image/instance before this migration can carry that database. + +### Restore reports `must be owner of extension pg_stat_statements` (and similar) + +`--no-comments` was omitted from `pg_dump`; the extension comments require superuser. The data restores fine — the errors only break the clean exit code. Rerun with `--no-comments` for a verifiable result. + +### Restore reports `unrecognized configuration parameter "transaction_timeout"` + +The dump was taken with a `pg_dump` newer than the target server (typically the image's default binary — a hand-run dump without the explicit `PG_BIN` path; the script pins `$PG_BIN` to the source major, which it verifies exists on both sides before starting). Rerun with the version-matched binary path. If the script's own pre-check fails instead (`pg_restore not present in the target image` — very old or custom images bundle fewer majors), run the dump and restore with a workstation-side PostgreSQL client of the matching major rather than the in-pod binaries. + +### A database reports `FAIL ... SOURCE CHANGED DURING DUMP` + +Source writes were not fully stopped: the script re-reads the source row counts after each transfer and refuses to PASS a database whose source moved mid-copy. Enforce the write stop (Step 2 — `CONNECTION LIMIT 0` plus `pg_terminate_backend`), then redo that database (`bash migrate.sh ` after dropping it on the target). + +### Restore reports `schema "metric_helpers" already exists` (and similar) + +The `-N metric_helpers -N user_management` exclusions were omitted. These errors are harmless for the excluded schemas' objects, but rerun with the exclusions for a clean, verifiable exit code. + +### Rerunning after a failed restore + +Drop the target database first — a partial restore leaves objects that collide (`relation already exists`, duplicate-key COPY failures). Note `DROP DATABASE` must be executed as its own single statement: + +```bash +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "DROP DATABASE appdb" +``` + +Then rerun the script with **only the failed database** as an argument (`bash migrate.sh `) — it recreates the database with the correct owner and locale; add `REUSE_DUMPS=1` to restore from the completed dump instead of re-dumping (only if source writes have stayed stopped). Do not rerun it without arguments after a partial success: restoring into the databases that already transferred produces `already exists` collisions that mark them as failed. + +### Application passwords changed after the migration + +The target CR was created **before** the credential secrets were pre-staged (Step 1), so the operator generated new passwords. To restore the source credentials after the fact, update both the secret and the role together — the secret alone is not enough, and an `ALTER ROLE` alone would be reverted by the operator's next sync from the secret. The commands below stay safe for passwords containing quotes, backslashes, or JSON metacharacters: the secret is patched with the base64 value (JSON-safe by construction), and the SQL uses psql's `:'pw'` literal quoting instead of string interpolation: + +```bash +PW_B64=$(kubectl --context $SRC_CTX -n $SRC_NS get secret \ + app-owner.$SRC_CLUSTER.credentials.postgresql.acid.zalan.do -o jsonpath='{.data.password}') + +kubectl --context $TGT_CTX -n $TGT_NS patch secret \ + app-owner.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + --type merge -p "{\"data\":{\"password\":\"$PW_B64\"}}" + +kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ + psql -U postgres -v pw="$(printf %s "$PW_B64" | base64 -d)" <<'SQL' +ALTER ROLE app_owner PASSWORD :'pw'; +SQL +``` + +Verify with a login test as in Step 4. + +### Transfer is slow + +Throughput is bounded by the workstation's link to both API servers (every byte traverses it twice: exec stream in, exec stream out). Run the migration from a machine with good connectivity to both platforms (e.g. a jump host), and keep the default file mode so progress is resumable per database. If a very large restore needs parallelism, note `pg_restore -j N` cannot read from stdin — copy the dump file into the target pod (`kubectl cp`) and restore from the local path there. If the measured rehearsal duration does not fit any acceptable window, this relay is the wrong tool for that instance — plan a network-connected migration path instead. diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md new file mode 100644 index 000000000..efb7e789f --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md @@ -0,0 +1,134 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515007 +--- + +# How to Migrate a PostgreSQL Instance to Another Node + +## Issue + +A PostgreSQL cluster managed by the PostgreSQL Operator uses node-local storage +(for example TopoLVM). One or more instances must be moved off their current +node — because the node is being decommissioned, or an instance must run on a +specific compute node. Because the data lives in a node-local PersistentVolume, +the pod cannot simply be rescheduled; the volume is pinned to its node. + +## Environment + +- Alauda Application Services PostgreSQL Operator (Zalando-based, + `acid.zalan.do/v1` `postgresql` resource). +- Node-local storage such as TopoLVM (each PVC is bound to one node). +- At least one target node with enough free capacity. Because migration + re-clones data, the target node should have capacity for roughly **twice** the + instance's PVC size during the transition. + +## Resolution + +The migration relies on a property of the Operator: when an instance's PVC and +pod are deleted, the StatefulSet recreates the pod, a fresh PVC is provisioned +wherever the pod is scheduled, and Patroni re-clones the data from the current +leader. Data is preserved through streaming replication, not by moving the +volume. + +> Validated on ACP 4.2 and 4.3: after deleting a replica's PVC and pod, the +> member was recreated on a node, re-synced from the leader, and previously +> written rows were present on the resynced member. + +In the examples below set `$NAMESPACE` and `$CLUSTER_NAME` for the target +cluster. Replace placeholder node names with your own. + +### 1. Confirm the current placement + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME +kubectl get pvc -n $NAMESPACE -o wide | grep $CLUSTER_NAME +``` + +Note which member is the leader (do not delete the leader's volume first): + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +### 2. Restrict scheduling to the target node(s) + +So the recreated pod lands only on a desired node, cordon the other eligible +nodes (or use a `nodeSelector`/label that only the target nodes carry). Keep at +least the target node schedulable. + +```bash +# Label ONLY the target node(s) so the recreated pod can land there and nowhere else +kubectl label node target=true --overwrite +``` + +Do **not** label the source node — labeling both source and target would let the +pod reschedule back onto the source. If you prefer cordoning over labels, cordon +all non-target nodes instead and skip the `nodeSelector` step below. + +If you use a label-based selector, set it on the instance: + +```bash +kubectl patch postgresql -n $NAMESPACE $CLUSTER_NAME --type merge \ + -p '{"spec":{"nodeSelector":{"target":"true"}}}' +``` + +### 3. Migrate one member at a time + +Always migrate a **non-leader** member first. Delete its PVC and pod together — +the PVC deletion blocks until the pod that mounts it is gone, so delete the pod +in parallel: + +```bash +# Delete the data PVC (it will stay in Terminating until the pod is gone) +kubectl delete pvc pgdata-$CLUSTER_NAME-1 -n $NAMESPACE --wait=false + +# Delete the pod to release the PVC +kubectl delete pod $CLUSTER_NAME-1 -n $NAMESPACE +``` + +The StatefulSet recreates `$CLUSTER_NAME-1`; a new PVC is provisioned on the +scheduled node and Patroni re-clones from the leader. + +### 4. Verify the member rejoined with its data + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME-1 +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +The migrated member should return to role `Replica`, state `running`/`streaming` +with `Lag in MB` `0`. Spot-check data on the member (it is read-only / in +recovery): + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-1 -c postgres -- \ + psql -U postgres -tAc "SELECT pg_is_in_recovery();" # expect t +``` + +### 5. Migrate the (former) leader if required + +To move the leader, first perform a switchover so another member becomes leader, +then repeat steps 3–4 for the old leader: + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- \ + patronictl switchover $CLUSTER_NAME --force +``` + +### 6. Restore scheduling + +Uncordon any nodes you cordoned and remove temporary labels/`nodeSelector` once +all members are on their intended nodes. + +## Notes + +- Migrate members one at a time and wait for each to fully re-sync (`Lag = 0`) + before moving the next, so the cluster always retains a healthy quorum. +- For a single-instance cluster, temporarily scale to two instances, let the new + member sync on the target node, switch over, then scale back to one — this + avoids downtime that a delete-and-reclone of the sole instance would cause.