Skip to main content

Physical backup and recovery architecture

cnmsql physical backups are Percona XtraBackup streams stored in an S3-compatible object store. A Backup object is immutable once completed: it selects a source instance, creates a worker Job, uploads a single xbstream archive plus metadata, and records enough status for a future Cluster recovery.

PITR builds on this base-backup mechanism. This page focuses on the physical backup and restore-to-backup-point path.

Physical or logical

A physical backup copies the data directory: it is the base for disaster recovery and point-in-time recovery, and it restores onto the same server series. A logical backup (method: logical) is a SQL dump of the application schemas: use it to export single databases, restore them into a new or a running cluster, or move to another series. It is never a recovery base.

Object store configuration​

Backups need an S3-compatible object store on the source Cluster or directly on the Backup object:

spec:
  backup:
    objectStore:
      bucket: cnmsql-backups
      path: production
      endpoint: http://seaweedfs.objectstore.svc:8333
      region: us-east-1
      forcePathStyle: true
      credentials:
        accessKeyId:
          name: objectstore-creds
          key: accessKey
        secretAccessKey:
          name: objectstore-creds
          key: secretKey

When Backup.spec.objectStore is omitted, the backup inherits Cluster.spec.backup.objectStore. The object store supports custom endpoints, path-style addressing, signature v2 or v4, TLS CA bundles, insecure test endpoints, server-side encryption, storage class, static credentials, and IAM role inheritance.

Creating a backup​

A one-shot Backup references its Cluster. Use the plugin for a quick on-demand backup with sensible defaults:

kubectl cnmsql backup cluster-sample

This creates a Backup object with xtrabackup method, prefer-standby target, and online mode. The Backup reconciler runs the XtraBackup-to-object-store data path.

You can also define a Backup by hand when you need more control:

apiVersion: mysql.cnmsql.co/v1alpha1
kind: Backup
metadata:
  name: backup-sample
spec:
  cluster:
    name: cluster-sample
  method: xtrabackup
  target: prefer-standby
  online: true

target: prefer-standby takes the backup from a healthy replica when one is available and falls back to the primary. target: primary requires the current primary.

The reconciler creates a Kubernetes Job owned by the Backup. The Job uses the same cnmsql instance image as the Cluster so the XtraBackup version matches the server version. Object-store credentials are mounted into the short-lived Job, not into the long-running database Pods.

Shaping the worker Job​

You can shape the worker Job's pod with spec.backup.jobTemplate on the Cluster (the default for every backup) or spec.jobTemplate on an individual Backup, which overrides the cluster default field by field. The template is a curated subset of the pod spec, not a full PodTemplateSpec: the operator keeps ownership of the bootstrap init container, the scratch and TLS volumes and mounts, the worker command and args, and the object-store credential env, so a template cannot break them. A ScheduledBackup carries a spec.jobTemplate too and propagates it to every generated Backup.

The template exposes:

  • resources: requests and limits for the worker container. Streaming xbstream can be memory-hungry. During recovery the cluster-level template's resources also size the <instance>-restore bootstrap Job (the other bootstrap Jobs use the instance's spec.resources).
  • nodeSelector, tolerations, affinity: keep backups off the critical nodes. On instance bootstrap Jobs the template's nodeSelector and affinity are not applied — they follow the instance's scheduling — while tolerations are added to the instance's.
  • priorityClassName: the pod priority for the worker Job; on instance bootstrap Jobs it overrides the instance's.
  • labels, annotations: merged onto the generated Job and its pod. Operator labels win on conflict.
  • ttl: how long a finished worker Job is kept before Kubernetes garbage-collects it (ttlSecondsAfterFinished). Defaults to 24h so you can inspect its logs. A zero duration (0s) deletes the Job as soon as it finishes.
  • activeDeadline: how long the worker Job may run before Kubernetes kills it (activeDeadlineSeconds), failing the Backup with reason DeadlineExceeded instead of leaving a stalled upload running forever. Defaults to 24h; raise it for very large databases. A zero duration (0s) disables the deadline.
apiVersion: mysql.cnmsql.co/v1alpha1
kind: Cluster
spec:
  backup:
    jobTemplate:
      ttl: 1h
      activeDeadline: 6h
      priorityClassName: batch-low
      nodeSelector:
        node-role.kubernetes.io/backup: ""
      resources:
        requests:
          memory: 512Mi
        limits:
          memory: 2Gi

Backup data path​

The worker Job:

  1. resolves the selected source instance;
  2. connects to its instance-manager endpoint over mTLS;
  3. requests GET /cluster/backup;
  4. streams XtraBackup stdout directly to the object store;
  5. computes SHA256 while uploading;
  6. uploads metadata.json;
  7. exits and lets the controller mirror Job outcome into Backup status.

The backup stream itself is data and is not logged. Child process stderr is captured as structured logs.

Object layout​

cnmsql uses deterministic, inspectable object keys:

<path>/<cluster>/<backup-name>/<backup-id>/backup.xbstream
<path>/<cluster>/<backup-name>/<backup-id>/metadata.json

metadata.json includes cluster identity, Backup identity, source instance, method, backup ID, object key, timing, size, checksum, and GTID/binlog metadata when available.

SHA256 in cnmsql metadata is the integrity source of truth. S3 ETag may be useful provider metadata, but it is not used as the backup checksum.

Backup status​

The Backup status reports:

  • phase: pending, running, completed, or failed;
  • instanceName: selected backup source;
  • method;
  • backupId;
  • jobName;
  • destinationPath;
  • sha256;
  • beginGTID and endGTID;
  • beginBinlog and endBinlog;
  • startedAt and stoppedAt;
  • error;
  • conditions.

A completed Backup is not rerun. To take another backup, create another Backup object or use a ScheduledBackup.

Restore from a backup​

A new Cluster can recover from a completed Backup:

apiVersion: mysql.cnmsql.co/v1alpha1
kind: Cluster
metadata:
  name: restored-cluster
spec:
  instances: 3
  imageName: ghcr.io/cnmsql/cnmsql-instance:8.4
  storage:
    size: 10Gi
  bootstrap:
    recovery:
      backup:
        name: backup-sample
  backup:
    objectStore:
      bucket: cnmsql-backups
      path: production
      endpoint: http://seaweedfs.objectstore.svc:8333
      credentials:
        accessKeyId:
          name: objectstore-creds
          key: accessKey
        secretAccessKey:
          name: objectstore-creds
          key: secretKey

The primary's volume is bootstrapped by its one-shot <instance>-restore bootstrap Job, which downloads backup.xbstream, verifies the checksum, extracts it, and runs XtraBackup prepare and copy-back into the data directory. The operator then creates the first instance's Pod, which starts as the recovered primary. Additional replicas clone from that recovered primary through the normal replica join path.

The recovery source is resolved only until the bootstrap primary's volume is bootstrapped. After that the source Backup, its object store and its Secrets can be deleted without affecting the recovered cluster.

Without recoveryTarget, the Cluster restores to the backup's consistent point. With recoveryTarget, the PITR path replays archived binlogs after the base backup restore.

Restore from raw object store (no Backup CR)​

A Cluster can also recover directly from an object-store bucket without any Backup object existing in the API server. This is the disaster-recovery path: the source cluster's API server (and its Backup CRs) may be gone, GC'd by retention, or in another cluster entirely. Recovery is driven entirely by the objects already in S3.

Point bootstrap.recovery.source at an externalClusters entry. The entry carries its own objectStore (bucket, path, credentials) and its name is the S3 key prefix the backups were stored under. No source Cluster or Backup CR needs to exist anywhere.

apiVersion: mysql.cnmsql.co/v1alpha1
kind: Cluster
metadata:
  name: recovered-cluster
spec:
  instances: 3
  imageName: ghcr.io/cnmsql/cnmsql-instance:8.4
  storage:
    size: 10Gi
  bootstrap:
    recovery:
      source: prod-cluster          # externalClusters entry name = S3 key prefix
      backupID: ""                   # empty = latest; set to pin a specific backup
      recoveryTarget:                # optional PITR, identical to the Backup path
        targetGTID: "uuid:1-99"
  externalClusters:
    - name: prod-cluster
      objectStore:
        bucket: cnmsql-backups
        path: production
        endpoint: http://seaweedfs.objectstore.svc:8333
        credentials:
          accessKeyId:
            name: objectstore-creds
            key: accessKey
          secretAccessKey:
            name: objectstore-creds
            key: secretKey

The operator lists the base backups under the prefix (logical backups in the same prefix are ignored, they carry a different manifest), selects the latest completed one (or the entry matching backupID when set), derives the archive and metadata keys, and restores exactly as the Backup-based path does. source and backup are mutually exclusive. PITR with recoveryTarget works identically: the binlog archive is resolved from the same object store under the source name.

Failure surfaces​

Common failure points are reported through Backup phase, Backup conditions, Job status, and Events:

  • missing referenced Cluster;
  • missing object-store configuration;
  • unsupported backup method;
  • no healthy source instance;
  • mTLS connection failure to the source instance manager;
  • XtraBackup failure;
  • object-store upload/download failure;
  • checksum mismatch;
  • failed restore prepare or copy-back;
  • a stalled stream: the worker fails the Backup after five minutes without progress instead of waiting for the Job's activeDeadline, with reason SourceStalled when the source instance stopped sending (XtraBackup hung, for example behind a lock) or ObjectStoreStalled when the object store stopped accepting bytes. A restore download that receives nothing for five minutes fails the same way;
  • an archive too large for its upload: the worker sizes its multipart parts so that twice the data volume size (spec.storage.size) fits in the 10,000-part S3 limit, with 64MiB parts at least. A stream that still outgrows that fails the Backup with reason ArchiveTooLarge and removes the short object, instead of recording a truncated archive as complete. Grow spec.storage.size to raise the limit;
  • raw-S3 recovery: source does not name an externalClusters entry;
  • raw-S3 recovery: the referenced external cluster entry has no objectStore;
  • raw-S3 recovery: no base backups found under the source prefix;
  • raw-S3 recovery: the requested backupID is not present in the object store;
  • bootstrap.recovery.backup names a logical Backup: the Cluster is blocked with LogicalBackupNotRecoverable, because recovery restores a physical data directory.

The controller-manager never handles backup payload bytes. Large data movement stays in Jobs so retries are isolated and observable through Kubernetes primitives.

Operational notes​

  • A recovery cluster reads its source Backup only until its primary's volume is bootstrapped; afterwards the Backup, its object store and its Secrets can be deleted. Keep completed Backup objects while other clusters may still need to recover from them.
  • Preserve both backup.xbstream and metadata.json; recovery needs metadata as well as bytes.
  • Use the same major-version-compatible cnmsql instance image for recovery.
  • Prefer standby backups for large clusters when replicas can absorb the backup load.
  • Object-store retention and immutability are external responsibilities until cnmsql retention GC is implemented.
  • Physical backup alone restores only to the backup consistency point. Enable continuous archiving when PITR is required.

Verification coverage​

Unit tests cover object-store key construction, checksum helpers, backup target selection, Job rendering, status transitions, and restore command behavior. The Kind + SeaweedFS e2e suite validates backup upload, metadata writing, restore into a new Cluster, and recovered data correctness.