Skip to main content

Backup retention and deletion

cnmsql currently separates Kubernetes object lifecycle from object-store artifact lifecycle. This is deliberate: a Kubernetes object deletion should not silently destroy the only copy of a recovery point unless the user explicitly opts into that behavior.

What happens today​

By default, deleting a Backup object does not delete remote S3-compatible objects.

The remote objects remain under:

<path>/<cluster>/<backup-name>/<backup-id>/backup.xbstream
<path>/<cluster>/<backup-name>/<backup-id>/metadata.json

The Kubernetes Job is owned by the Backup, so Kubernetes garbage collection may remove the Job. Object-store artifacts are not owned by Kubernetes and are not removed unless you opt the Backup into remote cleanup with reclaimPolicy: Delete, described below.

ScheduledBackup owner references​

ScheduledBackup.spec.backupOwnerReference controls only Kubernetes owner references on generated Backup objects:

  • self: generated Backups are owned by the ScheduledBackup.
  • cluster: generated Backups are owned by the Cluster.
  • none: generated Backups are standalone.

These modes do not by themselves change S3 deletion behavior. To also reclaim a generated Backup's remote objects when it is deleted, set the schedule's spec.reclaimPolicy: Delete (see the reclaim policy section below); otherwise a garbage-collected Backup leaves its remote objects behind.

Why remote cleanup is not automatic​

Remote backup cleanup is data-destructive. A Backup object may be deleted accidentally, by namespace cleanup, by owner-reference cascade, or by a GitOps prune. Automatically deleting backup.xbstream and metadata.json in those cases could destroy the recovery window.

cnmsql therefore keeps reclaimPolicy: Retain as the default and never removes remote artifacts unless you opt in.

Opt-in reclaim policy for remote cleanup​

Set spec.reclaimPolicy: Delete on a Backup to reclaim its archive from the object store when the Backup object is deleted:

apiVersion: mysql.cnmsql.co/v1alpha1
kind: Backup
metadata:
  name: my-backup
spec:
  cluster:
    name: my-cluster
  reclaimPolicy: Delete

The operator then manages the mysql.cnmsql.co/cleanup-backup-files finalizer for you. Reconciling a Delete-policy Backup stamps the finalizer; flipping the policy back to Retain strips it. On deletion:

  1. The operator resolves the destination object store. It prefers the store snapshotted onto status.objectStore at backup time, then spec.objectStore, then the referenced Cluster's spec.backup.objectStore. The snapshot means cleanup still works after the source Cluster is gone.
  2. It deletes the backup's archive directory wholesale: both backup.xbstream and metadata.json.
  3. Only then is the finalizer removed and the Kubernetes object allowed to go away. A cleanup failure requeues the deletion (and emits a Warning CleanupFailed event) rather than silently leaving half-cleaned state.
  4. If the object store still cannot be resolved (no snapshot, no spec override, and the Cluster is gone), cleanup is skipped so the finalizer never wedges the object's deletion.

For Backups generated by a ScheduledBackup, set spec.reclaimPolicy: Delete on the schedule. Every generated Backup inherits the policy and the operator stamps the finalizer, so deleting one also reclaims its archive:

apiVersion: mysql.cnmsql.co/v1alpha1
kind: ScheduledBackup
metadata:
  name: nightly
spec:
  schedule: "0 0 2 * * *"
  cluster:
    name: my-cluster
  reclaimPolicy: Delete

reclaimPolicy defaults to Retain everywhere, so existing Backups and schedules keep the non-destructive default.

To delete a Delete-policy Backup object but keep its remote archive, first patch the policy back to Retain (which drops the finalizer), then delete the object.

Reclaiming a Cluster's whole archive on teardown​

The per-Backup policy only covers base backups that still have a Backup object. It does not cover the continuous binlog archive, and it does not help when you delete a whole Cluster and its object store keeps accruing charges. For that, set spec.backup.reclaimPolicy: Delete on the Cluster:

spec:
  backup:
    reclaimPolicy: Delete
    objectStore:
      bucket: my-backups
      # ...

When set, the operator adds a mysql.cnmsql.co/cleanup-object-store finalizer to the Cluster. On Cluster deletion it wipes the cluster's entire archive prefix (every base backup, the archived binlogs, and the archive index) before releasing the finalizer. As with the Backup path, a cleanup failure requeues and emits a CleanupFailed event; if the object store is permanently unreachable, an operator can remove the finalizer by hand to force teardown.

This is destructive and defaults to Retain, so deleting a Cluster leaves its archive intact unless you opt in. If the credential Secret is deleted before the finalizer runs (for example a fast namespace teardown), cleanup cannot authenticate and the Cluster stays in Terminating until the Secret returns or the finalizer is removed manually.

Retention GC​

Set spec.backup.retentionPolicy on a Cluster to have the operator expire old archives automatically. The value is a time window: <n>d, <n>w, or <n>m (days, weeks, months, where a month is 30 days):

spec:
  backup:
    retentionPolicy: 30d
    objectStore:
      bucket: my-backups
      # ...

A retentionPolicy requires an object store; setting one without spec.backup.objectStore is rejected by validation.

What gets deleted​

The operator runs a throttled retention pass (at most once per hour, tracked in status.lastRetentionRunTime) on clusters that have a policy, an object store, and an established primary. Each pass:

  1. Expires old base backups. A base backup is deletable when its completedAt is older than now - window. Its whole archive directory (backup.xbstream + metadata.json) is removed.
  2. Always keeps the newest base backup as a floor, even if it is older than the window. A cluster must always have something to recover from. So the deletable set is {expired} \ {newest}.
  3. Expires uncoverable binlog segments. The PITR horizon is the oldest retained base backup's start time. Binlog segments whose last event predates that horizon can no longer be replayed onto any retained base, so they are deleted and _index.json is rewritten to match.

Binlog GC is conservative: a segment with an unknown (zero) last-event time is kept rather than risk shortening the PITR window. The index is rewritten last, so a mid-run failure leaves a still-valid index and orphans are cleaned on the next pass.

A successful pass that deletes anything emits a Normal BackupRetention event. Transient object-store errors requeue the reconcile rather than corrupting the archive.

What it does not do​

  • It does not delete Backup Kubernetes objects, only object-store artifacts. Pruning expired Backup CRs is the scheduler's job; see ScheduledBackup retention below.
  • It is purely time-based. Count-based bounds live on the ScheduledBackup, not on the Cluster policy.

ScheduledBackup retention (Backup objects)​

The Cluster retentionPolicy above bounds object-store bytes. It says nothing about the Backup Kubernetes objects a ScheduledBackup creates, which otherwise accumulate one-per-slot forever. A ScheduledBackup bounds its own Backup objects with three opt-in knobs:

apiVersion: mysql.cnmsql.co/v1alpha1
kind: ScheduledBackup
metadata:
  name: nightly
spec:
  schedule: "0 0 2 * * *"
  cluster:
    name: my-cluster
  successfulBackupsHistoryLimit: 7   # keep the 7 newest completed Backups
  failedBackupsHistoryLimit: 3       # keep the 3 newest failed Backups
  retentionPolicy: 30d               # and drop any terminal Backup older than 30d
  • Count (successfulBackupsHistoryLimit / failedBackupsHistoryLimit): keep the newest N completed / failed Backups; delete older ones of that phase.
  • Time (retentionPolicy): same <n>d/w/m syntax as the Cluster policy; delete terminal Backups older than the window.

A Backup is garbage-collected when it exceeds a count limit or ages past the window. Only terminal Backups (completed/failed) are eligible; pending and running are never touched. The newest completed Backup is always kept as a floor. Every knob is opt-in; an unset knob disables that axis.

How the two layers fit together​

  • Cluster spec.backup.retentionPolicy bounds the object store: expired base backups and uncoverable binlogs are removed. This is the only thing that reclaims the continuous binlog archive.
  • ScheduledBackup history limits / retentionPolicy bound the number/age of the Backup objects a schedule owns.
  • reclaimPolicy links the two: ScheduledBackup GC deletes only the Backup object, but a Backup with reclaimPolicy: Delete reclaims its archive on delete via the cleanup finalizer. So a Delete-policy schedule can drive both layers from its own history limits; a Retain-policy schedule prunes objects while the Cluster policy reclaims the bytes.

What operators should do now​

  • Use object-store lifecycle rules carefully, and align them with the recovery window you need.
  • Keep Backup objects for important restore points so their status remains easy to inspect.
  • Preserve both backup.xbstream and metadata.json.
  • Test recovery before deleting old prefixes manually.
  • Document any external cleanup automation outside cnmsql.

Manual cleanup checklist​

Before deleting remote backup data:

  • Confirm no Cluster uses bootstrap.recovery.backup or bootstrap.recovery.source for that Backup.
  • Confirm no runbook references the backup ID.
  • Confirm a newer base backup exists and is restorable.
  • Confirm PITR archive coverage still satisfies the required recovery window.
  • Delete both the archive and metadata object together.