Cluster lifecycle architecture

This document explains how cnmsql reconciles a Cluster into running Percona Server for MySQL instances. The operator follows the CloudNativePG pattern: Kubernetes owns the desired state, while each database pod runs an instance manager that owns local mysqld lifecycle and reports database state back to the operator.

cnmsql does not use StatefulSets. It creates one Pod and one PVC per instance so it can control cloning, promotion, fencing, retained storage, and recovery explicitly.

Cluster shape

A fresh cluster usually defines an image, an instance count, storage, and an initdb bootstrap:

apiVersion: mysql.cnmsql.co/v1alpha1
kind: Cluster
metadata:
  name: cluster-sample
spec:
  instances: 3
  imageName: ghcr.io/cnmsql/cnmsql-instance:8.4
  storage:
    size: 10Gi
  mysql:
    binlogFormat: ROW
  bootstrap:
    initdb:
      database: app
      owner: app

spec.imageName selects the exact instance image. Alternatively, an imageCatalogRef can resolve an image by MySQL series. cnmsql is built for Percona Server for MySQL; the instance image includes mysqld, XtraBackup, the manager binary, and the small tool set needed for backup and recovery.

Reconciled resources

For each instance, the operator reconciles stable Kubernetes objects with predictable names:

• Pod: <cluster>-1, <cluster>-2, and so on.
• PVC: one data volume per instance, retained during scale-down. See Storage for configuration and resizing.
• Bootstrap Job: one <instance>-<mode> Job per instance volume, which runs before the instance Pod exists (see Instance bootstrap).
• Headless per-instance Service: stable DNS for instance-to-instance traffic.
• Secrets: root, application, replication, backup, and control credentials when the user does not provide them.
• TLS material: cert-manager issuers/certificates for manager mTLS and MySQL TLS.
• Role Services: <cluster>-rw, <cluster>-ro, and <cluster>-r.

The operator labels owned resources with the cluster and instance identity. Role labels are dynamic: the current primary receives role=primary, and the other ready instances receive role=replica.

Instance bootstrap

An instance's data volume comes up in a fixed order: the operator creates the PVC, bootstraps it with a one-shot Job, and only then creates the instance Pod. The Job is named <instance>-<mode> — initdb (fresh data directory), restore (physical recovery and PITR), join (replica clone), or import (initdb then logical load) — mounts the volume, and runs the same manager instance … command under the instance's ServiceAccount. It follows the instance's scheduling so the volume binds where the Pod can run, and the Job and the Pod never run at the same time.

The PVC annotation mysql.cnmsql.co/pvc-status records the volume's state: initializing on creation, ready once the Job succeeded and the operator deleted it. The cluster waits for a running Job in phase Pending, with a reason like "Waiting for bootstrap Job cluster-1 (restore) of cluster-1".

A failed Job stays for inspection and is surfaced on the Cluster:

• the BootstrapFailed condition turns True with the Job's own reason (for example BackoffLimitExceeded or DeadlineExceeded);
• the phase becomes Blocked (or Degraded when a replica's join fails on an established cluster), with a reason naming every failed Job;
• a BootstrapJobFailed Warning Event records the transition.

The Job is replaced only when the spec it was built from changes — a different recovery source, a longer deadline. To retry with the same spec, delete it by hand:

kubectl logs job/<instance>-<mode>
kubectl delete job <instance>-<mode>

Re-initialising an instance deletes its bootstrap Jobs together with its Pod and PVC. Scaling an instance down deletes its bootstrap Jobs — running ones included, even when its Pod was never created — and deletes its volume only while it is still initializing; a bootstrapped volume is retained.

Bootstrap modes

The first instance can start in one of two supported ways.

bootstrap.initdb creates a new MySQL data directory, initializes the root and application users, creates the application database, and applies optional post-init SQL.

bootstrap.recovery.backup restores from a Backup object into an empty PVC; bootstrap.recovery.source restores directly from an object-store bucket by referencing an externalClusters entry. A raw-S3 recovery discovers the latest or named (backupID) base backup in the destination, needing no source Cluster or Backup CR to exist. When a recovery target is present, PITR planning and binlog replay run before the recovered mysqld is started as the new primary.

Replicas do not run initdb. They join by pulling an XtraBackup stream from the current primary over the instance-manager mTLS endpoint, preparing it, and configuring GTID replication.

Instance manager

Each Pod runs a cnmsql instance manager as PID 1. It is responsible for:

• rendering version-aware MySQL configuration;
• starting and stopping mysqld cleanly;
• bootstrapping the data directory (initdb, restore, join, import) in the instance's one-shot bootstrap Job;
• exposing an mTLS control API for status and backup streaming;
• running an in-pod role reconciler that promotes or follows based on Cluster status;
• running the binlog archiver when continuous archiving is enabled.

The manager uses MySQL's admin interface where available so control operations do not get locked out by application connection pressure. Older versions fall back to local socket access and reserved privileges.

Configuration surface

cnmsql renders the managed MySQL configuration and lets users add safe MySQL settings through:

spec:
  mysql:
    parameters:
      require_secure_transport: "ON"
      max_connections: "500"
    binlogFormat: ROW
    semiSync:
      enabled: true
      timeoutMillis: 1000

The operator owns settings required for replication, backup, PITR, and lifecycle control. User parameters are applied under the mysqld section, but managed keys are protected by the renderer.

Scheduling and pod shape are controlled through the Cluster spec: resources, affinity, topologySpreadConstraints, priorityClassName, schedulerName, imagePullPolicy, imagePullSecrets, env, envFrom, podSecurityContext, and securityContext.

TLS material is generated through cert-manager unless you provide Secret names under spec.certificates:

spec:
  certificates:
    serverCASecret: my-server-ca
    serverTLSSecret: my-server-tls
    clientCASecret: my-client-ca
    replicationTLSSecret: my-replication-tls

Partial overrides are allowed. For example, setting only serverTLSSecret reuses your server certificate while cnmsql still generates the CA issuer and operator client certificate.

Status model

During reconciliation and periodic resyncs, the operator queries each instance-manager /status endpoint over mTLS and combines that with Kubernetes Pod readiness before writing the observed topology into Cluster.status.

Important fields include:

• instances and readyInstances;
• instanceNames;
• currentPrimary and targetPrimary;
• currentPrimaryTimestamp;
• gtidExecutedByInstance;
• continuousArchiving;
• phase, phaseReason, and Kubernetes conditions.

Ready=True means the desired topology is available. Progressing=True means the cluster is still creating, cloning, restoring, or changing primary. Degraded=True surfaces a failure that needs operator attention. Two examples are a Pod stuck failing to start (failedInstances) and a reachable replica whose replication has aborted with a recorded error (replicationBrokenInstances). This holds even before the cluster first finishes provisioning, so a replica that comes up but cannot replicate is reported instead of looking like it is still bootstrapping.

Scale behavior

Scale-up is ordered. The operator creates one new replica at a time and waits for it to become healthy before adding the next one. This bounds load on the primary and makes failures easier to diagnose.

Scale-down removes highest-ordinal replicas first. The Pod is deleted, but the PVC is retained so the user can inspect or delete data deliberately. A volume that never finished bootstrapping (annotated initializing) holds no data and is deleted together with the instance's bootstrap Jobs — running ones included, also when the Pod was never created, as when a scale-down races a join Job. cnmsql never scales below one instance and never removes the current primary as part of ordinary scale-down. When the primary's ordinal is above the new count, the operator performs a planned switchover to an in-range replica first and removes the former primary afterwards; the cluster is not Ready until it has.

A later scale-up reuses a retained PVC as it is: the instance starts from the data it held when it was removed and catches up from the primary's binary logs. By default it is Ready, and in the -ro and -r Services, as soon as its replication threads run, so it can serve reads that are as old as the volume until it has caught up. Set spec.replication.maxReadyLag to hold it out of the read Services until it has, or delete the PVC before scaling up to clone the replica afresh.

Operational notes

• Use at least three instances when relying on automatic failover.
• Keep binlogFormat: ROW for replication and PITR safety.
• Set require_secure_transport yourself if applications must be forced onto TLS; cnmsql renders TLS material but does not force the setting by default.
• Treat retained PVCs after scale-down or failed rejoin as database data, not disposable scratch space.
• Watch Events and conditions during bootstrap. Unsupported shapes are reported clearly instead of creating partial resources.

Verification coverage

The controller has unit coverage for supported and unsupported cluster shapes, image resolution, generated secrets, Pod/PVC templates, status transitions, role labels, services, scale-up, scale-down, and recovery wiring. The Kind e2e suite validates single-instance bootstrap, replicated clusters, role services, switchover, failover, backup, restore, binlog archiving, and PITR flows.