Backup and restore

Kuben keeps its whole desired state in PostgreSQL: projects, environments, apps, releases, runs, users, policies and encrypted secrets. The Kubernetes objects are rendered from it, so backing up the database backs up Kuben.

State	Back up with
The database	kuben backup (scheduled by kuben setup and the Helm chart)
The secret keyring	Separately, once and after every key rotation
App data (volumes)	VolumeSnapshots or Velero, on the schedule your data needs

Scheduled backups

kuben setup installs kuben-backup.timer (daily at about 03:17); the Helm chart runs the …-backup CronJob (backup.schedule) and writes to a kept volume. Both keep the newest backup.keep backups (default 7) under backup.dir. Each backup is a directory kuben-<UTC time> with:

• database.dump: a pg_dump custom archive of the database;
• manifest.json: the Kuben version, the schema, the dump’s size and SHA-256, and the fingerprints of the secret keys it was sealed with.

The server reports the subsystem backups as degraded, and kuben doctor warns, when the newest good backup is older than backup.max_age_hours (default 26). Every attempt, good or failed, is recorded in the database.

Copy backups off the host

A backup on the same disk or volume as the database does not survive the loss of that disk. Sync backup.dir (or the chart’s backup volume) to object storage or another site.

By hand

kuben backup                     # into backup.dir
kuben backup --out /mnt/offsite  # anywhere else

kuben backup needs PostgreSQL’s client tools (pg_dump) at least as new as the server. It never changes the database schema.

The secret keyring

Secret values are sealed with keys from the keyring (secrets.keyring_file, by default secrets.keyring in the state directory; in the chart the Secret …-secrets-keyring, kept on uninstall). The keyring never enters the database or its backups, so a stolen backup does not reveal secrets.

Back the keyring up separately. kuben backup --include-keyring copies it into the backup for convenience, and whoever holds such a backup can read every secret: encrypt it.

Point-in-time recovery

kuben backup restores to the moment of the backup. For a smaller recovery point, archive PostgreSQL’s WAL with pgBackRest, WAL-G or your provider’s point-in-time recovery, and restore the database with that tool. kuben doctor tells whether WAL archiving is on. After such a restore, run the same fencing as kuben restore does (below) by restoring through it, or at least sign everyone out and revoke API tokens.

Restore

kuben restore --from /backups/kuben-20260917T031700Z --check   # intact?
kuben restore --from /backups/kuben-20260917T031700Z

The target database must be empty. A restore:

1. checks the manifest and the dump’s checksum, and that this Kuben is at least as new as the backup;
2. finds the keyring (the configured one, or the backup’s, installed in its place) and refuses keys that are not the backup’s;
3. loads the dump with pg_restore and applies newer migrations;
4. fences the time after the backup: every session ends and every API token is revoked (sign in again and issue new tokens), and every app’s generation jumps far ahead, so anything a cluster or agent saw after the backup is outranked instead of mistaken for newer state;
5. starts a run for every app, which writes it again, to the old cluster or a new one.

It prints how much time lies between the backup and the restore: changes made in that window are not in the restored database.

Before an upgrade

Run kuben backup by hand and check it with kuben restore --check. See Upgrading.