Upgrading

Helm

helm upgrade kuben oci://ghcr.io/teamtem-dev/charts/kuben -n kuben-system --reuse-values

Without --version, Helm takes the newest stable chart; pre-releases are only installed when you name them. Add --version 1.2.1 to pin a chart version; the chart’s appVersion selects the image tag.

Two things that usually need care with operators are automatic here:

CRDs. Helm does not update the crds/ folder on upgrade. The binary therefore server-side-applies its own CRDs on every start, so new fields appear as soon as the new pod runs. Downgrading leaves newer CRD fields in place; they are ignored by older binaries.
Migrations. Database migrations run at boot, under a lock, so several replicas starting together do not race.
Preflight and backup. A pre-upgrade hook runs with the new image: it takes a backup, then kuben upgrade-check. If the check fails, Helm stops before anything changed (backup.beforeUpgrade).

The upgrade is a rolling update with maxUnavailable: 0 and a leader hand-over: the controllers pause for a few seconds, the API not at all; see High availability.

What an upgrade checks

kuben upgrade-check is run by the new version against the installation it is about to take over, and changes nothing. It fails when:

the step is not supported: Kuben goes forward one minor version at a time (patch releases in between do not count); a major step needs --major after its release notes; going back is never supported;
the database was migrated by a newer Kuben, or a migration failed half-way;
migrations are pending and there is no backup from the last 24 hours;
a linked cluster agent speaks a protocol the new version no longer does (the hub speaks its own protocol version and the one before it, so agents are upgraded after the hub, one version behind at most).

It warns about operations in flight (they resume after the upgrade) and little free disk space.

Every start records its version, and every migration run is journaled as running, then succeeded or failed. An upgrade interrupted half-way resumes forward on the next start: each migration is a transaction of its own and is never applied twice. A binary older than the database’s schema refuses to start instead of running against it.

From 1.0 and 1.1

Kuben 1.0 and 1.1 kept their data in SQLite. Since 1.2, Kuben keeps it in PostgreSQL only and does not open SQLite, and the data of an earlier install is not carried over: a 1.0 or 1.1 install starts again with an empty database.

Helm. The persistence.* values and the SQLite volume <fullname>-data are gone from the chart; it runs its own PostgreSQL unless database.url or database.existingSecret is set. An upgrade of a release that still has the volume <fullname>-data (kuben-data for a release named kuben) stops with a message. Set legacy.sqliteVolumeAcknowledged=true to start with a new, empty database instead; the old volume stays untouched.
Server. kuben setup stops when /etc/kuben/config.toml still names a SQLite database. Delete that file and run it again: setup then writes one for the server’s own PostgreSQL.

Upgrading from 1.2 to 2.0

What changes

Kuben 2.0 is the same product with its server, CLI and cluster agent rewritten in Go. What an installation relies on stays as 1.2 left it:

The database schema. 2.0.0 adds no migration: it runs on the database 1.2 migrated, and 1.2.x runs on it again after a rollback. The first new migration comes with 2.1.
The REST API (/api/v1, the same OpenAPI document), the custom resources (the same CRD manifest), the Helm values, the configuration keys and KUBEN_* variables, and the commands and flags of kuben.
The release files: the archive names, checksums.txt and its signature, the image and the chart, so install.sh and your automation keep working.
[runtime] keys (worker_threads, max_blocking_threads, bulkhead) sized the Tokio thread pools of 1.x. They are still accepted and have no effect; see Configuration.
Output scripts read: kuben --version prints kuben <version> as before, for example kuben 2.0.0-alpha.1; kuben version adds the platform, kuben 2.0.0-alpha.1 (linux x86_64). A command line that does not parse still exits with 2, a failed command with 1.

Upgrade the hub first and its cluster agents after it: the hub speaks its own AgentLink protocol version and the one before it.

Check first

2.0 is a major step, so kuben upgrade-check of the 2.0 binary refuses it without --major. Read the release notes, download the 2.0 archive for your platform from the releases page, and run its binary against your installation before it takes over. On a server set up by kuben setup, run it as the service user so it can reach the database:

tar -xzf kuben-x86_64-unknown-linux-musl.tar.gz kuben
sudo install -m 0755 kuben /tmp/kuben-2.0
sudo -u kuben /tmp/kuben-2.0 upgrade-check --major

It also fails when migrations are pending without a recent backup, when the database is ahead or half-migrated, or when a linked agent speaks a protocol 2.0 no longer does (What an upgrade checks).

Try it on a server

install.sh takes the version as --version or KUBEN_VERSION:

curl -fsSL https://kuben.teamtem.com/install.sh | sh -s -- --version v2.0.0-alpha.1
# or
curl -fsSL https://kuben.teamtem.com/install.sh | KUBEN_VERSION=v2.0.0-alpha.1 sh

It downloads the archive, checks it against checksums.txt (and its signature when cosign is installed) and runs kuben setup, which replaces the binary and restarts kuben.service. Take a backup first with kuben backup.

Try it with Helm

Name the chart version: Helm does not pick a pre-release on its own. The chart’s pre-upgrade hook takes a backup and runs the 2.0 binary’s kuben upgrade-check, which refuses a major step unless told the release notes were read: set backup.upgradeCheckMajor=true for this one upgrade.

helm upgrade kuben oci://ghcr.io/teamtem-dev/charts/kuben -n kuben-system --reuse-values \
  --version 2.0.0-alpha.1 --set backup.upgradeCheckMajor=true

If the check fails, Helm stops before anything changed and the hook Job’s log says why (kubectl -n kuben-system logs job/kuben-backup-pre-upgrade). Set backup.upgradeCheckMajor=false again on the next upgrade within 2.x.

Roll back to 1.2

Because 2.0.0 changed neither the schema nor the resources, a rollback is a change of binary or image:

Helm: helm rollback kuben <revision> (helm history kuben -n kuben-system lists them). A rollback runs no pre-upgrade hook. An helm upgrade --version 1.2.1 instead needs --set backup.beforeUpgrade=false, because 1.2’s check refuses to go back from a version it has seen run.
Server: curl -fsSL https://kuben.teamtem.com/install.sh | sh -s -- --version v1.2.1.

Kuben records every version that started against the database, so after a rollback kuben upgrade-check of 1.2.x reports the step from 2.0 as going back. That is expected; the server starts regardless.

Upgrading to 2.1 (not released yet)

2.1 is the first release since 2.0 that changes the database: migration 0035_integrations adds the tables for Git connections (GitLab, Gitea/Forgejo, GitHub tokens) and organization registries, and build stages and kept build logs. The server applies it on start, after the backup the pre-upgrade hook (Helm) or kuben migrate takes; kuben upgrade-check reports it as a pending migration.

Going back from 2.1 to 2.0 is therefore not an image swap: 2.0 refuses a database that has a migration it does not know. Keep the backup taken before the upgrade, and restore it (kuben restore) if you need to go back.

Binary

Re-run the installer, optionally pinned. It runs kuben setup, which replaces the binary and restarts the service. To check before the new version takes over, install the binary only, check, then restart:

curl -fsSL https://kuben.teamtem.com/install.sh | sh -s -- --version v1.2.1 --binary-only
sudo -u kuben kuben upgrade-check
sudo systemctl restart kuben

When migrations are pending, the server takes a backup before it applies them (backup.before_upgrade), with the PostgreSQL client tools of the server. kuben migrate applies pending migrations without starting the server, the same way, if you prefer a separate step.

Versioning

Kuben follows semantic versioning. The Helm chart version, the image tag and the binary version are the same number. Pre-releases (v2.0.0-alpha.1, -beta.N, -rc.N) are published as GitHub pre-releases: they never move the latest image tag, install.sh and Helm skip them unless you name the version, and the image gets only its exact tag.

Every release ships with checksums, build attestations and an image scanned before it is pushed; see CI/CD and releases.