High availability

replicaCount can be raised: every replica serves the API, and only the holder of the kuben-controller Lease runs the controllers. The chart’s own PostgreSQL is a single instance, though, so for several replicas or high availability point Kuben at an external PostgreSQL.

Use an external PostgreSQL

Create a Secret with a url key and point the chart at it:

kubectl -n kuben-system create secret generic kuben-db \
  --from-literal=url='postgres://kuben:…@postgres.db.svc:5432/kuben?sslmode=require'

helm upgrade kuben oci://ghcr.io/teamtem-dev/charts/kuben -n kuben-system --reuse-values \
  --set database.existingSecret=kuben-db \
  --set replicaCount=3

database.existingSecret wins over database.url. With either set, the chart renders no PostgreSQL of its own. Migrations run at boot with a lock, so several replicas starting together do not race. The volume and the Secret of an earlier chart-run PostgreSQL stay behind; move their data yourself if you need it (see Backup and restore).

Connect as an ordinary role that owns its database, not as a superuser or a BYPASSRLS role: row-level security does not apply to those, and kuben doctor warns about it (database role).

Caution

replicaCount > 1 also renders with the chart’s own PostgreSQL, but that database is a single instance: when its pod or node goes down, every replica loses it.

What each replica does

• Every replica serves the API and the console, and keeps its own read models (projections) fed by Kubernetes watches. Reads never touch the API server.
• Only one replica runs the controllers: the holder of the coordination.k8s.io/v1 Lease kuben-controller in Kuben’s namespace. The others report controllers: standby.
• Login throttling is stored in the database, so all replicas share one budget.
• Sessions are cached per replica for KUBEN_SECURITY__SESSION_CACHE_TTL_SECS (5 s by default). A revoked session can stay valid on another replica for at most that long.

Leader election

The election is designed so that clock skew between nodes can never produce two leaders:

• Every write to the Lease is a compare-and-swap on its resourceVersion.
• A candidate treats the Lease as expired only after it has seen the record unchanged for the lease duration (15 s). It never compares renewTime with its own clock.
• The leader renews every 2 s. If renewal fails for 10 s it stops its controllers before the Lease can expire for the others.
• On graceful shutdown the leader releases the Lease, so the next replica takes over within a few seconds. After a crash, within 15 s.

The Lease right is a namespaced Role, not part of the ClusterRole. Read ADR-023 for the reasoning.

kubectl -n kuben-system get lease kuben-controller -o jsonpath='{.spec.holderIdentity}'

The kuben_leader metric is 1 on the leader and 0 elsewhere.

Rolling upgrades

The chart always uses rolling updates with maxUnavailable: 0 and maxSurge: 1: a new pod must be ready before an old one stops. A PodDisruptionBudget stops node drains from evicting more than one pod at a time. Together with leader hand-over, an upgrade pauses the controllers for a few seconds and the API not at all.

What to expect

Event	API	Controllers
Leader stops gracefully	unaffected	resume on another replica within seconds
Leader crashes	unaffected	resume within 15 s
Rolling upgrade	unaffected	short pause on hand-over
Session revoked	immediate on that replica, ≤ 5 s elsewhere	—
PostgreSQL unavailable	logins and audited writes fail; reads from projections continue	reconciliation continues; audit writes are counted as errors