Kuben | Blog

Introducing Kuben: a Kubernetes PaaS in a single binary

Sat, 12 Sep 2026 00:00:00 GMT

Today we are releasing Kuben 1.0, a platform as a service that runs on your own Kubernetes cluster and ships as a single binary.

You give it a container image. It gives your team an isolated environment with a public HTTPS address, zero-downtime rollouts, autoscaling, logs, a numbered release history with one-click rollback, and an audit log that records who changed what. Everything it creates is an ordinary Kubernetes object, labelled and visible, so kubectl and GitOps keep working next to it.

helm install kuben oci://ghcr.io/teamtem-dev/charts/kuben --namespace kuben-system --create-namespace

That is the whole install. No operator to deploy first, no message queue, no external database unless you want several replicas, no build farm.

Why another PaaS

Small teams end up in one of two places. Either they run a hosted platform and pay per seat and per gigabyte for the privilege of not thinking about infrastructure, or they get a Kubernetes cluster and discover that “deploy a service” now means a Deployment, a Service, an HPA, an Ingress or HTTPRoute, a Certificate, a NetworkPolicy, a ResourceQuota, some RBAC, and a wiki page nobody updates.

The self-hosted PaaS projects that fill this gap are good products, and we learned from them. But when we read their code closely, the same class of problems appeared again and again: a JWT in localStorage, a default secret in the repository, a build pod that carries a service-account token while running user code, log streams that leak, a “current cluster” global that races between requests. Not because the authors were careless, but because those systems grew fast in dynamic languages without a place where invariants live.

Kuben is our attempt to make that class of problem structurally impossible rather than merely avoided. We wrote it in Rust, kept it to one process, and wrote down eighteen invariants that every change is reviewed against. The security model page says what those choices buy you and, just as important, what they do not.

What 1.0 does

Projects → environments → apps. Every environment is its own namespace with a resource quota, a NetworkPolicy and Pod Security labels.
Deploy any image with readiness and startup probes, CPU autoscaling, environment variables and write-only secrets.
Day-2 operations built in: logs, restarts, releases with rollback, promotion between environments with a diff preview, persistent volumes, cron jobs.
Custom domains with automatic HTTPS through Gateway API and cert-manager. One listener per hostname; a domain belongs to exactly one app.
One-click templates for PostgreSQL, Redis, MariaDB, n8n, Uptime Kuma, Vaultwarden and Gitea, with generated credentials that never appear in a spec.
Teams: four roles, invitations, API tokens capped at a role and scoped to a project or environment, and an append-only audit log of every change and every denial.
Operations: several replicas on PostgreSQL with Lease-based leader election, backups, boot-time migrations and CRD updates, Prometheus metrics, kuben doctor.

What it is not, yet

Kuben does not build images in 1.0. Your CI builds and pushes; Kuben rolls out with one PATCH. Git builds with BuildKit, preview environments per pull request, scale-to-zero and SSO are the roadmap, in roughly that order. The custom resources for builds and previews already exist so that the schema is stable when the features land.

Kuben is also not a sandbox for hostile tenants. Environments isolate teams from each other’s mistakes; untrusted code needs a stronger boundary than a namespace.

The shape of the thing

A few numbers, because footprint was a design goal and not an afterthought:


Release binary, console included	≤ 26 MiB, and CI fails if it grows past that
Container image	≤ 30 MiB, distroless, non-root, read-only filesystem
Memory requested by the pod	64 Mi
Console JavaScript	≤ 200 kB brotli
Platforms	Linux and macOS on x86_64 and arm64, Windows x86_64

Every release is checksummed, attested with build provenance and scanned before the image is pushed, and every binary embeds its own dependency list so you can audit it in place months later.

Try it

The quickstart takes about five minutes on any cluster from 1.29 up. On a single server, k3s plus the binary install works well. The user guides walk through the everyday scenarios: deploying from CI, inviting a team, rolling back, promoting, custom domains.

Kuben is Apache-2.0 and built by Teamtem. Issues, feature requests and pull requests are welcome on GitHub. We would love to hear what you deploy with it.

Why Kuben ships as one binary

Fri, 11 Sep 2026 00:00:00 GMT

A platform that deploys other people’s software tends to become a small distributed system of its own: an API, a controller, a worker, a scheduler, a database, a message queue, a cache, and a reverse proxy in front. Each piece is reasonable. Together they are the reason “self-hosted PaaS” usually means “a second platform to operate”.

Kuben is one process. This post is about how that works without becoming a monolith that falls over at the first real load, and what we gave up to get there.

Roles, not services

kuben serve runs a set of roles in one process: api, controller, and (reserved) activator. By default a process runs all of them. Each role is a supervised subsystem with its own lifecycle and health, so a panic in a controller restarts that controller, not the API.

The interesting part is that the roles talk through Kubernetes and the database, not through in-process channels. The API writes an App custom resource; the controller watches it and reconciles. That is the same path a kubectl apply takes, which means GitOps tools and the console are peers. It also means that splitting the roles across processes later requires no code change; the Helm chart already runs several API replicas next to one active controller.

Kubernetes holds desired state

The first design decision, and the one everything else follows from: Kubernetes is the source of truth for what should exist. Projects, environments, apps and releases are custom resources in kuben.dev. SQL holds what Kubernetes has no home for: users, sessions, tokens, memberships, login throttling and the audit log, referring to cluster objects only by uid.

This is the opposite of the “store everything in our database and sync to the cluster” approach. It costs us some convenience (a list of apps is a watch, not a SELECT), and it buys a lot: no reconciliation drift between two sources of truth, backups that are kubectl get -o yaml, and a console that can never show an app the cluster does not have.

Projections instead of caches

If the API reads from Kubernetes, doesn’t every page load hit the API server? No. Each process runs informers over the resources it cares about and keeps projections: small in-memory read models with only the fields the console needs. The API answers from those, and pushes deltas to browsers over server-sent events.

We considered caching full objects the way kube-rs reflectors do. On a cluster with a few hundred pods that is hundreds of megabytes, mostly managedFields and status nobody reads. Projections are why the Kuben pod requests 64 Mi of memory and why the pod reports ready only after every informer has listed once: a fresh replica must never answer 404 for an app that exists.

SQLite by default, PostgreSQL when you grow

SQLite on a persistent volume is the default database. It is not a toy here: WAL mode, a single-writer pool, embedded migrations under a lock, foreign keys always on, and a Recreate deployment strategy because a ReadWriteOnce volume cannot be mounted by two pods.

The same repositories run on PostgreSQL through sqlx, and CI runs the store tests against both backends on every change so they cannot drift. The moment you set database.url to a PostgreSQL DSN, the chart drops the volume, switches to rolling updates with maxUnavailable: 0, and lets you raise replicaCount.

Three replicas, one controller

Several replicas of one process that includes a controller would reconcile the same objects concurrently. The fix is not “run the controller elsewhere”; it is a Kubernetes Lease. Every replica serves the API from its own projections; only the Lease holder runs the controllers, and the rest report controllers: standby.

The election never compares timestamps with the local clock, so skew between nodes cannot produce two leaders. That deserves its own post: Leader election without trusting clocks.

Embedding the console

The web console is a React application built with Vite and embedded into the binary at compile time, pre-compressed, served with a strict Content Security Policy and an immutable cache header. There is no Node in the image, and there is no image layer for the frontend. The /api the console talks to is the same one your CI uses, and its TypeScript client is generated from the OpenAPI document the binary serves; CI fails if either drifts.

What we gave up

A second Tokio runtime as a bulkhead between the API and the controllers was in the original design. We shipped one runtime and kept the flag (runtime.bulkhead) for when a measurement, not a hunch, says we need it.
Builds are not in the binary yet. When they arrive, they will run as Kubernetes Jobs against a persistent BuildKit daemon, never inside the Kuben process and never with cluster credentials.
An in-process queue for things like log fan-out was replaced, for now, by bounded REST reads and SSE. The bounded, reference-counted log hub is on the roadmap.

The result

One binary you can curl | bash onto a server next to k3s, or one Helm release inside any cluster. A footprint you can put on a dashboard. And a platform whose every object is a Kubernetes object, which is the part we care about most. If you want to see it, the concepts page is the map, and the architecture decisions are the reasons.

Leader election without trusting clocks

Thu, 10 Sep 2026 00:00:00 GMT

When the Helm chart first allowed replicaCount > 1 on PostgreSQL, every replica ran the full controller set. Two replicas reconciled the same App, both server-side-applied with force, and the objects flapped. The chart also used the Recreate strategy regardless of database, so every upgrade had downtime. And login throttling lived in each process’s memory, so three replicas gave an attacker three times the budget.

This post is about the first problem and its fix, recorded as ADR-023.

The requirement

Exactly one replica runs the controllers at any time. Every replica keeps serving the API, because the API reads from its own projections and does not need the controllers. Failover after a crash should take seconds, not minutes, and a graceful restart should barely be noticeable.

That is a leader election, and Kubernetes has a primitive for it: the coordination.k8s.io/v1 Lease, an object with a holder identity, a lease duration, and a renew time. The usual implementation, in client-go and in kube-rs helpers, renews the Lease periodically and treats it as expired when now - renewTime > leaseDuration.

The problem with `now`

“Now” on the candidate’s node and renewTime written by the leader’s node are two different clocks. Skew of a few seconds between nodes is ordinary; a paused VM or a clock step can produce much more. If a candidate’s clock runs ahead, it sees a perfectly healthy Lease as expired, takes it, and for a while two processes believe they lead. The leader finds out only at its next renewal, when its compare-and-swap fails. In that window both reconcile.

Most systems accept this window and make reconciliation idempotent enough that it rarely matters. We wanted the window to not exist.

Never compare a remote timestamp with a local clock

The rule we adopted: a candidate treats the Lease as expired only after it has observed the Lease record unchanged for the full lease duration, measured on its own clock. It reads the Lease, remembers resourceVersion and renewTime, and starts a local timer. If the leader renews, resourceVersion changes and the timer restarts. If nothing changes for 15 seconds, the Lease is dead by any clock, because a live leader would have renewed it several times in that interval.

Every write is a compare-and-swap on resourceVersion. Two candidates that both decide the Lease is dead race on that write and exactly one wins; the other reads the new version and goes back to waiting.

The numbers:


Lease duration	15 s
Leader renewal interval	2 s
Leader gives up if renewal fails for	10 s

The last row is the other half of the safety argument. If the leader cannot renew for 10 seconds, it stops its controllers before the 15-second mark at which a candidate could take over. The leader’s own clock is enough for that decision, because it only compares its own timestamps. So the leader has always stepped down before anyone else can step up.

Graceful and ungraceful

On shutdown, the leader releases the Lease by clearing the holder identity in one more compare-and-swap. Candidates notice the change immediately and the next one takes over within a couple of seconds. This is the path a rolling upgrade takes, and the chart’s maxUnavailable: 0 plus a PodDisruptionBudget mean that a node drain never removes more than one pod.

After a crash there is nobody to release the Lease, so the controllers pause for up to 15 seconds. That is the trade: a slightly longer failover for the guarantee that there is never a second leader.

What runs under the Lease

Anything that must happen exactly once per cluster runs only on the leader: the reconcilers, applying the CRDs at boot, and the Gateway listener management. Anything that must happen on every replica, informers, projections, the API, session caching, runs everywhere. The rule is written into the review checklist so that new once-per-cluster work does not end up on every replica by accident.

The Lease right is a namespaced Role on that one object, not part of the ClusterRole, and the metric kuben_leader is 1 on exactly one pod; alert if sum(kuben_leader) != 1 for more than a minute.

The other two fixes

Since the same decision touched replica behaviour, we fixed the two related problems at once:

Rolling updates on PostgreSQL. Recreate stays for SQLite, whose volume is ReadWriteOnce; PostgreSQL deployments roll one pod at a time.
Login throttling in the database. The three counters (email + IP, IP, account) are rows keyed by SHA-256 hashes, shared by all replicas, so the budget is the same with one replica or ten.

There is one thing that stays eventually consistent by design: a revoked session is cached per replica for up to 5 seconds (KUBEN_SECURITY__SESSION_CACHE_TTL_SECS). We judged that a bounded, documented window there beats a database round-trip on every request.

Try it

helm upgrade … --set database.existingSecret=kuben-db --set replicaCount=3, then kubectl -n kuben-system get lease kuben-controller -w and delete the leader pod. The high availability page has the full setup and the table of what to expect in each failure mode.

How we release Kuben: attested, scanned, and built from the bytes we checked

Wed, 09 Sep 2026 00:00:00 GMT

A platform that deploys your software runs with a service account that can read every Secret in the cluster. Its release process should be at least as careful as the code. This post walks through Kuben’s, step by step, with the reasoning for each.

The shape

tag v1.0.2 ─► plan (tag == Cargo version)
               ├─► web (bun: vite build + size-limit) ─► build ×5 (cargo auditable) ─┬─► publish (checksums, attestations, GitHub Release)
               │                                                                     │        └─► verify-install ×3
               │                                                                     └─► image (budget → Trivy → push, SBOM, provenance)
               │                                                                              └─► chart (helm push)

A release starts with a signed git tag. Nothing else can trigger it.

1. The tag must match the code

The plan job checks that the tag equals [workspace.package] version in Cargo.toml. If someone tags v1.0.3 on a commit that says 1.0.2, nothing is built. It sounds trivial; it prevents the class of “the binary says one version and the release page another” confusion that makes incident timelines painful.

2. Build without a cache

CI uses a Rust cache and a Turborepo remote cache to stay fast, and pull requests can only read those caches; writes happen on main. Releases go further: they use no cache at all. The web bundle is built by calling the package scripts directly with Bun rather than through the task runner, and the Rust builds and the Zig toolchain run cache-free. A poisoned cache, however it got there, cannot reach a published artifact.

3. Five targets, tested on four operating systems

Binaries ship for Linux and macOS on x86_64 and arm64 and for Windows x86_64. Linux builds are static musl binaries, cross-compiled with cargo-zigbuild so that the arm64 build does not need QEMU. The same five targets are exercised by the test matrix on every pull request; a platform we ship is a platform we test.

4. Every binary carries its dependency list

The builds run under cargo auditable, which embeds the exact list of crates (about 6 KiB for Kuben’s 378 crates) in a linker section of the binary. Months later, trivy, grype, osv-scanner or cargo audit bin can point at a deployed binary or image and tell you whether any of its dependencies has an advisory. We scan the Linux binaries this way before they are packaged, and fail on any HIGH or CRITICAL vulnerability with a fix available.

5. Checksums and attestations

Each archive gets a .sha256 file, and checksums.txt lists all of them. The publish job also generates GitHub build provenance attestations, so gh attestation verify kuben-x86_64-unknown-linux-musl.tar.gz --repo Teamtem-dev/kuben proves the archive was built by this workflow from this commit.

install.sh reads checksums.txt and verifies the archive before extracting anything. It downloads over HTTPS only, wraps its whole body in a main() called on the last line so that a truncated download runs nothing, and never calls the GitHub API. After every release, a job runs the script on Ubuntu x64, Ubuntu arm64 and macOS against the real assets and checks kuben --version.

6. The image is a `COPY`

The release image is not built from source. deploy/release.Dockerfile copies the already-checksummed musl binary into distroless/static:nonroot. The multi-arch build takes seconds, needs no emulation, and, more to the point, the bytes in the image are the exact bytes that were checksummed, attested and scanned in the previous steps.

The Dockerfile sets USER 65532:65532 numerically. Kubernetes cannot verify runAsNonRoot: true against a user name, and rejects the pod; a number it can check.

7. Gates before the push

The linux/amd64 image is loaded locally, measured against its 30 MiB budget and scanned by Trivy. Only if both pass is the multi-arch image pushed, with an SBOM and provenance attached. Then the Helm chart is pushed to the same OCI registry. Tags such as v1.1.0-rc.1 become pre-releases and never move the latest image tag or the X.Y tag.

8. Then, every day

A lockfile that passed yesterday can be vulnerable today; advisories are published after code is merged. A scheduled workflow runs daily: cargo deny check advisories on main, and trivy image on the published latest. A failure opens one issue, Scheduled security check failed, and later failures comment on it instead of opening more. The fix is always the same: update, merge, cut a patch release so the image follows.

The CI side

The release pipeline inherits the rules of the CI workflow:

Every action is pinned to a commit SHA, with the tag in a comment, and Dependabot keeps them current with a 7-day cooldown against freshly published malicious packages.
Rust is installed with the runner’s own rustup; third-party tools are installed at exact versions by a checksum-verifying installer. We deliberately do not use Trivy’s own GitHub Actions after their tags were hijacked in March 2026.
permissions: contents: read is the default; each job requests what it needs; persist-credentials: false on every checkout.
The workflows themselves are audited with zizmor on every pull request for template injection, excessive permissions and cache poisoning.

Why write this down

None of this is novel. All of it is easy to skip, and each skipped step is a way for someone else’s code to end up in your cluster with cluster-admin. Writing the pipeline down, in the CI/CD page and here, is how we keep ourselves from skipping steps when a release is urgent. If you see a gap, we would like to hear about it, privately if it is a vulnerability.

Kuben | Blog

Introducing Kuben: a Kubernetes PaaS in a single binary

Why another PaaS

What 1.0 does

What it is not, yet

The shape of the thing

Try it

Why Kuben ships as one binary

Roles, not services

Kubernetes holds desired state

Projections instead of caches

SQLite by default, PostgreSQL when you grow

Three replicas, one controller

Embedding the console

What we gave up

The result

Leader election without trusting clocks

The requirement

The problem with now

Never compare a remote timestamp with a local clock

Graceful and ungraceful

What runs under the Lease

The other two fixes

Try it

How we release Kuben: attested, scanned, and built from the bytes we checked

The shape

1. The tag must match the code

2. Build without a cache

3. Five targets, tested on four operating systems

4. Every binary carries its dependency list

5. Checksums and attestations

6. The image is a COPY

7. Gates before the push

8. Then, every day

The CI side

Why write this down

The problem with `now`

6. The image is a `COPY`