CI/CD and releases

Every CI job calls the same Turborepo task a developer runs locally. Pull requests run only the job groups their files touch; pushes to main, the merge queue and manual runs run everything.

On every pull request

Job	What it runs
Detect changes	selects job groups from the changed paths; a change to CI itself or to the task runner selects everything (fail open)
Go	go mod verify, golangci-lint, scripts/go-check.sh (gofumpt, go vet, exhaustive switches, sum types, NilAway), go test -race with every database test against PostgreSQL 17 and every API-server test against envtest (a skip fails the job), and govulncheck on the module
Go agent runtime (kind)	the agent’s executor against a cluster whose Deployments really roll out
Oracle	the Go hub and the released Rust 1.2.0 binary on two empty databases, answering the same requests; the answers must match
Web	Biome, tsc, bun test, the Vite build, the Astro build, size-limit (200 kB initial JS, 80 kB per lazy chunk, 25 kB CSS, brotli), and the console in a browser against a mocked API (Playwright, axe, Kuben’s CSP)
Console on the Go server	the console as it ships, embedded in the Go binary on PostgreSQL, nothing mocked: first-run setup, sign-in, a project, an environment, an image app, team, API tokens, the audit log and sign-out, with Persian for sign-in and the project list (Playwright, axe, CSP)
Generated files	regenerates schema.d.ts from the frozen openapi.json and fails on any diff
API compatibility	oasdiff between the spec of the base branch and the pull request: additions pass, breaking changes need the breaking label
Shell scripts	execute bits, shellcheck, install.sh under sh/dash/bash, change-detection tests, helm lint --strict, helm template, and a Trivy misconfiguration scan of the rendered chart
Binary size budget	the static hub (CGO_ENABLED=0) with the embedded console and module list, gated at 45 MiB, then a Trivy scan of the binary
End-to-end (kind)	the Go hub and agent: login → project → environment → namespace with quota and NetworkPolicy → deploy → scale → logs → restart → delete and GC, the agent as a pod, HTTPS through a Gateway; also the build engine, kuben setup on the runner and on k3s
Workflow security	zizmor on the workflows and the setup action: template injection, excessive permissions, cache poisoning, unpinned actions
CI success	fails if any job failed or was cancelled. This is the only required check in branch protection

Which groups a branch would run: git diff --name-only origin/main...HEAD | scripts/ci-changes.sh.

Console visual snapshots

The Web job also compares screenshots of the console’s key pages (home, an app, an environment, incidents, audit, settings, sign-in) in English and Persian, light and dark, against apps/console/e2e/__screenshots__/ (apps/console/e2e/visual.pw.ts; the API is mocked and the clock pinned). A snapshot that does not exist yet is written instead of compared, so a new page never fails its first run, and the job uploads the folder as the console-screenshots artifact. To add or refresh them after an intended visual change, either commit the files from that artifact, or on Linux (the CI platform):

cd apps/console && bun run build
cd e2e && npm install --no-package-lock && npx playwright install chromium-headless-shell
npx playwright test visual.pw.ts --update-snapshots

and commit apps/console/e2e/__screenshots__/. Snapshots are per platform (-linux), so a run on macOS writes its own files next to them rather than failing.

Details that matter:

• Every action is pinned to a commit SHA; Dependabot keeps them current with a 7-day cooldown.
• Go is installed by actions/setup-go at the version in go.work; third-party tools are installed at exact versions by a checksum-verifying installer, and the Go tools are pinned in tools/go.mod. Trivy’s own GitHub Actions are deliberately not used after their tags were hijacked in March 2026.
• permissions: contents: read is the default; each job asks for what it needs; persist-credentials: false on every checkout.
• The Turborepo remote cache is read by pull requests and written only on main, so a pull request cannot plant a poisoned cache. Release jobs use no cache at all.

Releases

tag v2.0.1 ─► plan (tag shape: v2.X.Y[-(alpha|beta|rc).N])
               ├─► test (go test -race, PostgreSQL + envtest, govulncheck)
               └─► web (bun: vite build + size-limit) ─► publish (goreleaser: build ×5, checksums, signatures, attestations, GitHub Release)
                                                              ├─► verify-install ×3 (install.sh against the real release)
                                                              └─► image (budget ≤ 50 MiB → Trivy → push amd64+arm64, SBOM, provenance)
                                                                       └─► chart (helm push oci://ghcr.io/teamtem-dev/charts) ─► smoke

The 2.x release keeps the names, layout and signatures of the 1.x (Rust) releases, so install.sh and existing automation keep working. 1.2.x security fixes are released from the release/1.2 branch, which keeps the Rust release workflow.

Artifacts of a release:

Target	File
x86_64-unknown-linux-musl	kuben-x86_64-unknown-linux-musl.tar.gz
aarch64-unknown-linux-musl	kuben-aarch64-unknown-linux-musl.tar.gz
aarch64-apple-darwin	kuben-aarch64-apple-darwin.tar.gz
x86_64-apple-darwin	kuben-x86_64-apple-darwin.tar.gz
x86_64-pc-windows-msvc	kuben-x86_64-pc-windows-msvc.zip

The Linux archives also hold kuben-agent. Next to them: a CycloneDX SBOM per archive (kuben-<target>.cdx.json), checksums.txt over all of these and its keyless Sigstore signature checksums.txt.sigstore.json, install.sh, bundle.lock.json (what setup installs besides Kuben, pinned), the image ghcr.io/teamtem-dev/kuben:{X.Y.Z, X.Y, latest} and the chart oci://ghcr.io/teamtem-dev/charts/kuben. Tags like v2.0.0-rc.1 become GitHub pre-releases: the image gets only its exact tag, latest and X.Y do not move, and install.sh and Helm skip them unless asked for that version.

Design decisions worth knowing:

• The image is built from the published binaries, not from source. The release Dockerfile is a single COPY, so the bytes in the image are exactly the ones that were checksummed and attested.
• Everything is signed without a key to lose. checksums.txt (and through it every archive, SBOM, install.sh and the bundle lock), the image and the Helm chart are signed with cosign’s keyless flow: the certificate names the release workflow and the tag, and the signature is recorded in Sigstore’s transparency log. install.sh checks it whenever cosign is installed.
• One place for third-party pins. bundle.lock.json holds the versions and digests of k3s, the Gateway API CRDs, cert-manager (chart and images) and the chart’s PostgreSQL; the binary embeds it and tests fail when the chart or setup disagree with it.
• Every binary carries its dependency list. Go embeds the exact module list (go version -m), so trivy, grype, osv-scanner or govulncheck -mode=binary can audit a deployed binary or image.
• Releases build without a cache.
• The image gates run before the push: size budget and Trivy on linux/amd64 first, then the multi-arch push.
• USER 65532:65532 is numeric, so Kubernetes can verify runAsNonRoot.

Acceptance on a real server

CI proves the install on a fresh runner (k3s with a test CA) and on kind, but a public certificate and an address reachable from elsewhere need a real server. Before a milestone closes, scripts/vps-acceptance.sh runs on a throwaway VPS whose wildcard DNS record points at it:

sudo KUBEN_BIN=./kuben DOMAIN=apps.example.com ACME_EMAIL=ops@example.com \
  scripts/vps-acceptance.sh | tee acceptance.md

It installs with HTTPS (Let’s Encrypt; ACME_STAGING=1 for the staging server), creates the first admin, deploys an app with kuben deploy, checks the app and the console over public HTTPS, checks that HTTP redirects, reads the app’s Doctor, stops kuben.service for a minute while the app keeps answering, runs setup again (nothing may change), and appends the footprint of the whole stack. The report is Markdown; the admin password stays in a root-only file. Check the two printed curl commands from another machine.

Cutting a release

1. Make sure CI success is green on the commit. The version comes from the tag: there is no version file to bump.
2. Tag and push:

   git tag -s v2.0.1 -m "v2.0.1"
   git push origin v2.0.1

3. The plan job refuses a tag outside the release train (v2.X.Y, or -alpha.N, -beta.N, -rc.N before a minor’s GA), and nothing is published unless the tests pass on the tagged commit.

Every day

Advisories are published after code is merged, so a go.sum that passed yesterday can be vulnerable today. A scheduled workflow runs daily: govulncheck on the Go modules of main, and trivy image on the published latest image for HIGH and CRITICAL vulnerabilities with a fix available. A failure opens (or comments on) one issue, Scheduled security check failed. The fix is to update the dependency, merge, and cut a patch release so the image follows.

This website

apps/site is built in the Web job like any other package and deployed from main to a Cloudflare Worker with static assets by its own workflow. Internal links are validated at build time, so a renamed page fails the build instead of shipping a 404.