Troubleshooting and observability
First stops
Section titled “First stops”| Tool | Use |
|---|---|
kuben doctor |
prerequisites: database, cluster, Gateway API, cert-manager, metrics-server; on an installed server also Kuben’s Gateway, its issuer and ports 80 and 443 |
GET …/apps/{app}/doctor |
why one app is or is not reachable (below) |
GET /api/v1/healthz/details |
per-subsystem status (sign-in required) |
/livez, /readyz |
Kubernetes probes. /readyz stays 503 until the informers have synced |
kubectl -n kuben-system get lease kuben-controller |
which pod runs the controllers (holderIdentity); the others report the subsystem controllers as standby in /api/v1/healthz/details |
kubectl get apps,environments -A |
conditions (Ready, Exposed) with reason and message |
:9090/metrics |
Prometheus metrics |
Conditions
Section titled “Conditions”Apps and environments explain themselves through conditions.
App Ready
| Reason | Meaning |
|---|---|
Progressing |
a rollout is in progress; new pods are not all ready yet |
Available |
the desired number of pods is ready |
RolloutFailed |
no progress for 10 minutes; the previous pods keep serving. The message carries the Deployment’s own reason (image pull, quota, failing probe) |
Scheduled |
the app has only scheduled processes (CronJobs), and they are in place |
AwaitingBuild |
a Git-sourced app has no image yet; it runs once a build has produced one |
InvalidSource, MultiplePorts, ScheduledWithPort, InvalidSchedule, UnknownSize, VolumeNeedsSingleReplica, InvalidDomainTls, NoProcesses |
the spec cannot be rendered; the message names the field |
App Exposed
| Reason | Meaning |
|---|---|
RouteApplied |
the HTTPRoute exists and is attached |
NoGateway |
neither KubenConfig.spec.gatewayClassName nor spec.gateway is set; the Gateway condition of KubenConfig says more |
NoHostname |
no domain and no base domain |
GatewayAPIMissing |
Gateway API CRDs are not installed |
Environment phase: Pending, Ready, Terminating or Degraded, with deletionScheduledAt for soft-deleted environments.
Metrics
Section titled “Metrics”Kuben exposes Prometheus metrics on KUBEN_SERVER__METRICS_BIND (0.0.0.0:9090):
| Metric | Watch for |
|---|---|
kuben_reconcile_errors_total |
a rising rate: a controller cannot apply something (RBAC, quota, a bad image) |
kuben_audit_write_errors_total |
any increase: audit records are being lost, usually a database problem |
kuben_leader |
1 on exactly one replica |
kuben_subsystem_failures_total, kuben_subsystem_panics_total |
a subsystem restarted by the supervisor |
kuben_sse_lagged_total |
browsers falling behind the event stream |
Alerting rules that pay off: increase(kuben_audit_write_errors_total[5m]) > 0, sum(kuben_leader) != 1 for more than a minute, and rate(kuben_reconcile_errors_total[10m]) > 0.
An app’s logs and events
Section titled “An app’s logs and events”Anyone who may read an app’s logs gets its recent lines, per pod (at most ten pods, 200 lines each by default), and the lines of the crashed container before the current one:
curl -b cookies "$KUBEN/api/v1/projects/shop/environments/prod/apps/web/logs?tail=100"curl -b cookies "$KUBEN/api/v1/projects/shop/environments/prod/apps/web/logs?previous=true"With follow=true the answer stays open as server-sent events: a line event for every new line (pod, process, time, line), and an end event when a pod’s log stops (a restarted container and the pods of a new rollout are followed on their own). Each user may follow four logs at once and a replica a hundred; more answer 429. A followed log ends after an hour, and closing the connection stops the reads at once.
curl -N -b cookies "$KUBEN/api/v1/projects/shop/environments/prod/apps/web/logs?follow=true&tail=10"…/events lists the Kubernetes events of the app’s own objects, newest first: pods, Deployments, ReplicaSets, Jobs and CronJobs, its route, and the certificates of its hosts (image pulls, failed probes, scheduling, HTTP-01 challenges). Kubernetes keeps events for about an hour.
An app’s Doctor
Section titled “An app’s Doctor”…/apps/{app}/doctor runs every check between a visitor and the app and says, for each, ok, warn, fail, or unknown when it could not be checked, with what to do about it. The report’s own status is the worst of them; a check that could not be made never counts as ok.
| Check | Fine when |
|---|---|
gateway-class |
the GatewayClass is accepted by its controller |
gateway |
Kuben’s Gateway is programmed; its addresses are shown |
issuer |
the ClusterIssuer is ready (without one, apps are plain HTTP: a warning) |
port-80, port-443 |
the Gateway accepts connections on its address, tried from the server (443 only with TLS) |
route |
the Gateway accepted the app’s route |
certificate |
each host’s certificate is issued; a host served without TLS is a warning |
dns |
each host resolves to the Gateway’s address |
agent |
for an app its cluster’s agent delivers: the agent is linked and was heard from lately |
curl -b cookies "$KUBEN/api/v1/projects/shop/environments/prod/apps/web/doctor"Kuben’s own logs
Section titled “Kuben’s own logs”Kuben logs structured JSON by default (KUBEN_TELEMETRY__LOG_FORMAT=json; pretty for a terminal). Every request carries a request id, which the audit log records too.
| Log line | Meaning |
|---|---|
gateway listener limit reached |
the Gateway has 60 Kuben listeners; use a wildcard certificate for generated hosts |
hostname requested by two namespaces |
two apps in different environments claim the same domain; the first keeps it |
migrations applied |
boot-time migrations succeeded |
Common situations
Section titled “Common situations”The console says an app is Ready but the domain does not answer. Run Check DNS: mismatch or unresolved means DNS, not Kuben. If DNS is ok, look at the certificate: kubectl -n kuben-system get certificates and cert-manager’s log.
/readyz stays 503 after start. The informers are still listing objects, or the service account lacks list/watch on a resource. kubectl -n kuben-system logs deploy/kuben shows which.
Two replicas both log reconciliation. Leader election is off. It is always on in the chart; for the binary set KUBEN_KUBE__LEADER_ELECTION=true when more than one process has the controller role.
Autoscaling never scales. metrics-server is missing or not serving metrics.k8s.io. kuben doctor checks it.
Apps in staging can reach production. The CNI does not enforce NetworkPolicy (flannel). Install Cilium or Calico.
Getting help
Section titled “Getting help”Open an issue with the output of kuben doctor, kuben version, the relevant conditions and the log lines around the problem, with secrets redacted: github.com/Teamtem-dev/kuben/issues.