Kubernetes Production Checklist: Cluster Level and Application Level

A Deployment with replicas: 3 can still go down entirely from losing a single node. A "healthy" Pod can still be returning 500s to users because its readiness probe doesn't reflect real state.

This is the checklist I use when reviewing a workload before it goes to production, split into two levels:

• Cluster Level: things owned by infra/platform teams, control plane, nodes, networking, and security policy at the cluster level.
• Application Level: things within a dev/DevOps engineer's control when writing manifests for a workload.

Each item gets a short "why it matters" and a Bad/Good example.

Note: This checklist is assembled from real-world experience and various references, not every item is mandatory for every system, pick what fits your scale.

Let's go 🚀

Part 1: Cluster Level

1. Control Plane High Availability

The control plane includes the API server, scheduler, controller-manager, and etcd. Running a single control plane node means the whole cluster "freezes" (no scaling, no deploys, no new Pods) if that node dies, even though already-running Pods keep serving traffic normally (a common misconception: losing the control plane doesn't mean losing traffic right away, but you won't self-heal from the next incident).

• Bad: A single control plane node, no load balancer in front of the API server.
• Good: At least 3 control plane nodes (an odd number for quorum), sitting behind a Load Balancer.

# kubeadm init config - pointing the API server through an LB
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
controlPlaneEndpoint: "k8s-api.internal:6443" # LB endpoint, not a single node
etcd:
  local:
    dataDir: /var/lib/etcd

If you're on a managed control plane (EKS/GKE/AKS), the cloud handles this part for you, but it's still worth knowing what the provider does not cover (e.g. etcd backups are usually still your responsibility on a self-managed cluster).

2. etcd Quorum & Backup/Restore

etcd uses the Raft algorithm and needs a quorum (a majority of members alive) to function. With an odd member count, fault tolerance looks like this:

etcd members	Max tolerable loss
3	1
5	2
7	3

What does etcd store? Every object you can kubectl get (Pod, Deployment, Service, ConfigMap, Secret, RBAC, PV/PVC...) as key-value pairs. Not stored: container images, Pod logs, actual PV/PVC data, metrics. Only the API server talks to etcd directly, so losing quorum doesn't crash running Pods, it just blocks writing new state (deploy, scale...).

• Bad: Running 2 or 4 etcd members (an even count doesn't improve fault tolerance over the odd number below it, just wastes resources).
• Good: 3 or 5 members, and always have periodic backups. HA doesn't replace backups, you can still recover from a snapshot after losing quorum.

# Backup an etcd snapshot
etcdctl snapshot save /backup/etcd-$(date +%Y%m%d).db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

# Restore when needed
etcdctl snapshot restore /backup/etcd-20260718.db \
  --data-dir=/var/lib/etcd-restore

3. Worker Nodes Spread Across Zones

If every worker node sits in a single availability zone, one zone outage takes down the whole app, no matter how many replicas you have.

• Bad: All nodes in ap-southeast-1a.
• Good: A node pool spread across multiple zones, labeled topology.kubernetes.io/zone so the scheduler knows how to spread Pods out.

kubectl get nodes -L topology.kubernetes.io/zone
# NAME       STATUS   ZONE
# node-1     Ready    ap-southeast-1a
# node-2     Ready    ap-southeast-1b
# node-3     Ready    ap-southeast-1c

The zone label alone isn't enough. Application Level item 4 covers how to actually put it to use with Topology Spread Constraints.

4. Namespace Separation & Resource Quota

Without per-namespace resource limits, one "greedy" team/app (a memory leak, a load test left running) can eat up the whole cluster's resources, leaving other teams' apps OOMKilled or stuck Pending indefinitely.

• Bad: A shared namespace, no ResourceQuota, no LimitRange.
• Good: One namespace per team/environment, with clear quotas.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi
    pods: "50"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-a
spec:
  limits:
    - default:
        cpu: 500m
        memory: 512Mi
      defaultRequest:
        cpu: 100m
        memory: 128Mi
      type: Container

LimitRange makes sure any container that forgets to declare resources still gets sane defaults, preventing one unbounded Pod from "eating" an entire node.

5. RBAC: Least Privilege

Every workload should have exactly the permissions it needs, nothing more. cluster-admin on every ServiceAccount is the fastest way to turn one small app vulnerability into a full cluster takeover.

• Bad: Granting a ClusterRoleBinding with cluster-admin to an app's ServiceAccount "just to be safe."
• Good: A dedicated Role/RoleBinding, scoped to the right namespace and the right resources.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: shop-api-role
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["configmaps", "secrets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: shop-api-binding
  namespace: production
subjects:
  - kind: ServiceAccount
    name: shop-api-sa
    namespace: production
roleRef:
  kind: Role
  name: shop-api-role
  apiGroup: rbac.authorization.k8s.io

Audit regularly, RBAC tends to bloat over time as nobody cleans up permissions that are no longer used.

6. NetworkPolicy

By default, every Pod in the cluster can talk to every other Pod, including Pods from two completely unrelated teams. If one Pod is compromised, an attacker can move laterally across the whole cluster.

• Bad: No NetworkPolicy at all, all east-west traffic is allowed.
• Good: Default-deny, only open the traffic flows you actually need.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-shop-api-from-ingress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: shop-api
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
      ports:
        - port: 8080
  policyTypes: ["Ingress"]

7. Pod Security Standards

Every namespace should enforce a minimum security baseline, no container should run privileged, run as root, or mount arbitrary host paths.

• Bad: Nothing enforced, any Pod deploys fine, including privileged: true.
• Good: Label the namespace with pod-security.kubernetes.io to enforce the restricted or baseline standard.

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

8. Image Supply Chain & Admission Control

An image with a serious CVE, or pulled from an untrusted registry, is the most common attack path into production.

• Bad: Pulling images straight from public Docker Hub, no scanning, no provenance checks.
• Good: CI/CD scans images (Trivy, Grype...) before push, only allow pulls from a whitelisted internal registry, use an admission controller (Kyverno, OPA Gatekeeper) to block unscanned images.

# Scan in CI before pushing
trivy image --severity HIGH,CRITICAL --exit-code 1 myregistry.io/shop-api:1.4.0

# Example Kyverno policy: only allow images from the internal registry
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
    - name: allowed-registries
      match:
        resources:
          kinds: ["Pod"]
      validate:
        message: "Only images from myregistry.io are allowed"
        pattern:
          spec:
            containers:
              - image: "myregistry.io/*"

9. Cluster/Node Autoscaling

Application-level autoscaling (HPA) is pointless if the cluster has no room for new Pods, they'll just sit Pending.

• Bad: A fixed node pool, HPA scales to 10 replicas but the cluster only fits 6.
• Good: Enable Cluster Autoscaler or Karpenter, keep some headroom (low-priority pause-pod overprovisioning) to cut latency when you need to scale fast.

# Pause pod overprovisioning: reserves headroom, yields it immediately when needed
apiVersion: apps/v1
kind: Deployment
metadata:
  name: overprovisioning
spec:
  replicas: 3
  selector:
    matchLabels: { app: overprovisioning }
  template:
    metadata:
      labels: { app: overprovisioning }
    spec:
      priorityClassName: overprovisioning-low-priority
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests: { cpu: 1, memory: 2Gi }

Note: Cluster Autoscaler scale-down must respect PodDisruptionBudget. Misconfigure it and node removal can hang indefinitely (related to Application Level item 3).

10. Ingress HA + TLS

The ingress controller is a classic single point of failure if it only runs 1 replica, lose that one Pod and all inbound traffic stops reaching the cluster.

• Bad: Ingress controller at 1 replica, no TLS (plain HTTP traffic).
• Good: Ingress controller with ≥ 2 replicas plus anti-affinity and a PDB, TLS enforced via cert-manager.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop-api
  namespace: production
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts: ["api.example.com"]
      secretName: shop-api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: shop-api
                port: { number: 80 }

11. Observability Stack

Without centralized metrics/logs, you won't know something's wrong until a user reports it, and by then it's already too late.

• Bad: Manually running kubectl logs on each Pod whenever there's an incident, no alerts.
• Good: Prometheus + Grafana (or a managed equivalent) collecting metrics, centralized logging (Loki/ELK), alerts based on SLO burn rate, not just raw CPU/memory.

# ServiceMonitor so Prometheus auto-discovers the metrics endpoint
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: shop-api
  namespace: production
spec:
  selector:
    matchLabels: { app: shop-api }
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

12. StorageClass & Backup

For stateful workloads, losing data because of a wrong reclaim policy or a backup that's never been restore-tested is the most serious production risk, worse than downtime itself.

• Bad: reclaimPolicy: Delete on a volume holding important data, backups that are "configured" but never actually restore-tested.
• Good: reclaimPolicy: Retain for important volumes, periodic Velero backups with periodic restore tests to confirm the backups actually work.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: production-ssd
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

# Velero: backup and test restore
velero backup create db-backup-$(date +%Y%m%d) --include-namespaces production
velero restore create --from-backup db-backup-20260718 --namespace-mappings production:production-restore-test

---

Part 2: Application Level

1. Readiness / Liveness / Startup Probes

This is the most important trio, and also the easiest to misconfigure:

• Readiness: "Don't send me traffic right now." Failing it removes the Pod from Endpoints, no new traffic, but it is not restarted.
• Liveness: "I'm stuck, restart me." Failing it kills and recreates the container.
• Startup: For slow-starting apps. While the startup probe hasn't passed, liveness/readiness are temporarily disabled, so the app doesn't get killed mid-boot.

• Bad: A liveness probe configured too aggressively (failureThreshold: 1, timeoutSeconds: 1) on a heavy endpoint. An app under high load gets mistaken for "dead," Kubernetes restarts it repeatedly, creating a restart storm that makes the load even worse.
• Good: Keep liveness simple, just check the process is alive (no DB calls, no external service calls). Readiness is where you check real dependencies (DB, cache...). A separate startup probe for slow-booting apps.

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
startupProbe:
  httpGet:
    path: /healthz
    port: 8080
  failureThreshold: 30   # allows up to 30 x 10s = 5 minutes for the app to boot
  periodSeconds: 10

Important note for slow-starting apps: besides startupProbe, don't forget minReadySeconds at the Deployment level (not a probe field, it lives at spec.minReadySeconds). It defaults to 0, meaning the moment the readiness probe passes for the first time, the Pod counts as "available" and the rollout immediately moves on to the next Pod. For apps that boot slowly or need real warm-up time after readiness passes (loading caches, opening connection pools...), a rollout moving too fast can replace all 3 Pods before you catch a hidden issue in the new Pod (e.g. a connection leak that only shows up a few dozen seconds in). Set minReadySeconds to a sensible value (e.g. 30-60s) so the rollout "waits a bit longer" and actually observes Pod stability before calling it a success, cutting the risk of rolling out a whole batch of broken Pods back to back.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: shop-api
spec:
  replicas: 3
  minReadySeconds: 30   # wait 30s after Ready before counting as "available", useful for slow-booting apps
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
        - name: shop-api
          startupProbe:
            httpGet: { path: /healthz, port: 8080 }
            failureThreshold: 30
            periodSeconds: 10

2. Resource Requests & Limits

Without declaring resources, your Pod falls into the BestEffort QoS class, the first candidate to be evicted when a node runs low on resources, no matter how important the app actually is.

• Bad: No requests/limits set.
• Good: Set both, based on real usage data (not guesswork), to reach the Guaranteed QoS class for important workloads.

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

Tip: run the Vertical Pod Autoscaler in Off mode (recommendation-only, no auto-apply) for a few days to get realistic request/limit numbers instead of guessing.

3. Pod Disruption Budget

PodDisruptionBudget (PDB) protects you from voluntary disruption: node drains during upgrades, cluster autoscaler scale-down. It does not protect you from a manual kubectl delete or a node dying unexpectedly (involuntary disruption).

• Bad: No PDB, the infra team drains a node for patching and accidentally takes out all your Pods at once.
• Bad, the opposite way: minAvailable: 3 while replicas: 3. Technically correct, but leaves kubectl drain hanging forever, no node can ever be maintained.
• Good: Set a sensible buffer, e.g. allow at most 1 Pod down at a time.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: shop-api-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: shop-api

4. Pod (Anti-)Affinity & Topology Spread Constraints

Having 3 replicas that the scheduler happens to place on the same node/zone is no different from having 1 replica, that node/zone going down wipes out everything.

• Bad: Nothing declared, letting the scheduler decide on its own (it tends to pack Pods onto whichever node has the most free resources, easily landing them on the same node).
• Good: Use podAntiAffinity to keep 2 Pods of the same app off the same node, and topologySpreadConstraints to spread Pods evenly across zones.

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels: { app: shop-api }
        topologyKey: kubernetes.io/hostname
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels: { app: shop-api }

5. HPA / VPA

HPA scales the number of Pods (horizontal), VPA adjusts each Pod's request/limit (vertical). Running both at once on the same metric (usually CPU) makes them fight each other: VPA changes the request, HPA recalculates based on the new request, creating an unstable loop.

• Bad: Both HPA and VPA in Auto mode, both watching CPU.
• Good: HPA manages CPU/replica count, VPA runs in Off mode (Initial, recommendation-only) purely for reference, or VPA manages a different resource (e.g. memory) separate from HPA.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shop-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: shop-api
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 60 }
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300   # slow scale down, avoids flapping
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60

6. Graceful Shutdown

The #1 cause of dropped requests during a rollout: removing a Pod from Endpoints and sending SIGTERM happen in parallel, not in sequence. There's a short window where a Pod has already received SIGTERM (and may have started shutting down) but is still receiving traffic because it hasn't been removed from Endpoints on the kube-proxy/load balancer side yet.

• Bad: The app exits immediately on SIGTERM, no preStop, every rollout drops a few requests.
• Good: The app handles SIGTERM to drain in-flight connections, plus a preStop sleep to "buy time" for the Endpoints removal to propagate.

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 30

// Example: handling SIGTERM in the app (Go)
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM)
<-sigCh
server.Shutdown(context.Background()) // drain in-flight connections
os.Exit(0)

7. Deployment Strategy

The default rollout strategy (RollingUpdate) isn't always safe left at its defaults, and for high-risk changes, a plain rolling update isn't enough to catch problems early.

• Bad: Using the default maxUnavailable: 25% for a critical service. You could lose up to a quarter of capacity during a rollout at peak hours.
• Good: For critical services, set maxUnavailable: 0 so capacity never drops, use maxSurge to spin up new Pods before killing old ones. For high-risk changes, consider canary/blue-green.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

# A simple canary: run 2 Deployments side by side, split traffic via Service weight
# (or use Argo Rollouts / Flagger for automated percentage-based canary)

8. Secrets Handling

Secrets passed as environment variables (env) are easy to leak via kubectl describe pod, crash dump logs, or a child process inheriting the parent's entire environment.

• Bad:

env:
  - name: DB_PASSWORD
    value: "supersecret123"   # hardcoded straight into the manifest

• Good: Mount secrets as a volume (not exposed when listing env vars), and for larger systems, use an external secret store (Vault, AWS Secrets Manager) via the External Secrets Operator instead of plain Kubernetes Secrets (a Secret is only base64-encoded by default, not actually encrypted unless you've enabled encryption at rest).

volumes:
  - name: db-secret
    secret:
      secretName: db-credentials
containers:
  - name: shop-api
    volumeMounts:
      - name: db-secret
        mountPath: /etc/secrets
        readOnly: true

9. Labels & Metadata

Without consistent labeling, troubleshooting, per-team cost tracking, and writing accurate selectors all get harder as the cluster grows.

• Bad: Ad-hoc labels, every service with its own naming style.
• Good: Follow Kubernetes' recommended labels.

metadata:
  labels:
    app.kubernetes.io/name: shop-api
    app.kubernetes.io/instance: shop-api-production
    app.kubernetes.io/version: "1.4.0"
    app.kubernetes.io/component: backend
    app.kubernetes.io/part-of: shop-platform
    app.kubernetes.io/managed-by: helm

10. Image Tag & Pull Policy

:latest is a "tag that means nothing": nobody can be sure which version is actually running, and rollback is nearly impossible since you don't know what the "previous version" even was.

• Bad: image: myregistry.io/shop-api:latest
• Good: Pin a specific tag (produced by your own CI/CD), and for higher-assurance systems, pin by sha256 digest for true immutability.

containers:
  - name: shop-api
    image: myregistry.io/shop-api:1.4.0
    imagePullPolicy: IfNotPresent   # or Always if the tag can be overwritten (not recommended, but Always is mandatory if it can)

11. ConfigMap & Reload Strategy

Editing a ConfigMap doesn't automatically make running Pods pick up the new values. If your app only reads config at startup, you'll "fix it" and think you're done, but the Pods are still running the old config.

• Bad: Edit the ConfigMap, assume it's done, not realizing the Pod needs a restart to pick up the new value.
• Good: Have a clear strategy: either the app watches the config file and reloads it itself, or use a checksum annotation to force a new rollout whenever the ConfigMap changes (a common pattern with Helm).

# Helm: checksum annotation forces a Pod restart when the ConfigMap changes
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Why does the annotation have to live under spec.template.metadata? A Deployment manifest actually has two metadata blocks: one at the Deployment level, one inside spec.template (which describes the Pod). The controller only rolls out when spec.template changes, since that's the part hashed into pod-template-hash. Putting the annotation under the outer metadata triggers nothing. checksum/config has no special meaning to Kubernetes, it's just a Helm convention for forcing the annotation's value to change whenever the ConfigMap's content changes.

Suggestion: use Stakater Reloader instead of hand-writing a checksum into every chart. Reloader watches ConfigMaps/Secrets itself and patches a similar annotation when it detects a change, no chart edits needed:

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

Note: if you're using ArgoCD with selfHeal: true, Reloader's direct patch can get reverted by ArgoCD, which sees it as drift from Git. In that case, prefer the checksum annotation approach above.

---

Part 3: Going Live, Final Checks

Before hitting release for real, run through these 4 questions:

• Have you load-tested the scaling path? Configuring HPA/Cluster Autoscaler doesn't mean it behaves as expected under real load. Test with a 5-10x traffic spike.
• Rollback or roll-forward? Decide ahead of time, don't debate it mid-incident. Is kubectl rollout undo ready to go, or is the process fix-forward?
• Do you have a runbook? When a Pod crash-loops, when a node dies, does on-call know the first move without guessing?
• Have you reviewed cost & right-sizing? requests set way above real usage is silent waste that adds up to real money over a month.

---

Conclusion

This checklist isn't meant to be applied mechanically, 100% of every item, to every system, a personal side project doesn't need a 5-member etcd or canary deployments. But the closer you get to real production, with real users, the more deliberately you should work through each item instead of settling for "it runs."

Hope this checklist helps you feel more confident shipping workloads to Kubernetes production. Save it and start applying it to your systems.

Happy Coding! 👨‍💻

If you're running into technical challenges, need help with systems, DevOps tools, or career direction, I'm confident I can help. Get in touch at hoangviet.io.vn, I'd love to chat and collaborate with you.