September 10, 2026
11 min read
By Kokil Thapa | Last reviewed: September 2026
Kubernetes Troubleshooting: A Field Guide is what you reach for when a deployment looked fine in CI and then failed in production at 2 a.m. Pods stall in Pending. Services return 503 errors. Ingress routes stop resolving. The cluster API still answers, but your application does not. This guide gives you a repeatable workflow—symptoms first, then scope, then evidence—so you stop guessing and start fixing. If you run workloads alongside traditional Linux system administration stacks, the same discipline applies: observe, narrow, verify, document.
Quick answer: Kubernetes Troubleshooting: A Field Guide starts at the symptom—pod state, Service endpoints, Ingress rules—then runs
kubectl describe, kubectl logs, and kubectl get events to find the failing layer (scheduling, container, network, storage) before applying a targeted fix.What Is the First Step in Kubernetes Troubleshooting?
Start with the user-visible symptom, not the cluster control plane. A 502 from your load balancer and a pod stuck in CrashLoopBackOff need different paths. Write down three facts before you touch kubectl: which namespace, which resource name, and when the failure started.
Then classify the failure layer. Scheduling problems live at the node and scheduler level. Container problems show up in pod status and logs. Network problems appear when pods run but traffic never arrives. Storage problems surface as mount failures or I/O errors inside the container.
Capture context in sixty seconds
Run these commands and save the output. They answer most first-pass questions without editing anything live.
kubectl config current-context
kubectl get nodes
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by='.lastTimestamp' | tail -20If only one namespace matters, add -n production to every command. Namespace scoping keeps noise down on busy clusters. I've seen teams waste an hour debugging the wrong environment because the context was still set to staging.
Which kubectl Commands Should You Run First?
Think of kubectl as your stethoscope. You listen before you operate. The official kubectl reference documents dozens of verbs, but five cover most incidents.
kubectl get— current state of pods, services, deployments, ingresses.kubectl describe— conditions, events, and scheduler messages attached to a resource.kubectl logs— stdout/stderr from the failing container.kubectl exec— live inspection when the container stays up long enough.kubectl debug— ephemeral debug containers on Kubernetes 1.23+ without rebuilding images.
For a pod named api-7f9c8 in namespace production, this sequence usually reveals the root cause within minutes.
kubectl -n production describe pod api-7f9c8
kubectl -n production logs api-7f9c8 --previous
kubectl -n production logs api-7f9c8 -c app --tail=200
kubectl -n production get events --field-selector involvedObject.name=api-7f9c8Paste large JSON payloads into a JSON formatter when you pull raw API output. Readable structure saves time when you compare two failing pods side by side.
Read pod phases correctly
Pod phase is not the full story. Running with restarts still means trouble. Check READY columns and restart counts together.
kubectl -n production get pod api-7f9c8 -o jsonpath='{.status.containerStatuses[*].state}{"\n"}'waiting.reason values like CrashLoopBackOff, ImagePullBackOff, and CreateContainerConfigError each point to a different fix path. Our dedicated walkthrough on debugging CrashLoopBackOff in Kubernetes covers the container exit path in detail.
How Do You Fix Pods Stuck in Pending or CrashLoopBackOff?
Pending means the scheduler has not placed the pod on a node. CrashLoopBackOff means the kubelet started the container, it exited, and Kubernetes is backing off retries. Treat them as separate playbooks.
Pending pods: scheduling and resources
Run kubectl describe pod and scroll to Events. Look for messages about insufficient CPU, memory, persistent volume claims, taints, or node selectors.
- Insufficient resources — raise node capacity, lower resource requests and limits, or enable cluster autoscaling.
- PVC not bound — check StorageClass, provisioner health, and whether the claim exists in the same namespace.
- Taints without tolerations — review taints and tolerations on node pools dedicated to GPU or batch workloads.
- Topology spread — hard spread constraints can block scheduling when zones are uneven.
kubectl describe pod api-7f9c8 -n production | grep -A5 Events
kubectl get pvc -n production
kubectl describe nodes | grep -E 'Taints|Allocatable'CrashLoopBackOff: application and config
Containers crash for three recurring reasons: bad entrypoint, missing environment variables or secrets, and failed health probes. Always check the previous container log first. The current instance may not have written anything useful yet.
kubectl -n production logs api-7f9c8 --previous
kubectl -n production get secret db-credentials -o yaml
kubectl -n production describe pod api-7f9c8 | grep -A3 LivenessOn Laravel workloads I have moved to Kubernetes, misconfigured APP_KEY, database URLs, or cache drivers caused immediate exits. The Kubernetes for Laravel getting started guide lists the env vars worth validating before you chase cluster-level issues.
How Do You Troubleshoot Kubernetes Networking and Ingress?
Networking failures follow a chain: Pod → Endpoints → Service → Ingress → external load balancer. Break the chain at the first broken link. The Kubernetes networking model article explains why each hop matters.
Start inside the cluster with a temporary debug pod.
kubectl run tmp-curl --rm -it --image=curlimages/curl -- sh
curl -v http://api-service.production.svc.cluster.local:8080/healthIf in-cluster curl works but external traffic fails, the problem is likely Ingress, DNS, or the cloud load balancer—not the application container.
Verify Service endpoints
kubectl -n production get svc api-service -o wide
kubectl -n production get endpoints api-service
kubectl -n production describe ingress api-ingressEmpty endpoints mean selectors do not match pod labels. That produces 503 errors even when pods look healthy. Compare spec.selector on the Service with labels on running pods.
Ingress misconfiguration is common after TLS certificate rotation or host rule changes. Cross-check your Ingress controller logs alongside the Ingress resource definition. For deeper packet-level issues, eBPF tooling covered in Cilium eBPF networking can trace drops that kubectl alone will not show.
What Tools Help With Advanced Kubernetes Troubleshooting?
Plain kubectl gets you through most incidents. When it does not, add focused tools rather than installing everything on the cluster at once.
| Tool | Best For | Trade-off |
|---|---|---|
kubectl debug | Ephemeral debug containers, node inspection | Requires RBAC; not all distros enable it |
stern | Tailing logs across many pods by label | Another binary to install locally |
k9s | Interactive cluster navigation | UI habits vary; not ideal for runbooks |
| Prometheus + Grafana | Metrics trends, saturation, latency | Needs upfront instrumentation |
| Falco | Runtime security events, suspicious syscalls | Alert tuning takes time |
| Velero | Backup verification before risky changes | Storage backend required |
For small teams running edge clusters, K3s simplifies the control plane but does not remove debugging discipline. Fewer moving parts mean fewer places to look, not zero failures.
Observability signals worth wiring early
Metrics tell you a pod is restarting. Logs tell you why. Traces show which downstream call failed. If you deploy through Argo CD GitOps, compare the live manifest with the last known-good Git commit before you patch by hand.
Horizontal scaling misconfiguration also masquerades as application failure. Review Horizontal Pod Autoscaling metrics sources when CPU looks fine but latency spikes under load.
How Do You Run a Production Incident Response on Kubernetes?
Incidents need a timeline, an owner, and a rollback path. Panic edits without either extend outages. On production deployments I maintain, we treat cluster changes like application releases: one change at a time, verified, logged.
Rollback before you refactor
When a Deployment fails after a new image tag, roll back first. Ask questions later.
kubectl -n production rollout history deployment/api
kubectl -n production rollout undo deployment/api
kubectl -n production rollout status deployment/apiIf the cluster change involved RBAC, network policies, or storage classes, undo that manifest too. Partial rollbacks leave you in a hybrid state that is harder to debug than the original failure.
Security-related regressions deserve the same urgency as uptime. A misapplied ClusterRoleBinding can expose secrets cluster-wide. The Kubernetes RBAC hardening guide lists the bindings worth auditing after any incident touching auth.
Performance and storage angles
Sometimes the pod runs but responses time out. Node disk pressure evicts pods silently if you are not watching. Check DiskPressure and MemoryPressure conditions on nodes.
kubectl describe nodes | grep -E 'Conditions:|Pressure'
kubectl top pods -n production
kubectl top nodesPersistent volume issues show up as mount timeouts or read-only filesystem errors. Review persistent volume lifecycle docs when PVCs flap between Bound and Lost. For booking platforms like Adventure Third Pole Trek, database-backed Laravel apps on Kubernetes need stable storage and correct connection pooling—not just healthy pod status.
When latency persists despite healthy probes, read Kubernetes performance tuning before you scale nodes blindly. Scaling hides inefficiency; it rarely fixes it.
Document what you changed
Every incident should produce a short runbook entry: symptom, commands run, root cause, fix, and prevention. Future you—or the next contractor—will hit the same error at Dashain peak traffic. Good notes beat heroic memory every time.
Ongoing cluster care belongs in the same bucket as application maintenance. If your team lacks dedicated platform engineers, support and maintenance or enterprise application development partnerships can cover runbooks, upgrades, and on-call escalation without a full internal SRE hire.
Key Takeaways
- Classify the failure layer—scheduling, container, network, or storage—before changing manifests.
- Run
kubectl describe,logs --previous, and namespace-scoped events on every pod incident. - Trace networking from in-cluster curl outward through Service endpoints and Ingress rules.
- Roll back Deployments first during outages; diagnose root cause after traffic recovers.
- Wire metrics, logs, and GitOps diffs early so incidents start with evidence, not guesses.
- Update runbooks after every production fix so Kubernetes Troubleshooting: A Field Guide stays current for your team.
People Also Ask
Why is my Kubernetes pod stuck in Pending?
Pending usually means the scheduler cannot place the pod. Check for insufficient CPU or memory requests, unbound PVCs, node taints, mismatched node selectors, or topology spread constraints that no node satisfies. kubectl describe pod events name the exact blocker.
How do I debug CrashLoopBackOff quickly?
Read kubectl logs podname --previous first. Then verify secrets, config maps, environment variables, and probe settings. Most crash loops are application exit codes or misconfigured health checks—not cluster failures.
What is the difference between a Pod and a Service error?
Pod errors affect one container instance—image pull failures, OOM kills, probe failures. Service errors mean traffic routing broke—empty endpoints, wrong selectors, or ports that do not match container declarations. Test with in-cluster curl before blaming the app.
When should I escalate from kubectl to platform tools?
Escalate when metrics show a trend kubectl snapshots miss, when network drops happen inside the CNI layer, or when you need backup restore before a risky fix. Tools like Prometheus, Falco, and Velero complement—not replace—the basic describe-and-logs workflow.
Build a Troubleshooting Culture, Not Just a Cheat Sheet
Kubernetes Troubleshooting: A Field Guide works when your team repeats the same evidence-first steps under pressure. Symptom, scope, describe, logs, one fix, verify—that rhythm saves hours on every outage. Start with the kubectl baseline in this article, link it to your GitOps repo and monitoring dashboards, and keep runbooks next to the services they protect.
If you are shipping Laravel, eCommerce, or API workloads and want the cluster plus the application maintained together, review our web development services or reach out through contact us to plan a production-ready deployment with documented incident playbooks.
Frequently Asked Questions
Start with the user-visible symptom, not the cluster control plane. Before running kubectl, record three facts: namespace, resource name, and when the failure started. A load balancer 502 and a pod in CrashLoopBackOff follow different paths. Classify whether the problem sits in scheduling, the container, networking, or storage. This symptom-first, layer-mapping approach stops you from guessing and gives every later command a clear purpose.
Treat kubectl as a stethoscope: listen before you operate. Five verbs cover most incidents. kubectl get shows current pod, Service, Deployment, and Ingress state. kubectl describe exposes conditions, events, and scheduler messages. kubectl logs captures stdout and stderr. kubectl exec inspects a running container. kubectl debug attaches ephemeral debug containers on Kubernetes 1.23 and later without rebuilding images. For a failing pod, run describe, logs with --previous, current logs with a tail limit, then namespace-scoped events.
Pending means the scheduler has not placed the pod on any node. Run kubectl describe pod and read the Events section. Common blockers include insufficient CPU or memory requests, an unbound persistent volume claim, node taints without matching tolerations, mismatched node selectors, or topology spread constraints no node can satisfy. Check PVC status in the same namespace, review node taints and allocatable resources, and compare spread rules against zone capacity before changing application code.
Read kubectl logs with --previous first—the current instance may not have written anything useful yet. Then check secrets, ConfigMaps, environment variables, and liveness probe settings.
Pod errors affect one container instance—image pull failures, OOM kills, or probe failures. Service errors mean traffic routing broke—empty endpoints, wrong selectors, or port mismatches. Test in-cluster before blaming the app.
Trace the chain from Pod to Endpoints to Service to Ingress to the external load balancer, and break at the first broken link. Start inside the cluster with a temporary curl pod hitting the Service DNS name and health path. If in-cluster curl succeeds but external traffic fails, suspect Ingress rules, DNS, or the cloud load balancer—not the application container. Cross-check Service selectors against pod labels, verify endpoints are populated, and review Ingress controller logs after TLS certificate rotation or host rule changes.
When kubectl get endpoints returns nothing for a Service, selectors do not match running pod labels. Traffic still hits the Service object, but nothing receives it—producing 503 errors even when pods look healthy in kubectl get pods. Compare spec.selector on the Service with labels on your running pods. Fix the label mismatch or update the Service selector. Always verify endpoints after deploys that change labels, because pod phase Running alone does not prove routing works.
Stay on kubectl until snapshots miss the story. Escalate when metrics show trends describe and logs cannot explain, when packet drops happen inside the CNI layer that kubectl alone will not reveal, or when you need backup verification before a risky change. Prometheus and Grafana expose saturation and latency trends. Falco surfaces suspicious runtime syscalls. Velero confirms backups before destructive fixes. stern tails logs across many pods by label. These tools complement—not replace—the describe-and-logs baseline every incident should start with.
Capture context before editing anything live. Run kubectl config current-context to confirm you are not debugging staging by mistake—a mistake I have seen waste an hour on busy clusters. Then kubectl get nodes, kubectl get pods across all namespaces filtered to non-Running phases, and kubectl get events sorted by last timestamp, keeping the last twenty lines. If one namespace matters, add -n production to every command. Save all output before deeper investigation.
Every incident needs a timeline, an owner, and a rollback path. Triage the symptom, apply one change at a time, verify recovery, then document for your runbook. Treat cluster changes like application releases—panic edits without ownership extend outages. When a Deployment fails after a new image tag, roll back first and ask questions later using rollout history and rollout undo. If the change touched RBAC, network policies, or storage classes, undo that manifest too. Partial rollbacks leave a harder-to-debug hybrid state than the original failure.
Roll back first when a Deployment fails after a new image tag. Use kubectl rollout history, rollout undo, and rollout status to restore the last known-good revision before root-cause analysis. Traffic recovery beats understanding mid-outage. Once stable, investigate why the new image or manifest failed. The same urgency applies to security regressions—a misapplied ClusterRoleBinding can expose secrets cluster-wide and deserves immediate reversal, not post-mortem curiosity while exposure continues.
Plain kubectl handles most incidents. When it does not, add focused tools rather than installing everything at once. kubectl debug supports ephemeral debug containers and node inspection but requires RBAC and is not enabled on every distribution. stern tails multi-pod logs by label. k9s offers interactive navigation though habits vary for runbooks. Prometheus plus Grafana need upfront instrumentation but reveal restart trends and latency. Falco flags runtime security events with tuning overhead. Velero verifies backups before risky changes. For small edge clusters, K3s simplifies the control plane but does not remove debugging discipline.
Map each symptom to scheduling, container, network, or storage before changing manifests. Scheduling problems appear at the node and scheduler level—Pending pods, resource shortages, taints, unbound PVCs. Container problems surface in pod status, restart counts, and logs—CrashLoopBackOff, ImagePullBackOff, CreateContainerConfigError. Network problems show when pods run but traffic never arrives—empty endpoints, Ingress misconfiguration, DNS failures. Storage problems appear as mount failures, read-only filesystem errors, or PVCs flapping between Bound and Lost. Running with restarts still means trouble; check READY columns together with restart counts.
A pod in Running phase can still produce 503 responses at the load balancer when the Service layer breaks. Empty endpoints mean selectors no longer match pod labels after a deploy changed labels without updating the Service. Port mismatches between Service spec and container declarations block traffic even when probes pass internally. Verify endpoints with kubectl get endpoints, compare selectors to pod labels, and test with in-cluster curl before assuming the application crashed. Ingress misconfiguration after TLS rotation can also block external traffic while internal health checks succeed.
Every incident should produce a short runbook entry covering symptom, commands run, root cause, fix, and prevention. Future you—or the next contractor—will hit the same error at peak traffic; good notes beat heroic memory. Compare live manifests against the last known-good Git commit when deploying through Argo CD GitOps before patching by hand. Update runbooks after every production fix so your field guide stays current. Ongoing cluster care belongs alongside application maintenance, especially when no dedicated platform engineer owns on-call escalation.
0 Comments
Leave a comment
Your email is not published. Comments appear once they have been read. Sign in to have your details filled in.
