September 11, 2026
12 min read
By Kokil Thapa | Last reviewed: September 2026
Production failures rarely wait for a convenient deploy window. When a Laravel API, WooCommerce worker, or queue consumer misbehaves inside a container, you need to debug a running container without tearing down the only copy that still holds useful state. That means reading live logs, inspecting processes, checking network paths, and sometimes attaching tools—all while traffic keeps flowing. On real client projects I maintain with Docker Compose multi-container setups, most incidents are solved from the outside in: identify the container, confirm it is still running, then narrow the fault with targeted commands rather than a full rebuild.
Quick answer: To debug a running container, use
docker exec or kubectl exec for an interactive shell, stream logs with docker logs -f or kubectl logs -f, inspect processes with ps and top, and add ephemeral debug sidecars with kubectl debug when the main image lacks tools.What is the fastest way to debug a running container?
Start with identity and health. A wrong container ID wastes minutes. List running units, confirm restart count, then open logs before you change anything.
Docker: confirm the target container
docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Status}}\t{{.Ports}}"
docker inspect --format '{{.State.Status}} restarts={{.RestartCount}}' my-app
docker logs --tail 200 -f my-appThe -f flag follows new log lines in real time. That alone resolves many PHP-FPM, Node, or nginx misconfigurations. If the container keeps restarting, switch to CrashLoopBackOff debugging patterns—a running container and a crash loop need different playbooks.
Kubernetes: map pod to node and container
kubectl get pods -n production -o wide
kubectl describe pod my-app-7f8b9c -n production
kubectl logs my-app-7f8b9c -n production -c app -f --tail=200describe surfaces events: image pull failures, OOMKilled, failed probes. Events explain why a container looks "running" yet serves errors. The runtime underneath—Docker Engine, containerd, or CRI-O—does not change this first step.
Keep a one-page runbook on your team wiki. Include namespace, label selectors, and log aggregation URLs. During an outage, typing beats searching.
How do you inspect processes and logs inside a running container?
Logs tell you what happened. Processes tell you what is happening now. You need both to debug a running container effectively.
Interactive shell with docker exec
docker exec -it my-app sh
ps aux
top -bn1 | head -20
cat /proc/1/cmdline | tr '\0' ' '
ls -la /var/log/Use sh when bash is not installed. Alpine-based PHP and Node images often ship without bash. The official Docker documentation for docker exec covers TTY allocation and user context flags.
Run one-off commands without a shell
docker exec my-app php artisan queue:monitor
docker exec my-app curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1/health
docker exec -u www-data my-app ls -la storage/logsOne-off commands reduce risk. You do not leave an open shell on a production box. For Laravel apps, I often compare container behaviour with safe Laravel production debugging practices on the host or sidecar log shipper.
Structured log parsing
JSON logs belong in a parser, not your terminal scrollback. Pipe a sample through a JSON formatter to spot missing fields or wrong log levels. Correlation IDs should appear in both HTTP access logs and application logs.
- Capture 50–100 lines around the incident timestamp.
- Filter by request ID, user ID, or job ID.
- Cross-check database slow-query logs if latency spiked.
- Confirm log rotation is not deleting evidence mid-incident.
Resource inspection while running
docker stats my-app --no-stream
docker exec my-app df -h
docker exec my-app free -m
docker top my-app auxMemory pressure inside a container often triggers OOM kills at the cgroup level. Pair docker stats with container resource limits documentation so you know whether the limit or the leak came first.
How do you attach a debugger to a container without stopping traffic?
Stopping the container destroys ephemeral state. Debug attach techniques preserve the running process while you gather evidence.
Copy files out instead of editing in place
docker cp my-app:/var/log/app.log ./incident-app.log
docker cp my-app:/app/storage/logs/laravel.log ./laravel.log
kubectl cp production/my-app-7f8b9c:/tmp/heap.dump ./heap.dump -c appPull artifacts to your laptop for analysis. Do not install editors or strace inside production images during an incident. That changes the image layer cache and invites drift.
Non-invasive tracing
docker exec my-app strace -p 1 -f -e trace=network -s 80 2>&1 | head -50
docker exec my-app lsof -i -P -n | headstrace adds overhead. Use short samples on busy APIs. If strace is missing, that confirms you should bake a debug variant tag or use an ephemeral toolkit container. Rootless containers may restrict some tracing capabilities—plan accordingly.
Debug image tags for PHP and Node
A pattern I use on production Laravel systems: build two tags from the same Dockerfile stage.
my-app:2026-09-11— slim runtime, no dev tools.my-app:2026-09-11-debug— addsbash,curl,strace, andvim-tiny.
Swap the tag only on a canary node during investigation. Never publish the debug tag as your default latest. Scan both with Trivy image scanning before deploy.
Live REPL and remote debug ports
For languages with remote debuggers, expose the debug port on an internal network only. Bind to 127.0.0.1 inside the container and reach it through kubectl port-forward or an SSH tunnel. Never expose port 9003 or 9229 on a public load balancer.
kubectl port-forward pod/my-app-7f8b9c 9003:9003 -n productionRemove debug ports from your Helm chart defaults. Enable them with an explicit incident flag that expires.
When should you use kubectl debug vs docker exec?
Docker exec assumes your image contains a shell and tools. Kubernetes ephemeral containers exist for when that assumption breaks.
| Scenario | docker exec / kubectl exec | kubectl debug (ephemeral) |
|---|---|---|
| Alpine app image with shell | Preferred — fast | Overkill |
| Distroless / scratch image | Fails — no shell | Preferred — inject toolkit |
| Need tcpdump / netshoot | Only if tools pre-installed | Add netshoot sidecar |
| Read-only root filesystem | Read-only exec OK | Writable debug volume if needed |
| Single Docker host | Native docker exec | Not applicable |
| Managed EKS/GKE/AKS | Via kubectl exec | Ephemeral debug container |
Ephemeral debug container example
kubectl debug -it my-app-7f8b9c -n production \
--image=nicolaka/netshoot:latest \
--target=app \
-- shThe --target flag joins the PID namespace of the app container. You can inspect localhost sockets and hit the same loopback ports. The Kubernetes task guide on debugging running pods documents copy-pod and node debugging variants as well.
On Docker Compose stacks, the equivalent is a one-shot sidecar service on the same network alias. For ECS Fargate tasks, use a second non-essential container in the task definition reserved for break-glass access. See ECS Fargate deployment patterns for task networking notes.
How do you debug network and DNS issues in running containers?
Application logs say "connection refused." Network tools tell you whether DNS, routing, or TLS caused it. This is where many container debug sessions actually spend their time.
Connectivity checks from inside the workload
docker exec my-app getent hosts db.internal
docker exec my-app curl -v telnet://redis:6379
docker exec my-app wget -qO- --timeout=3 http://api:8080/healthService names resolve differently on Docker bridge networks, Compose project networks, and Kubernetes cluster DNS. Confirm which resolver the container uses: cat /etc/resolv.conf.
Packet capture without rebuilding the app
kubectl debug my-app-7f8b9c -n production \
-it --image=nicolaka/netshoot \
--target=app -- tcpdump -i any port 443 -c 20Keep capture windows short. Store pcaps off the node when possible. On booking platforms like Adventure Third Pole Trek, intermittent payment callback failures often trace to DNS TTL or firewall rules—not application code.
Compare container vs host networking
Host network mode bypasses Docker NAT. Kubernetes hostNetwork pods behave similarly. If the issue disappears with host networking, suspect port mapping or ingress controller misconfiguration. Document your ingress path from load balancer to pod IP.
Environment and secrets verification
docker exec my-app env | sort | grep -E 'APP_|DB_|REDIS_'
kubectl exec my-app-7f8b9c -n production -c app -- printenv DATABASE_URLRedact output before sharing in tickets. Wrong APP_URL values cause bad redirects. Stale Redis URLs cause silent queue loss. Compare against your secret manager source of truth, not last week's deploy notes.
What production practices keep container debugging safe?
Break-glass access is not a free pass to mutate production. Guardrails keep debugging from becoming the next outage.
Observability before SSH envy
Centralise logs and metrics first. OpenTelemetry traces beat manual exec for recurring issues. Pair container tooling with AI-assisted debugging workflows only after your log pipeline is trustworthy. Models hallucinate on missing data.
RBAC and audit trails
Grant kubectl exec narrowly. Log who exec'd into which pod. On Docker hosts managed for clients, I restrict SSH and require jump hosts. Document every production exec in the incident ticket. Your future self reads that trail during postmortems.
Registry and supply chain hygiene
Debug images still flow through CI. Push them to the same private Harbor registry as production tags. Sign images if your policy requires it. The debug tag must never become the deploy default in GitLab CI variables.
When to escalate to platform support
Some problems sit below your app: CRI socket errors, kubelet disk pressure, or EC2 ENI limits. If exec works but traffic never reaches the pod, hand off to Linux system administration or your cloud provider with node-level evidence. Application developers should not spend an hour fixing cgroup v2 memory settings during a checkout outage.
For long-running maintenance contracts, combine container debug playbooks with support and maintenance SLAs so clients know what break-glass access includes. Enterprise teams building internal platforms benefit from baking these patterns into enterprise application development standards early.
Key Takeaways
- Always list containers, check restart counts, and tail logs before you exec—most faults show up in the last 200 lines.
- Use
docker execorkubectl execfor shells; usekubectl debugephemeral containers when the image is distroless or read-only. - Copy logs and heap dumps out with
docker cporkubectl cpinstead of installing tools into running production containers. - Debug network issues from inside the same network namespace with
curl,getent hosts, and shorttcpdumpcaptures. - Maintain a separate debug image tag, scan it like production, and restrict who can deploy it.
- Document every production exec session and feed findings back into observability so the next incident needs less shell access.
People Also Ask
Can you debug a container without stopping it?
Yes. That is the normal production path. Commands like docker exec, live log streaming, docker stats, and Kubernetes ephemeral debug containers inspect a running container while your app process keeps serving traffic. Restart only when you must pick up a fixed image or config.
Why does docker exec say "executable file not found"?
The shell binary is missing from the image. Distroless, scratch, and minimal Alpine builds often omit bash and sometimes sh. Try docker exec my-app /busybox sh if busybox exists, or inject a debug sidecar on Kubernetes instead of rebuilding under pressure.
How do you debug a running Docker Compose service?
Run docker compose ps, then docker compose logs -f service_name, then docker compose exec service_name sh. Compose service names resolve on the internal network, so test dependencies with curl http://other_service:port/health from inside the app container.
Is kubectl exec the same as SSH into a pod?
Similar outcome, different mechanism. kubectl exec asks the kubelet to start a process inside an existing container namespace. It respects RBAC, needs no SSH daemon, and leaves an API audit trail. SSH to the node is a separate break-glass path and usually unnecessary for app-level debugging.
Build debug-friendly containers from day one
You will debug a running container sooner or later. Slim images save bandwidth, but zero tools save nobody during a 2 a.m. incident. Ship observability, a documented exec playbook, and an ephemeral debug path for distroless workloads. If your team runs Laravel, WordPress, or custom APIs on Docker or Kubernetes and wants production discipline without theatre, review the Adventure Himalaya Nepal portfolio entry for how staged deploys and logging fit real ops—or reach out via contact us to audit your container workflow. For greenfield platforms, custom software development engagements can embed these patterns before the first production deploy. Read more container ops guides on the blog, learn about the author on about me, and study the Container Runtime Interface when exec failures point at the node—not your app.
Frequently Asked Questions
Start with identity and health, not a shell. List running containers, confirm restart count, then tail the last 200 log lines with follow mode before changing anything. On Docker use docker ps, docker inspect for State.Status and RestartCount, then docker logs --tail 200 -f. On Kubernetes use kubectl get pods, kubectl describe pod for events like OOMKilled or probe failures, then kubectl logs -f. Most PHP-FPM, Node, and nginx faults show up in recent logs alone.
Yes. That is the normal production path for live incidents.
The shell binary is missing from a minimal image. Distroless, scratch, and some Alpine builds omit bash or even sh. Try docker exec my-app /busybox sh if busybox exists. On Kubernetes, use kubectl debug with an ephemeral toolkit image such as nicolaka/netshoot instead of rebuilding the app image under pressure. Plan a separate debug image tag for future incidents.
Logs show what happened; processes show what is running now. Open an interactive shell with docker exec -it my-app sh, then run ps aux, top -bn1, and check /var/log. Prefer one-off commands like docker exec my-app php artisan queue:monitor or curl against http://127.0.0.1/health to avoid leaving shells open. Pair application logs with docker stats, df -h, and free -m to catch memory pressure that triggers cgroup OOM kills before the app logs explain it.
Use docker exec or kubectl exec when the image already has a shell and basic tools — that is the fast path for Alpine apps with sh installed. Switch to kubectl debug ephemeral containers when the image is distroless, scratch, read-only root filesystem, or lacks tcpdump and network utilities. The --target flag joins the app container PID namespace so you inspect localhost sockets from a netshoot sidecar. On plain Docker Compose, run a one-shot sidecar on the same network alias instead.
Start inside the same network namespace as the app. Run getent hosts db.internal to confirm DNS resolution, curl -v telnet://redis:6379 for TCP reachability, and wget against http://api:8080/health for HTTP paths. Check cat /etc/resolv.conf because Docker bridge, Compose project networks, and Kubernetes cluster DNS resolve service names differently. For packet-level proof, attach nicolaka/netshoot with kubectl debug and run a short tcpdump capture. If host networking makes the issue disappear, suspect port mapping or ingress misconfiguration.
Never stop the workload if it holds useful ephemeral state. Copy artifacts out with docker cp or kubectl cp — logs, Laravel storage logs, heap dumps — and analyse them locally instead of installing editors or strace into the running production layer. For live tracing, run short strace or lsof samples via docker exec, accepting overhead on busy APIs. For remote debug ports such as 9003 or 9229, bind inside the container to 127.0.0.1 and reach them through kubectl port-forward or an SSH tunnel, never a public load balancer.
Run docker compose ps to confirm the service is up, docker compose logs -f service_name to stream live output, then docker compose exec service_name sh for an interactive session. Compose service names resolve on the internal project network, so test dependencies from inside the app container with curl http://other_service:port/health. Follow the same outside-in order as Kubernetes: verify identity, check restart behaviour, read logs, exec only when logs are insufficient. Keep namespace, label selectors, and log URLs in a one-page runbook.
Similar outcome, different mechanism and risk profile. kubectl exec asks the kubelet to start a process inside an existing container namespace. It respects RBAC, needs no SSH daemon inside the pod, and leaves an API audit trail of who accessed what. SSH to the underlying node is a separate break-glass path for platform-level problems such as kubelet disk pressure or CRI socket errors. For application faults — wrong env vars, queue stalls, bad redirects — exec from the API layer is usually enough.
A crash loop needs a different playbook than a steady running container. Check RestartCount with docker inspect or kubectl describe pod events for image pull failures, OOMKilled, and failed liveness or readiness probes. Tail logs with follow mode, but also read describe output because a container can look running yet serve errors when probes or events tell a different story. Switch from exec-focused debugging to CrashLoopBackOff patterns: capture the last logs before exit, verify resource limits, and confirm the image tag matches what CI actually pushed.
Run docker exec my-app env | sort | grep -E 'APP_|DB_|REDIS_' or kubectl exec with printenv DATABASE_URL against the live workload, then compare output to your secret manager source of truth — not last week's deploy notes. Redact values before pasting into tickets. Wrong APP_URL values cause bad redirects; stale Redis URLs cause silent queue loss. Treat env inspection as read-only evidence gathering. Do not mutate secrets inside a running container during an incident unless your runbook explicitly allows it and you document the change.
Build two tags from the same Dockerfile stage: a slim runtime tag with no dev tools, and a debug tag adding bash, curl, strace, and vim-tiny. Swap the debug tag only onto a canary node during investigation. Never publish the debug tag as default latest or as the GitLab CI deploy default. Scan both tags with Trivy before deploy, push both to the same private Harbor registry, and sign images if policy requires. Remove debug ports from Helm chart defaults and enable them with an explicit incident flag that expires.
Centralise logs and metrics first — OpenTelemetry traces beat manual exec for recurring issues. Grant kubectl exec narrowly, log who exec'd into which pod, and document every production exec in the incident ticket. Copy files out instead of editing in place inside running images. Restrict SSH on Docker hosts and require jump hosts for client-managed servers. When exec works but traffic never reaches the pod, escalate to platform support with node-level evidence rather than spending an hour on cgroup memory settings during a checkout outage.
Missing strace confirms you need a planned debug path, not an improvised install during the outage. Options are a pre-built debug image tag swapped onto a canary, a kubectl debug ephemeral container with nicolaka/netshoot sharing the app PID namespace, or a one-shot sidecar on the same Docker Compose network. Rootless containers may restrict some tracing capabilities, so account for that in your runbook. Never install vim, tcpdump, or editors into the production layer mid-incident — that changes image cache state and invites drift.
Escalate when the fault sits below the application layer. If kubectl exec succeeds but traffic never reaches the pod, suspect ingress paths, ENI limits, kubelet disk pressure, or CRI socket errors on the node. Application developers should hand off cgroup v2 memory settings, runtime failures, and cloud networking blocks with evidence: describe events, short packet captures stored off-node, and docker stats or node metrics from the incident window. Combine your container exec playbook with maintenance SLAs so clients know what break-glass access includes versus platform support.
0 Comments
Leave a comment
Your email is not published. Comments appear once they have been read. Sign in to have your details filled in.
