Kubernetes Pod Debugging & Cluster Troubleshooting Playbook

A Systematic Approach to K8s Incident Response

When microservices in a Kubernetes cluster enter CrashLoopBackOff, OOMKilled, or ImagePullBackOff states, following a structured diagnostic sequence saves critical recovery time.

1. Rapid Pod Inspection Sequence

# Step 1: List failing pods across all namespaces
kubectl get pods --all-namespaces --field-selector=status.phase!=Running

# Step 2: Describe the failing pod to inspect event history
kubectl describe pod auth-service-7f6d9c8b-x29q --namespace production

# Step 3: Fetch container logs including previous crash instances
kubectl logs auth-service-7f6d9c8b-x29q --namespace production --previous --tail=100

2. Common Failure Modes & Resolution

• CrashLoopBackOff: Application crashed on startup. Check missing environment variables or failed database connectivity.
• OOMKilled (Exit Code 137): Pod exceeded memory resource limit. Increase resources.limits.memory.
• Pending: Node affinity mismatch, insufficient CPU/Memory on worker nodes, or PVC binding issues.

3. Ephemeral Debug Containers

Attach a temporary debug container with network diagnostics tools to a running pod without restarting it:

kubectl debug -it auth-service-7f6d9c8b-x29q 
    --image=nicolaka/netshoot 
    --target=auth-container 
    --namespace=production