CrashLoopBackOff Troubleshooting Guide

CrashLoopBackOff is one of the most common Pod failure states in Kubernetes. When a container repeatedly crashes and exceeds the backoff limit defined by restartPolicy, kubelet enters an exponential backoff restart cycle, with wait times increasing from 10 seconds up to 5 minutes.

Symptom Identification

When a Pod is in CrashLoopBackOff state, you will observe the following:

• kubectl get pods shows STATUS as CrashLoopBackOff with the RESTARTS count continuously increasing
• kubectl describe pod Events repeatedly show Back-off restarting failed container
• The Pod exits within seconds of each startup, then enters progressively longer wait cycles
• The service is unavailable, and the related Deployment’s Ready replica count is below the desired count

# Quickly list all CrashLoopBackOff Pods across all namespaces
kubectl get pods -A | grep CrashLoopBackOff

# View restart history for a specific Pod
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].lastState}'

Root Cause Analysis

CrashLoopBackOff is fundamentally a container process exiting with a non-zero exit code. Common root causes, ordered by frequency:

1. Application startup failure — missing configuration files, unset environment variables, database connection timeouts, unreachable dependency services
2. OOMKilled — container memory usage exceeds resources.limits.memory, terminated by the kernel OOM Killer (exit code 137)
3. Health check failure — Liveness Probe misconfigured, application killed before it becomes ready
4. Permission issues — SecurityContext restrictions prevent the container from accessing required resources, such as write failures on a read-only filesystem
5. Image issues — incorrect ENTRYPOINT script in the image, missing dependency libraries, architecture mismatch
6. Insufficient resources — node CPU/memory exhaustion prevents Pod scheduling or causes repeated eviction and restart

Diagnostic Steps

Step 1: Obtain Exit Code and Recent Logs

The exit code is the most important clue for determining root cause.

# View container exit code
kubectl describe pod <pod-name> | grep -A5 "Last State"

# View logs from the previous crash (critical!)
kubectl logs <pod-name> --previous

# For multi-container Pods, specify the container name
kubectl logs <pod-name> -c <container-name> --previous

# View the last N lines
kubectl logs <pod-name> --previous --tail=100

Exit code quick reference:

Step 2: Check Resource Usage and Limits

# View Pod resource limits
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].resources}'

# View actual memory usage (requires metrics-server)
kubectl top pod <pod-name>

# View node resource pressure
kubectl describe node <node-name> | grep -A10 "Conditions"

Step 3: Check Liveness / Readiness Probe

# View Probe configuration
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].livenessProbe}' | jq .

# View Probe failure events
kubectl describe pod <pod-name> | grep -i "unhealthy\|probe\|liveness\|readiness"

Step 4: Debug In Place

If the container crashes immediately on startup and cannot be entered, use debug mode:

# Create a debug Pod with the same configuration but overridden entrypoint
kubectl debug <pod-name> -it --copy-to=<pod-name>-debug --container=<container-name> -- /bin/sh

# If the image has no shell, use an ephemeral container
kubectl debug -it <pod-name> --image=busybox --target=<container-name>

Remediation Steps

Choose the appropriate fix based on the diagnostic results:

Application Configuration Error

# Check if ConfigMap / Secret exists and has correct content
kubectl get configmap <name> -o yaml
kubectl get secret <name> -o yaml

# Verify environment variable injection
kubectl get pod <pod-name> -o jsonpath='{.spec.containers[*].env}' | jq .

After fixing the configuration, recreate the Pod:

kubectl delete pod <pod-name>
# The Deployment will automatically create a new Pod

OOMKilled

# Increase memory limits
kubectl patch deployment <deployment-name> -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"<container-name>","resources":{"limits":{"memory":"512Mi"},"requests":{"memory":"256Mi"}}}]}}}}'

Also investigate whether the application has a memory leak rather than simply increasing limits.

Misconfigured Probe

# Increase initialDelaySeconds to give the application more startup time
# Increase failureThreshold to tolerate occasional failures
kubectl patch deployment <deployment-name> --type=json -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/initialDelaySeconds","value":30},
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/failureThreshold","value":5}
]'

Verify the Fix

# Observe new Pod status
kubectl get pod -w

# Confirm Pod enters Running and Ready state
kubectl get pod <pod-name> -o wide

# Verify restart count is no longer increasing
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].restartCount}'

Preventive Measures

• Set appropriate resources.requests and resources.limits for all containers
• Liveness Probe initialDelaySeconds should exceed the application’s expected startup time
• Readiness Probe should only return success after the application is fully ready
• Use Pod Disruption Budget (PDB) to limit the number of simultaneously crashing Pods
• Include image startup tests in CI/CD pipelines to catch ENTRYPOINT issues early