Skip to main content
Lesson 11 of 11

Troubleshooting

Move from pod status to Events, then to logs, port-forward, and a shell inside the container.

The debugging loop

Do not change code or manifests at random. Work from the outside (Kubernetes) to the inside (your app):

Step Command Asks Look for
1. Overview kubectl get pods Which Pod is unhappy? STATUS and RESTARTS. Red flags: ImagePullBackOff, CrashLoopBackOff, Pending, Error, and climbing restarts. Running with 0 restarts is healthy.
2. Cluster report kubectl describe pod <name> What did Kubernetes try, and where did it fail? The Events section at the bottom (Warning or Failed): failed pulls, missing Secrets or ConfigMaps, and insufficient CPU or memory.
3. App logs kubectl logs <name> Why did the code crash? Stack traces, database connection errors, and unhandled exceptions.

From kubectl get pods, follow the status:

Status Next check
ImagePullBackOff or ErrImagePull describe, then read Events and fix the image name, tag, or pull secret
CrashLoopBackOff describe for the Last State exit code, then logs --previous
Pending describe for FailedScheduling: requests, nodeSelector, or taints
Running but misbehaving port-forward, then logs, then exec
Running with 0 restarts Healthy
Command Focus Example problem found
kubectl get High-level status A Pod stuck in Pending
kubectl describe Kubernetes infrastructure Failed to pull image "nginx:invalid-tag"
kubectl logs Inside the application Fatal error: Could not connect to the database at 10.0.0.5

Events

Events are Kubernetes’ internal log of what the cluster does: scheduling a Pod, pulling an image, mounting storage, or failing. A failed event usually contains the exact reason.

Scenario Command Why
You know which Pod is failing kubectl describe pod <name> Only that Pod’s events, with no noise. A healthy order is Scheduled → Pulling → Pulled → Created → Started. A failing Pod stops at the broken step.
Several things break, or the culprit is unknown kubectl get events --sort-by=.lastTimestamp A timeline across the whole namespace, with the newest events at the bottom.

Columns of kubectl get events are TYPE (Normal or Warning), REASON (Scheduled, Pulled, Failed, BackOff, and similar), and MESSAGE.

Kubernetes deletes events after about one hour. For older crashes, switch to kubectl logs.

Common failure states

Status Meaning Typical causes Diagnose Fix
ImagePullBackOff (after ErrImagePull) Image download failed, and Kubernetes waits before retrying A typo in the name or tag (nginx:alpne), a tag that does not exist (nginx:99.99), or a private registry without a pull secret get, then describe Events (Failed to pull image … manifest unknown). There are no logs because the container never started. kubectl set image deployment/<name> <container>=<correct-image>
CrashLoopBackOff The image was pulled and the container started, but the app crashed or exited. The restart delay grows The process exits immediately (a bare echo), missing env, config, or database settings, an app bug, a failing liveness probe, or OOMKilled get (high RESTARTS), then describe, then Last State exit code (1 is an app error, 137 is an OOM kill), then kubectl logs <pod> --previous Fix the app or config. Raise the memory limit if it was OOM
Pending Accepted, but no node can run it A request larger than any node’s free capacity (the most common cause), an unsatisfiable nodeSelector, or taints with no toleration describe, then Events: FailedScheduling … 0/1 nodes are available: 1 Insufficient memory. Logs will not work because nothing started. Lower requests, fix selector labels, or add node capacity

Why --previous matters: Kubernetes keeps restarting the crashed container, so plain kubectl logs may attach to a brand-new container that has not printed the crash yet. --previous fetches the logs of the last crashed instance.

Command Description
kubectl logs <pod> Current container’s logs
kubectl logs <pod> --previous Logs of the previously crashed instance
kubectl logs <pod> -c <container> Pick a container in a multi-container Pod

Pod is Running but the app misbehaves

If describe shows no errors but the app does not respond correctly, move from cluster-level checks to inspection inside the Pod.

Step 1: isolate the network with port-forward. Is the problem the app, or the routing layer (Service, Ingress, or selectors)? Tunnel straight to the Pod, bypassing Services and Ingress:

# Terminal 1
kubectl port-forward pod/web 8080:80
# Terminal 2
curl http://localhost:8080
  • Success means the Pod and app are healthy. Look at the Service: wrong selector labels or the wrong target port.
  • Failure or a hang means the problem is inside the app. Go to Step 2.

Step 2: read the output.

kubectl logs web
kubectl logs web -c web-container
kubectl logs web --previous

The first command reads the active container. -c selects one container in a multi-container Pod. --previous reads a crashed or restarted instance.

Step 3: inspect inside the container.

kubectl exec -it web -- sh
env | grep DB_HOST
cat /etc/app/config.json
nc -zv database-service 5432
exit

Check whether the expected environment variables are present, whether the config file is where the app expects it, and whether the Pod can reach downstream services.

Tool Question Command Use
port-forward Is it the app or the Service and network? kubectl port-forward pod/<name> <local-port>:<pod-port> Direct test that isolates the Pod from the Service layer
logs What did the app say? kubectl logs <name> [--previous] Read STDOUT and STDERR for crashes and errors
exec What is physically true inside the Pod now? kubectl exec -it <name> -- sh Check environment variables, config files, and network paths