Common Troubleshooting Patterns
Purpose: Recognize common issue patterns for faster diagnosis.
Overview
Pattern recognition is key to efficient troubleshooting. This guide helps you identify common patterns and their solutions.
Pattern Recognition Framework
Pattern: Network Connectivity
Symptoms:
- Connection timeouts
- Connection refused
- Network unreachable
- Services unreachable
Quick Checks:
- Check network policies:
kubectl get networkpolicies - Check service endpoints:
kubectl get endpoints - Test connectivity:
kubectl exec [pod] -- wget [service] - Check DNS:
kubectl exec [pod] -- nslookup [service]
Common Causes:
- Network policy blocking traffic
- Service has no endpoints
- DNS resolution failing
- Firewall rules
Quick Fix:
- Review network policies
- Verify service selectors
- Check DNS configuration
Pattern: Resource Exhaustion
Symptoms:
- OOMKilled pods
- Pending pods
- High CPU/memory usage
- Pods restarting
Quick Checks:
- Check resource usage:
kubectl top pods - Check resource quotas:
kubectl get resourcequota - Check pod limits:
kubectl describe pod - Check node capacity:
kubectl top nodes
Common Causes:
- Resource quota too restrictive
- Node capacity exhausted
- Memory leaks
- Resource requests too high
Quick Fix:
- Increase resource quotas
- Scale cluster
- Fix memory leaks
- Adjust resource requests
Pattern: Permission Denied
Symptoms:
- Forbidden errors
- Access denied
- Permission denied
- Unauthorized
Quick Checks:
- Check RBAC:
kubectl get role,rolebinding - Check service account:
kubectl get serviceaccount - Test permissions:
kubectl auth can-i - Check pod service account:
kubectl get pod -o jsonpath='{.spec.serviceAccountName}'
Common Causes:
- Missing RBAC configuration
- Wrong service account
- Insufficient permissions
- File permission issues
Quick Fix:
- Create Role and RoleBinding
- Use correct service account
- Grant required permissions
Pattern: Image Pull Failure
Symptoms:
- ImagePullBackOff
- ErrImagePull
- Failed to pull image
- Authentication errors
Quick Checks:
- Check pod events:
kubectl describe pod - Verify image name:
kubectl get pod -o jsonpath='{.spec.containers[0].image}' - Check image pull secrets:
kubectl get secrets - Test image pull:
docker pull [image]
Common Causes:
- Image doesn't exist
- Wrong image name/tag
- Missing image pull secret
- Registry authentication failure
Quick Fix:
- Use correct image name
- Configure image pull secrets
- Verify image exists
- Check registry access
Pattern: DNS Resolution Failure
Symptoms:
- Name or service not known
- DNS resolution timeout
- Services unreachable by name
- External DNS failures
Quick Checks:
- Check CoreDNS:
kubectl get pods -n kube-system | grep coredns - Test DNS:
kubectl exec [pod] -- nslookup [service] - Check DNS config:
kubectl exec [pod] -- cat /etc/resolv.conf - Check DNS policy:
kubectl get pod -o jsonpath='{.spec.dnsPolicy}'
Common Causes:
- CoreDNS not running
- Wrong DNS configuration
- Network policy blocking DNS
- Custom DNS without cluster DNS
Quick Fix:
- Ensure CoreDNS is running
- Use default DNS policy
- Include cluster DNS in custom config
- Check network policies
Pattern: Certificate/TLS Failure
Symptoms:
- Certificate verify failed
- TLS handshake failure
- Certificate expired
- x509 errors
Quick Checks:
- Check certificate expiration:
openssl x509 -noout -dates -in cert.pem - Test certificate:
openssl s_client -connect [host]:443 - Check certificate chain:
openssl s_client -showcerts - Review error messages: Check pod logs
Common Causes:
- Expired certificate
- Invalid certificate
- Wrong CA
- Incomplete certificate chain
Quick Fix:
- Renew certificate
- Use valid certificate
- Add custom CA if needed
- Include full certificate chain
Pattern Recognition Tips
1. Look for Error Messages
Key Error Messages:
OOMKilled→ Resource exhaustionImagePullBackOff→ Image pull failureForbidden→ Permission issueConnection refused→ Network issuecertificate verify failed→ Certificate issue
2. Check Pod Status
Status Patterns:
Pending→ Scheduling issue (resources, constraints)CrashLoopBackOff→ Application error, resource issueImagePullBackOff→ Image pull failureError→ Application error, permission issueRunningbut not working → Network, configuration, dependency issue
3. Review Events
Event Patterns:
FailedScheduling→ Resource constraintsFailed→ Application or configuration errorOOMKilling→ Memory exhaustionFailedMount→ Volume or storage issue
4. Check Resource Usage
Resource Patterns:
- High CPU → Performance issue, resource limits
- High memory → Memory leak, resource limits
- Low resources → Resource quota, node capacity
Quick Reference Matrix
| Symptom | Quick Check | Common Cause | Quick Fix |
|---|---|---|---|
| Connection timeout | Network policies | Policy blocking | Review/update policy |
| OOMKilled | Resource limits | Memory limit exceeded | Increase limit |
| Pending pod | Resource quota | Quota exhausted | Increase quota |
| Forbidden | RBAC | Missing permissions | Create Role/RoleBinding |
| ImagePullBackOff | Image name | Image doesn't exist | Fix image name |
| DNS failure | CoreDNS | DNS not configured | Fix DNS config |
| Certificate error | Certificate | Expired/invalid | Renew certificate |
Building Your Pattern Library
Document Patterns
For Each Pattern:
- Symptoms
- Quick checks
- Common causes
- Quick fixes
- Prevention
Practice Recognition
Ways to Practice:
- Work through scenarios
- Review real incidents
- Build mental models
- Share with team
Continuous Learning
Improve Recognition:
- Review incidents
- Update patterns
- Share knowledge
- Practice regularly
Related Documentation
Remember: Pattern recognition comes with experience. The more issues you see, the faster you'll recognize patterns.