Troubleshooting: Handoff and Runbooks
Common issues and solutions for monitoring, runbooks, training, and handoff.
Monitoring Issues
Grafana Not Accessible
Symptoms:
- Cannot access Grafana UI
- Connection refused
- Timeout errors
Investigation:
bash
# Check pod status
kubectl get pods -n monitoring
# Check service
kubectl get svc -n monitoring
# Check port-forward
kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80Solutions:
- Verify pod is running:
kubectl get pods -n monitoring - Check service exists:
kubectl get svc -n monitoring - Verify port-forward: Use correct service name and port
- Check firewall rules (if applicable)
Prometheus Not Scraping Metrics
Symptoms:
- No metrics in Prometheus
- Targets showing as down
- Missing metrics
Investigation:
bash
# Check Prometheus targets
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
# Open http://localhost:9090/targets
# Check ServiceMonitor
kubectl get servicemonitor -n monitoring
# Check pod metrics endpoint
kubectl exec [pod-name] -n [namespace] -- curl http://localhost:8080/metricsSolutions:
- Verify ServiceMonitor exists:
kubectl get servicemonitor - Check service labels match ServiceMonitor selector
- Verify metrics endpoint is accessible
- Check network policies (if applicable)
Alerts Not Firing
Symptoms:
- Alerts configured but not firing
- No alert notifications
- Alerts in pending state
Investigation:
bash
# Check alert rules
kubectl get prometheusrule -n monitoring
# Check Prometheus alerts
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090
# Open http://localhost:9090/alerts
# Check Alertmanager
kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-alertmanager 9093:9093
# Open http://localhost:9093Solutions:
- Verify alert rules are loaded: Check Prometheus UI
- Check alert conditions: Verify thresholds
- Verify Alertmanager configuration: Check notification channels
- Test alert rules: Manually trigger conditions
Runbook Issues
Runbook Not Clear
Symptoms:
- Customer team confused by runbook
- Steps unclear
- Missing information
Solutions:
- Review runbook with customer team
- Add more detail to unclear steps
- Include examples
- Add screenshots/diagrams
- Test runbook with fresh eyes
Runbook Outdated
Symptoms:
- Procedures don't match current system
- Commands don't work
- Configuration outdated
Solutions:
- Review and update runbook
- Test all procedures
- Verify all commands
- Update configuration examples
- Establish regular review schedule
Runbook Not Accessible
Symptoms:
- Customer team can't find runbooks
- Runbooks not in expected location
- Access issues
Solutions:
- Verify runbook location
- Check access permissions
- Provide clear navigation
- Create index/table of contents
- Ensure version control access
Training Issues
Customer Team Struggling
Symptoms:
- Team members not understanding
- Exercises too difficult
- Low confidence
Solutions:
- Review prerequisites
- Provide additional practice time
- Break down complex topics
- Offer one-on-one sessions
- Adjust training pace
Training Too Fast
Symptoms:
- Information overload
- Not enough practice time
- Concepts not sinking in
Solutions:
- Slow down training pace
- Add more practice exercises
- Extend training duration
- Provide additional sessions
- Focus on fundamentals
Training Materials Incomplete
Symptoms:
- Missing information
- Unclear instructions
- Examples don't work
Solutions:
- Review and complete materials
- Test all exercises
- Add missing information
- Fix examples
- Get feedback and improve
Handoff Issues
Customer Team Not Ready
Symptoms:
- Low confidence
- Not completing exercises
- Asking basic questions
Solutions:
- Extend training period
- Provide additional practice
- Offer more support
- Adjust handoff timeline
- Consider gradual transition
Documentation Incomplete
Symptoms:
- Missing documentation
- Incomplete procedures
- Outdated information
Solutions:
- Complete all documentation
- Review handoff checklist
- Fill in gaps
- Update outdated information
- Don't handoff until complete
Support Model Unclear
Symptoms:
- Customer team unsure of support
- Unclear escalation
- Confusion about availability
Solutions:
- Clearly define support model
- Document support procedures
- Provide support contacts
- Explain escalation process
- Set clear expectations
General Troubleshooting
Access Issues
Problem: Cannot access cluster/monitoring
Solutions:
- Verify credentials
- Check RBAC permissions
- Verify network connectivity
- Check firewall rules
- Contact administrator
Configuration Issues
Problem: Configuration not working
Solutions:
- Verify configuration format
- Check configuration values
- Review configuration documentation
- Test in non-production
- Check for typos
Performance Issues
Problem: System slow or unresponsive
Solutions:
- Check resource usage
- Review metrics
- Identify bottlenecks
- Scale if needed
- Optimize configuration
Getting Help
Internal Resources
- Review documentation
- Check runbooks
- Review troubleshooting guides
- Check knowledge base
Support Channels
- Email: [support-email]
- Slack: [slack-channel]
- Phone: [phone-number]
- Ticket System: [ticket-url]
Escalation
- Level 1: On-call engineer
- Level 2: Team lead
- Level 3: Engineering manager
- Level 4: CTO/VP Engineering
Prevention
Best Practices
- Test Everything: Test all procedures before handoff
- Document Thoroughly: Complete documentation prevents issues
- Train Properly: Good training prevents operational issues
- Monitor Continuously: Monitoring catches issues early
- Review Regularly: Regular reviews catch problems
Regular Maintenance
- Monthly: Review documentation
- Quarterly: Comprehensive review
- After Changes: Immediate update
- After Incidents: Update based on lessons learned
Related Documentation
Remember: Good troubleshooting is systematic. Follow a methodical approach and document solutions for future reference.