Observability Tools Explained
Understanding Jaeger, OpenSearch, and the Three Pillars of Observability
DevOps Studio › Docs › Observability Tools
The Three Pillars of Observability
Observability consists of three main data types:
- Metrics - Numerical measurements over time (CPU, memory, request rate)
- Logs - Text-based event records (application logs, error messages)
- Traces - Request flow through distributed systems (end-to-end request paths)
Each tool serves different purposes:
| Tool Type | Purpose | Examples |
|---|---|---|
| Metrics | Measure system performance | Prometheus, CloudWatch |
| Logs | Search and analyze log data | OpenSearch, Elasticsearch, Loki |
| Traces | Track requests across services | Jaeger, Zipkin, Tempo |
What is Jaeger?
Jaeger is a distributed tracing system. It tracks requests as they flow through multiple services in a microservices architecture.
What Jaeger Does
- Traces requests across service boundaries
- Shows timing for each operation
- Identifies bottlenecks in distributed systems
- Visualizes service dependencies
- Helps debug complex request flows
Example Use Case
User Request → API Gateway → Auth Service → User Service → Database
↓ ↓ ↓ ↓ ↓
[Jaeger traces the entire flow, showing timing at each step]If a request is slow, Jaeger shows you:
- Which service is the bottleneck
- How long each service took
- The complete request path
When to Use Jaeger
✅ Use Jaeger when:
- You have microservices (multiple services)
- You need to debug slow requests
- You want to understand service dependencies
- You need to see request flow across services
❌ Don't use Jaeger for:
- Simple monolithic applications
- Log aggregation
- Metrics collection
- Full-text search
What is OpenSearch?
OpenSearch (fork of Elasticsearch) is a search and analytics engine primarily used for log aggregation and analysis.
What OpenSearch Does
- Stores and searches logs at scale
- Full-text search across log data
- Real-time log analysis
- Dashboard creation (with OpenSearch Dashboards)
- Security information and event management (SIEM)
Example Use Case
Application Logs → OpenSearch → Search/Query → Dashboards
↓ ↓ ↓ ↓
[All logs stored] [Indexed] [Fast search] [Visualization]You can search logs like:
- "Show me all errors from the last hour"
- "Find all requests from IP 192.168.1.1"
- "Analyze login failure patterns"
When to Use OpenSearch
✅ Use OpenSearch when:
- You need to search through large volumes of logs
- You want centralized log aggregation
- You need full-text search capabilities
- You're building SIEM solutions
- You need log retention and analysis
❌ Don't use OpenSearch for:
- Distributed tracing (use Jaeger)
- Metrics storage (use Prometheus)
- Simple log viewing (use CloudWatch Logs)
Jaeger vs OpenSearch: They're Not Competitors!
Important: Jaeger and OpenSearch solve different problems. They're complementary, not alternatives.
| Aspect | Jaeger | OpenSearch |
|---|---|---|
| Primary Use | Distributed tracing | Log aggregation & search |
| Data Type | Traces (request flows) | Logs (text events) |
| Best For | Microservices debugging | Log analysis & search |
| Query Type | Trace ID, service name | Full-text search, filters |
| Visualization | Trace timeline view | Dashboards, charts |
| Storage | Time-series trace data | Indexed log documents |
They Work Together!
A complete observability stack often includes:
┌─────────────────────────────────────────────────┐
│ Observability Stack │
├─────────────────────────────────────────────────┤
│ Prometheus → Metrics (CPU, memory, requests) │
│ OpenSearch → Logs (application logs, errors) │
│ Jaeger → Traces (request flows, timing) │
│ Grafana → Visualization (dashboards) │
└─────────────────────────────────────────────────┘Industry Usage
Jaeger
Highly Used For:
- ✅ Microservices architectures
- ✅ Kubernetes environments
- ✅ Cloud-native applications
- ✅ Service mesh (Istio, Linkerd)
Adoption:
- Created by Uber (open source)
- CNCF graduated project
- Widely used in production
- Industry standard for distributed tracing
When It's Essential:
- Microservices with 3+ services
- Need to debug cross-service issues
- Performance optimization
- Understanding service dependencies
OpenSearch
Highly Used For:
- ✅ Log aggregation and analysis
- ✅ Security monitoring (SIEM)
- ✅ Application log search
- ✅ Compliance and auditing
Adoption:
- Fork of Elasticsearch (2021)
- AWS-managed service available
- Growing adoption
- Standard for log management
When It's Essential:
- Large volumes of logs
- Need to search logs efficiently
- Compliance requirements
- Security event analysis
Which Should You Use?
For Lab 04 (Observability Stack)
Based on the planned stack (Prometheus, Grafana, Jaeger):
Recommended Approach:
- Prometheus - Metrics (required)
- Grafana - Visualization (required)
- Jaeger - Distributed tracing (recommended for microservices)
- OpenSearch - Optional, for advanced log analysis
Decision Matrix
Use Jaeger if:
- ✅ You have multiple services (microservices)
- ✅ You need to debug request flows
- ✅ You want to see service dependencies
- ✅ You're building Lab 04 with microservices
Use OpenSearch if:
- ✅ You need centralized log aggregation
- ✅ You want to search through logs
- ✅ You need log retention
- ✅ You're building SIEM capabilities
Use Both if:
- ✅ You want complete observability
- ✅ You have microservices + need log search
- ✅ You're building a production system
Real-World Examples
Example 1: E-Commerce Platform
Services: API Gateway → Auth → Products → Orders → Payment → Shipping
Observability Needs:
- Metrics: Prometheus (request rates, latency)
- Logs: OpenSearch (error logs, transaction logs)
- Traces: Jaeger (track order from cart to delivery)Example 2: Simple Web App
Services: Frontend → Backend → Database
Observability Needs:
- Metrics: Prometheus (sufficient)
- Logs: CloudWatch Logs (simple, no OpenSearch needed)
- Traces: Optional (Jaeger only if debugging complex flows)Lab 04 Recommendation
For Lab 04 - Observability Stack, I recommend:
Core Stack (Essential)
- Prometheus - Metrics collection
- Grafana - Visualization and dashboards
- Jaeger - Distributed tracing (if using microservices)
Optional Additions
- OpenSearch - For advanced log analysis (if time permits)
- Loki - Lightweight log aggregation (alternative to OpenSearch)
Why This Stack?
- Prometheus + Grafana = Industry standard for metrics
- Jaeger = Industry standard for distributed tracing
- OpenSearch = Powerful but adds complexity (can be added later)
Summary
| Question | Answer |
|---|---|
| What is Jaeger? | Distributed tracing system for tracking requests across services |
| Is Jaeger highly used? | Yes, industry standard for microservices tracing |
| Is OpenSearch better? | No, they solve different problems (traces vs logs) |
| Should I use both? | Yes, for complete observability (metrics + logs + traces) |
| For Lab 04? | Start with Prometheus + Grafana + Jaeger, add OpenSearch if needed |
Next Steps
- For Lab 04: Start with Prometheus, Grafana, and Jaeger
- Add OpenSearch later if you need advanced log analysis
- Understand the three pillars: Metrics, Logs, Traces
- Choose based on needs: Not all tools are needed for every use case
Additional Resources
Remember: Observability is about having the right tools for the right data. Metrics, logs, and traces each need different tools, and they work best together!