ADR 003: Monitoring Strategy — Separate Tools Per Telemetry Pillar

DevOps Studio › Docs › Architecture Decisions › ADR 003

Status: Accepted

Context

Lab 04 has to give operators enough telemetry to detect, investigate, and resolve production issues on the platform built in Labs 01–02. The three pillars of observability — metrics, logs, and traces — each have different access patterns and storage shapes:

• Metrics are numeric time series, queried and aggregated over time windows. High cardinality is expensive; long retention at full resolution rarely is.
• Logs are unstructured or semi-structured text, queried by full-text search and filtering. Volume is the dominant cost driver.
• Traces are causally-linked spans across services, queried by trace ID or service dependency, not by time-window aggregation.

A single tool optimized for one of these shapes tends to be a poor fit for the other two — a log store is a bad time-series database, and a metrics store is a bad place to full-text search a stack trace. The alternative to picking a best-of-breed tool per pillar is a single vendor platform that does all three, which trades operational simplicity for cost and lock-in.

Decision

Run one purpose-built tool per pillar, and unify them at the visualization layer instead of the storage layer:

• Prometheus for metrics — pull-based scraping, PromQL, and the de facto standard the rest of the Kubernetes ecosystem (kube-state-metrics, node-exporter, the EKS add-ons from ADR 002) already exposes metrics for.
• OpenSearch + Fluent Bit for logs — Fluent Bit runs as a lightweight per-node collector shipping container logs to OpenSearch for full-text search and retention.
• Jaeger for distributed traces — request-flow visualization and service dependency mapping across the microservices a customer runs on the Lab 02 platform.
• Grafana as the single pane of glass, wired to all three backends as data sources, plus Alertmanager for routing Prometheus alert rules.

Nothing here duplicates storage: Grafana holds no data of its own, it queries the three backends live.

Consequences

Positive:

• Each pillar uses the tool actually built for its data shape — no forcing logs into a metrics store or vice versa.
• Prometheus and Jaeger are what most customers' existing Kubernetes workloads are already instrumented for (via client libraries or auto-instrumentation), so integration cost is low.
• A single Grafana view means an operator doesn't have to context-switch across four UIs to correlate a metric spike with the logs and traces from the same time window.
• Components can be added or swapped independently — a customer standardized on Datadog or an existing ELK stack could replace one pillar without touching the other two.

Negative / tradeoffs:

• Four systems to run instead of one means four sets of upgrade cycles, four sets of resource limits to tune, and four failure modes to know when troubleshooting ("is Grafana down, or is its Prometheus datasource down?").
• Correlation across pillars happens by convention (shared timestamps, trace IDs surfaced in logs) rather than a single backend enforcing a shared schema — a unified vendor platform would guarantee that correlation, this stack has to earn it through consistent labeling and instrumentation.
• Running all four continuously outside of the lab window is the most expensive component of the platform to leave on (see the Cost Management guide); this stack is sized for "install, validate, tear down," not for being the customer's permanent production monitoring without further capacity planning.