OpenSearch - Log Aggregation and Search

OpenSearch is a distributed search and analytics engine derived from Elasticsearch. It's used for log aggregation, full-text search, and real-time analytics.

Overview

OpenSearch provides:

Centralized log storage - Aggregate logs from all services
Full-text search - Fast search across log data
Real-time analytics - Analyze logs as they arrive
Dashboards - OpenSearch Dashboards for visualization
Scalability - Handle large volumes of logs

Architecture

┌─────────────────────────────────────────────────┐
│              Application Services                 │
│  (Pods, Containers, Applications)                │
└──────────────────┬──────────────────────────────┘
                   │
                   ▼
         ┌──────────────────┐
         │  OpenSearch      │
         │  (Data Nodes)     │
         └────────┬──────────┘
                  │
                  ▼
         ┌──────────────────┐
         │  OpenSearch       │
         │  Dashboards       │
         │  (UI)             │
         └──────────────────┘

Installation

Using Helm (Recommended)

bash

# Add OpenSearch Helm repository
helm repo add opensearch https://opensearch-project.github.io/helm-charts/
helm repo update

# Install OpenSearch cluster
helm install opensearch opensearch/opensearch \
  --namespace observability \
  --create-namespace \
  --set persistence.enabled=true \
  --set persistence.size=20Gi

# Install OpenSearch Dashboards
helm install opensearch-dashboards opensearch/opensearch-dashboards \
  --namespace observability \
  --set opensearchHosts=https://opensearch:9200

Using Kubernetes Manifests

bash

# Apply OpenSearch manifests
kubectl apply -f manifests/opensearch-cluster.yaml
kubectl apply -f manifests/opensearch-dashboards.yaml

# Wait for pods to be ready
kubectl wait --for=condition=ready pod -l app=opensearch -n observability --timeout=300s

Configuration

Basic Configuration

The OpenSearch cluster is configured via the Kubernetes manifests in manifests/opensearch-cluster.yaml. Key configuration options are set via environment variables:

Cluster name: opensearch-cluster
Initial master nodes: opensearch-0, opensearch-1, opensearch-2
Node roles: Master, Data, and Ingest (all-in-one nodes)
Heap size: 2GB (adjustable via OPENSEARCH_JAVA_OPTS)

To customize, edit the StatefulSet in manifests/opensearch-cluster.yaml.

Resource Requirements

Minimum for development:

CPU: 2 cores per node
Memory: 4GB per node
Storage: 20GB per node

Production recommendations:

CPU: 4+ cores per node
Memory: 8GB+ per node
Storage: 100GB+ per node (SSD recommended)

Usage

Access OpenSearch Dashboards

bash

# Port forward to access UI
kubectl port-forward svc/opensearch-dashboards -n observability 5601:5601

# Open in browser
open http://localhost:5601

Default credentials:

Username: admin
Password: admin (change in production!)

Send Logs to OpenSearch

Using Fluent Bit (Recommended)

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: observability
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            docker
        Tag               kube.*
        Refresh_Interval  5

    [OUTPUT]
        Name  opensearch
        Match *
        Host  opensearch.observability.svc.cluster.local
        Port  9200
        Index kubernetes-logs
        Type  _doc

Using Logstash

ruby

input {
  beats {
    port => 5044
  }
}

output {
  opensearch {
    hosts => ["opensearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

Query Logs

Using OpenSearch Dashboards

Navigate to Discover
Select index pattern (e.g., kubernetes-logs-*)
Search using KQL (Kibana Query Language):
```
level: ERROR AND service: api-gateway
```

Using cURL

bash

# Search for errors
curl -X GET "opensearch:9200/kubernetes-logs/_search?q=level:ERROR" \
  -u admin:admin

# Get log count by service
curl -X GET "opensearch:9200/kubernetes-logs/_search" \
  -u admin:admin \
  -H 'Content-Type: application/json' \
  -d '{
    "size": 0,
    "aggs": {
      "by_service": {
        "terms": {
          "field": "service.keyword"
        }
      }
    }
  }'

Integration with Other Tools

Grafana Integration

Add OpenSearch as a data source in Grafana:

Go to Configuration > Data Sources
Add OpenSearch data source
Configure:
- URL: http://opensearch:9200
- Access: Server (default)
- Index name: kubernetes-logs-*
- Time field: @timestamp

Prometheus Integration

OpenSearch exposes metrics that Prometheus can scrape:

yaml

# prometheus-config.yaml
scrape_configs:
  - job_name: 'opensearch'
    static_configs:
      - targets: ['opensearch:9200']
    metrics_path: '/_prometheus/metrics'

Best Practices

Index Management

Use index templates for consistent structure
Set up index lifecycle management (ILM) for retention
Use date-based indices (e.g., logs-2024-01-15)

Performance Optimization

Shard sizing: 20-50GB per shard
Replica count: 1 for dev, 2+ for production
Refresh interval: Increase for high-volume logs
Use hot/warm/cold tiers for cost optimization

Security

Enable authentication (required in production)
Use TLS for transport security
Implement RBAC for access control
Rotate credentials regularly

Troubleshooting

Pods Not Starting

bash

# Check pod status
kubectl get pods -n observability -l app=opensearch

# Check logs
kubectl logs -n observability opensearch-0

# Check events
kubectl describe pod -n observability opensearch-0

High Memory Usage

bash

# Check memory usage
kubectl top pods -n observability

# Adjust heap size in config
# heap_size: 2g  # Should be ~50% of container memory

Connection Issues

bash

# Test connectivity
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -u admin:admin http://opensearch:9200/_cluster/health

# Check service endpoints
kubectl get endpoints -n observability opensearch

OpenSearch - Log Aggregation and Search

Overview

Architecture

Installation

Using Helm (Recommended)

Using Kubernetes Manifests

Configuration

Basic Configuration

Resource Requirements

Usage

Access OpenSearch Dashboards

Send Logs to OpenSearch

Using Fluent Bit (Recommended)

Using Logstash

Query Logs

Using OpenSearch Dashboards

Using cURL

Integration with Other Tools

Grafana Integration

Prometheus Integration

Best Practices

Index Management

Performance Optimization

Security

Troubleshooting

Pods Not Starting

High Memory Usage

Connection Issues

Cost Considerations

Storage Costs

Optimization Tips

Additional Resources

OpenSearch - Log Aggregation and Search ​

Overview ​

Architecture ​

Installation ​

Using Helm (Recommended) ​

Using Kubernetes Manifests ​

Configuration ​

Basic Configuration ​

Resource Requirements ​

Usage ​

Access OpenSearch Dashboards ​

Send Logs to OpenSearch ​

Using Fluent Bit (Recommended) ​

Using Logstash ​

Query Logs ​

Using OpenSearch Dashboards ​

Using cURL ​

Integration with Other Tools ​

Grafana Integration ​

Prometheus Integration ​

Best Practices ​

Index Management ​

Performance Optimization ​

Security ​

Troubleshooting ​

Pods Not Starting ​

High Memory Usage ​

Connection Issues ​

Cost Considerations ​

Storage Costs ​

Optimization Tips ​

Additional Resources ​

OpenSearch - Log Aggregation and Search

Overview

Architecture

Installation

Using Helm (Recommended)

Using Kubernetes Manifests

Configuration

Basic Configuration

Resource Requirements

Usage

Access OpenSearch Dashboards

Send Logs to OpenSearch

Using Fluent Bit (Recommended)

Using Logstash

Query Logs

Using OpenSearch Dashboards

Using cURL

Integration with Other Tools

Grafana Integration

Prometheus Integration

Best Practices

Index Management

Performance Optimization

Security

Troubleshooting

Pods Not Starting

High Memory Usage

Connection Issues

Cost Considerations

Storage Costs

Optimization Tips

Additional Resources