Scott Tompkins

Pipeline-Aware Observability Platform

4 min read

Pipeline-Aware Observability Platform

TL;DR: A monitoring system that shows what’s happening inside our data pipelines in real time. When something goes wrong, we can pinpoint the problem in minutes instead of hours—often before users even notice. Think of it as an MRI for infrastructure: we see exactly where things are healthy and where they’re struggling.

The Problem with Infrastructure Metrics

Standard infrastructure monitoring tells you CPU is high or a disk is filling up. That’s necessary, but it doesn’t answer the questions we actually ask during incidents. When an analyst says “search is slow,” the cause is usually somewhere in the data pipeline—data flows from agents through managers through Logstash into OpenSearch, and a problem could originate at any stage. CPU and memory graphs don’t tell you which stage is the bottleneck.

I needed monitoring that traced the full pipeline path, not just the servers it runs on.

How It Works

The stack is Prometheus for metrics, Loki for logs, and Grafana for visualization—a common combination, chosen deliberately because the integration between them is well-documented and the ecosystem is mature. Prometheus scrapes metrics from every pipeline stage. Vector collects and enriches logs before shipping them to Loki. Grafana ties everything together.

The key design choice was correlation across pipeline stages. I instrumented ingestion, processing, and indexing separately, then built dashboards that show them side by side. When indexing latency spikes, I can immediately see whether it correlates with a volume surge at ingestion, a queue backup in Logstash, or a health issue in OpenSearch. Isolated metrics are hard to interpret—“indexing latency is high” doesn’t tell you what to fix, but “indexing latency is high AND Logstash queues are backing up AND ingestion volume is normal” tells you Logstash is the bottleneck.

This correlation required consistent labeling across sources and careful dashboard design so that metrics from different components can be joined on common dimensions like tenant, pipeline stage, and data type. That was the most time-consuming part of the build.

I also wrote custom exporters where the default instrumentation didn’t go deep enough—Logstash pipeline internals and OpenSearch cluster state details that off-the-shelf exporters don’t expose.

What the Dashboards Cover

Rather than one sprawling dashboard, I built separate views for different operational contexts. A dashboard that tries to show everything shows nothing well—during an incident, I want the five metrics that matter for that type of problem, not fifty that might be relevant.

Pipeline health shows ingestion throughput, Logstash queue depth, and indexing latency together. Cluster health breaks down OpenSearch by node group (Client, Master, Hot, Warm, Ingest). Tenant activity shows per-tenant data volume and query patterns. Capacity planning tracks storage consumption and throughput trends over time so I can forecast when we’ll need to add resources.

Each dashboard links to Loki queries for the relevant logs in that time window, so the path from “this metric spiked” to “here’s what was happening” is one click.

Alerts are designed to be actionable—each one includes a runbook link and context about what typically causes that condition, so the person responding doesn’t start from zero.

Outcome

Diagnosis that used to take hours now takes minutes. When analysts report slow search, the pipeline dashboard usually reveals the bottleneck within five minutes. Capacity planning is based on trend data rather than reacting when things break. Alerts fire before problems are visible to users, which means fewer surprises and more operational confidence when volume spikes or a node misbehaves.

Technologies

Prometheus, Alertmanager, Grafana, Loki, Vector, custom exporters, OpenSearch (for complex log analysis).