Skip to main content

Workflow: Metrics-First RED Triage

Investigating service health using Rate, Errors, and Duration (RED) metrics with direct trace exemplars in Grafana Tempo.

Step 1: Compute RED Metrics for the Target Service​

Calculate request throughput, error rates, and duration percentiles over time:

tempo_traceql_metrics_range(
  query='rate({service.name="order-api"} [1m])',
  start="30m"
)

Step 2: Extract Exemplar Trace IDs from Metric Spikes​

Directly extract exemplar trace identifiers from metric series that breached latency or error thresholds:

tempo_get_exemplar_traces(
  query='rate({service.name="order-api", status=error} [1m])'
)

Step 3: Deep Dive into the Exemplar Trace​

Fetch the complete critical path and span summary for the extracted exemplar:

tempo_summarize_trace(trace_id="<exemplar_trace_id>")

This metrics-first approach prevents scanning thousands of unneeded traces by drilling directly into the exact requests that caused the metric spike.