Skip to main content

Workflow: Error Triage & Root Cause

Isolating the root origin of distributed errors across cascading downstream microservices.

Step 1: Quantify Error Rate Surge​

Evaluate the overall error percentage across your service topology:

tempo_traceql_metrics_range(
  query='rate({status=error} [5m]) / rate({} [5m]) * 100',
  start="2h"
)

Step 2: Search for Failed Traces​

Retrieve candidate failed traces for the focal service:

tempo_traceql_search(
  service="checkout-service",
  status="error",
  limit=5,
  start="30m"
)

Step 3: Extract the Root Cause Span​

Analyze the trace to find the earliest failing span in the causal dependency graph:

tempo_summarize_trace(trace_id="4a5b6c7d8e9f0123456789abcdef0123")

The tool identifies whether the failure originated in checkout-service or was caused by an upstream timeout from an external payment gateway.

Step 4: Find Correlated Error Traces​

Search for other traces affected by the identical error signature:

tempo_find_related_traces(
  trace_id="4a5b6c7d8e9f0123456789abcdef0123",
  strategy="same_service_errors"
)