Skip to main content

Workflow: Latency Spike Investigation

Step-by-step methodology for isolating microservice performance regressions and calculating critical paths in Grafana Tempo using the TalkOps Tempo MCP Server.

Step 1: Confirm Latency Trend with TraceQL Metrics​

Calculate P95 and P99 latency percentiles over 5-minute rolling windows:

tempo_traceql_metrics_range(
  query='quantile_over_time(0.99, {service.name="api-gateway"} [5m])',
  start="1h"
)

Confirm whether the latency degradation is isolated to a specific endpoint or widespread across the service.

Step 2: Search for Degraded Traces Above Threshold​

Search for concrete traces that breached your Service Level Objective (SLO):

tempo_traceql_search(
  service="api-gateway",
  min_duration="2s",
  limit=10,
  start="1h"
)

Step 3: Extract the Critical Path & Root Cause​

Analyze the slowest trace to isolate the bottleneck span:

tempo_summarize_trace(trace_id="b2c3d4e5f60718293a4b5c6d7e8f90a1")

The tool determines:

  • Critical Path Duration: The longest chain of synchronous dependent spans.
  • Root Cause Component: Identifies the exact service, operation name, and percentage of overall latency.
  • Time Gap Detection: Confirms whether the delay is synchronous execution or async timer wait states.

Step 4: Compare Against a Healthy Baseline Trace​

Perform a 5-dimensional diff against a fast trace from normal traffic:

tempo_compare_traces(
  trace_id_a="b2c3d4e5f60718293a4b5c6d7e8f90a1",
  trace_id_b="a0b1c2d3e4f567890123456789abcdef"
)

The diff highlights extra database round trips, new unindexed queries, or serialization overhead introduced in recent releases.