Skip to main content

Workflow: Missing Traces Diagnostic

Troubleshooting why expected distributed trace data is missing from Grafana Tempo.

Step 1: Verify Backend Reachability & Ingester Health​

Inspect the health of Tempo ring members, compactors, and ingesters:

tempo_get_diagnostics()

Confirm that the backend is in ready status and ingester ring tokens are balanced.

Execute a query with an empty filter to verify that any span data is arriving in the storage blocks:

tempo_traceql_search(
  q='{}',
  limit=5,
  start="15m"
)

Step 3: Audit Sampling Drops & Multi-Tenant Headers​

If empty searches return no results, audit these common failure points:

  1. Client-Side Head Sampling: Check Instrumentation CRDs to ensure aggressive head samplers (traceidratio: 0.01) are not discarding 99% of requests before transmission.
  2. Collector Tail Sampling: Audit collector drop policies to confirm spans are not being filtered out.
  3. Multi-Tenant Headers: Verify that the client's X-Scope-OrgID header matches the tenant identifier configured in your Tempo distributor and querier.