Tempo MCP Server

Name: TalkOps Tempo MCP Server
Author: TalkOps

Grafana Tempo provides massive, cost-effective scale for distributed tracing — but finding the root cause of a latency spike across millions of spans requires deep operational expertise. Constructing complex TraceQL structural queries, calculating critical paths across multi-tiered microservice call trees, correlating traces with logs and metrics, and configuring multi-tenant gateways each present significant engineering hurdles. When generic AI assistants try to interact with tracing backends, they frequently hallucinate TraceQL structural operators, generate unbounded searches that overload queriers, or exhaust LLM token limits on raw span payloads.

The TalkOps Tempo MCP Server fixes this. It provides AI assistants (Claude, Cline, Cursor, or TalkOps autonomous controllers) with structured, high-intent, and production-safe tools to interact with Grafana Tempo natively:

Smart TraceQL Search with K8s-Friendly Filters. Say "Find slow error traces in the checkout service on production" and the AI auto-translates Kubernetes metadata (namespace, service, deployment) into valid TraceQL with enforced time ranges and result limits.
Intelligent Trace Summarization. The server extracts the critical path, identifies error spans, isolates suspected root-cause services, and performs time-gap detection (distinguishing true critical-path delays from async background tasks).
Metrics-First RED Triage. Execute TraceQL metric queries (rate(), count_over_time(), quantile_over_time()) to evaluate request rates, error rates, and P95/P99 duration trends before drilling into specific traces.
Cross-Pillar Correlation. Seamlessly pivot from log messages to traces (parsing trace IDs directly from log lines) and metrics to traces (extracting exemplar trace IDs from TraceQL metric series).
Service Topology & Dependency Mapping. Map live dependency graphs from Tempo metrics-generator data with request rates and error percentages per edge.
Backend Diagnostics & Operator CRD Management. Perform 360-degree health checks across Tempo ring members, compactors, and ingesters, and manage TempoStack / TempoMonolithic Operator custom resources.

The Observability Trifecta

The Tempo MCP Server represents the Traces Pillar of the TalkOps cloud-native observability stack:

┌────────────────────────────────────────────────────────┐
│               TalkOps Observability Stack              │
├───────────────────┬───────────────────┬────────────────┤
│   PROMETHEUS MCP  │     LOKI MCP      │   TEMPO MCP    │
│     (Metrics)     │      (Logs)       │    (Traces)    │
│  28 Tools · TSDB  │  9 Tools · LogQL  │ 16 Tools · TraceQL│
└───────────────────┴───────────────────┴────────────────┘
                           ▲
                           │ Pipeline Plumbing
             ┌─────────────────────────────┐
             │    OPENTELEMETRY MCP        │
             │ (Collector & Instrumentation)│
             └─────────────────────────────┘

Key Features

TraceQL Search & K8s Canonical Mapping

Search via raw TraceQL or structured filters (namespace, service, duration, status).
Canonical mapping between Kubernetes metadata and OpenTelemetry resource attributes.
Built-in guardrails: mandatory time ranges, limit clamping, empty-query rejection.

Intelligent Trace Summarization & Comparison

LLM-optimized trace format support (Tempo 2.9+ application/vnd.grafana.llm).
Automatic critical path extraction and root cause identification.
5-dimensional trace comparison: span count, duration, errors, attributes, and critical path.
Related trace discovery: same_service_errors, same_endpoint, temporal_neighbors.

TraceQL Metrics & Cross-Pillar Pivots

Instant and range TraceQL metrics queries returning Prometheus-compatible vectors and matrices.
Pivot from log messages directly to full summarized traces (tempo_get_trace_from_log).
Extract exemplar trace IDs from metric spikes (tempo_get_exemplar_traces).

Backend Diagnostics & Topology

Multi-backend discovery with health probing and component status inspection.
Live service graph topology extraction (tempo_get_service_dependencies).
Automated PromQL alerting rule generation from trace patterns (tempo_generate_alerting_expression).

Architecture

How it works:

Your AI assistant connects over HTTP, SSE, or stdio and initializes the session.
The agent reads tempo://system/backends to discover available Tempo backends and verify health.
The AI executes targeted queries with enforced limits, analyzes critical paths, or maps service dependencies.
The service layer (tempo_service) handles HTTP calls with connection pooling, tenant header injection (X-Scope-OrgID), and LLM format negotiation (application/vnd.grafana.llm).

Tech Stack

Category	Technologies
Language	Python 3.12+
MCP Framework	FastMCP ≥ 2.13.3
Protocol	Model Context Protocol (MCP)
Target Engine	Grafana Tempo 2.4+ / 2.9+ / 3.0+ · TraceQL
Data Formats	LLM-Optimized Format (`application/vnd.grafana.llm`) · OTLP JSON
Multi-Tenancy	`X-Scope-OrgID` Header Injection · Cross-Tenant Queries
Transport Modes	`stdio` · `http` · `sse` · `streamable-http`
Packaging	Docker · uv

Available Tools Summary

Tool	Category	Description
`tempo_traceql_search`	Search	High-intent trace search with TraceQL and K8s filters.
`tempo_get_trace`	Retrieval	Retrieve a single trace by ID in LLM-optimized format.
`tempo_summarize_trace`	Analysis	Extract critical path, error root cause, and time-gap analysis.
`tempo_compare_traces`	Analysis	5-dimensional diff comparison between two traces.
`tempo_find_related_traces`	Correlation	Find related traces using error, endpoint, or temporal correlation.
`tempo_traceql_metrics_range`	Metrics	Execute TraceQL metric range queries for RED trends and SLOs.
`tempo_traceql_metrics_instant`	Metrics	Execute instant TraceQL metric snapshots.
`tempo_get_exemplar_traces`	Pivots	Extract exemplar trace IDs from metric query spikes.
`tempo_get_trace_from_log`	Pivots	Parse trace ID from log line and retrieve trace summary.
`tempo_get_service_dependencies`	Topology	Map service dependency graphs from metrics-generator metrics.
`tempo_get_attribute_names`	Schema	Discover indexed attribute names across resource and span scopes.
`tempo_get_attribute_values`	Schema	Enumerate distinct values for a trace attribute.
`tempo_get_k8s_attribute_map`	Schema	Canonical mapping between K8s metadata and OTel trace attributes.
`tempo_list_backends`	Diagnostics	List configured Tempo backends with health states.
`tempo_get_diagnostics`	Diagnostics	Comprehensive backend health, ring status, and component diagnostics.
`tempo_generate_alerting_expression`	Alerting	Generate PromQL alerting rules from trace patterns.

Quick Start

Running with Docker

docker run --rm -it \
  -p 8768:8768 \
  -e MCP_TRANSPORT=http \
  -e TEMPO_BASE_URL=http://tempo.monitoring:3200 \
  talkopsai/tempo-mcp-server:latest

Connect to Claude Desktop, Cursor, or Cline

Add the server to your MCP configuration file (claude_desktop_config.json or .cursor/mcp.json):

{
  "mcpServers": {
    "tempo": {
      "url": "http://localhost:8768/mcp",
      "description": "TalkOps Tempo MCP Server for Distributed Tracing"
    }
  }
}

Or run via stdio using uvx:

{
  "mcpServers": {
    "tempo": {
      "command": "uvx",
      "args": ["tempo-mcp-server"],
      "env": {
        "TEMPO_BASE_URL": "http://localhost:3200",
        "MCP_LOG_LEVEL": "INFO"
      }
    }
  }
}

Security Considerations

Tenant Scoping: Always configure TEMPO_MULTI_TENANT=true and TEMPO_DEFAULT_TENANT when connecting to multi-tenant Tempo clusters to avoid data leaks between teams.
Payload Clamping: The server enforces TEMPO_MAX_SEARCH_LIMIT=100 and supports Tempo 2.9+ LLM format to prevent oversized JSON responses from exhausting context windows.
Read-Only Inspection: Trace query and analysis tools are strictly read-only and safe for automated autonomous agent workflows.

Project Layout

tempo-mcp-server/
├── tempo_mcp_server/
│   ├── tools/           # 16 MCP Tools across 6 functional groups
│   ├── resources/       # 11 MCP Resources (tempo:// URIs & runbooks)
│   ├── prompts/         # 5 Guided workflow prompts
│   ├── services/        # Tempo HTTP service and K8s discovery
│   ├── server/          # FastMCP server setup & caching middleware
│   ├── models/          # Pydantic data schemas & TraceQL models
│   ├── config.py        # Environment variables & backend configs
│   └── main.py          # Server entry point
├── tests/               # Unit and integration test suites
├── docs/                # Workflow guides and test fixtures
├── Dockerfile
├── pyproject.toml
└── README.md

Next Steps

Configuration — Single/multi-backend settings, multi-tenancy, and search lookback guardrails.
Tools Reference — Complete parameter specifications and TraceQL examples for all tools.
Resources Reference — 11 read-only tempo:// URIs and operational runbooks.
Workflows — Step-by-step latency, error triage, and topology alerting workflows.
Examples & Scenarios — Real-world conversational scenarios and root cause investigations.

The Observability Trifecta​

Key Features​

Architecture​

Tech Stack​

Available Tools Summary​

Quick Start​

Running with Docker​

Connect to Claude Desktop, Cursor, or Cline​

Security Considerations​

Project Layout​

Next Steps​