Skip to main content

Workflow: Topology Mapping & Alert Generation

Visualizing microservice dependency graphs and generating automated Prometheus alerting rules from live trace patterns.

Step 1: Extract Real-Time Service Dependencies​

Retrieve upstream callers, downstream targets, request throughput, and edge error rates from Tempo's metrics-generator:

tempo_get_service_dependencies(service="api-gateway")

What the Tool Returns:

  • Nodes: Connected microservice components.
  • Edges: Inbound and outbound RPC connections with request rate (req/s) and error percentages.

Step 2: Generate Production PromQL Alerting Rules​

Translate high-latency or error trace patterns directly into a ready-to-deploy Prometheus alerting rule YAML:

tempo_generate_alerting_expression(
  service="api-gateway",
  threshold_duration_ms=1500,
  error_rate_threshold=5.0
)

Generated Alerting Rule:

groups:
  - name: tempo_trace_alerts
    rules:
      - alert: HighLatencyApiGateway
        expr: histogram_quantile(0.99, sum(rate(traces_spanmetrics_latency_bucket{service="api-gateway"}[5m])) by (le)) > 1.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency exceeded 1.5s for api-gateway"
          description: "P99 latency is currently {{ $value }}s on service api-gateway."

[!TIP] You can pass this generated rule group directly to the TalkOps Prometheus MCP Server (prom_upsert_rule_group) to deploy the rule onto your cluster with zero manual YAML editing.