Skip to main content

🔭 Observability Operator

Manages the complete cloud-native observability stack — Prometheus PromQL metrics, Alertmanager alert triage, OpenTelemetry collector pipelines, Loki LogQL log aggregation, and Tempo TraceQL distributed tracing.

The Observability domain coordinates 5 specialized sub-agents, each connected to a dedicated Model Context Protocol (MCP) server. Together they provide end-to-end telemetry investigation, automated remediation, and interactive UI visualization.

Architecture​

Context Engineering & A2UI Buffer Interceptor​

Observability investigations often return megabytes of raw telemetry data (distributed trace trees, multi-megabyte log streams, high-resolution metric matrices) that would exhaust the LLM's context budget.

k8s-autopilot utilizes the A2UIBufferMiddleware:

  1. When a sub-agent invokes an A2UI tool (prom_query_a2ui_chart, loki_query_a2ui, tempo_query_a2ui, otel_query_a2ui, or am_query_a2ui), the middleware intercepts the heavy payload and writes it directly to the tool artifact store.
  2. The LLM receives only a lightweight pointer token.
  3. The sub-agent calls build_obs_a2ui, which streams interactive React charts, timeline waterfalls, and filterable log tables to the frontend via the A2UI Protocol without loading raw bytes into the conversation prompt.

1. Prometheus Sub-Agent (prometheus-operator)​

Orchestrates Prometheus monitoring operations — from instant PromQL queries to exporter lifecycle management, rule authoring, and TSDB cardinality analysis.

Read-Only Fast-Path Tools & Resources​

  • Resource URIs: prom://system/backends, prom://topology/services, prom://tsdb/cardinality, prom://rules/groups, prom://kubernetes/prometheusrules
  • Query Tools: prom_query_instant, prom_query_range, prom_validate_promql, prom_explore_labels, prom_test_endpoint

State-Modifying Workflows (HITL Gated)​

  • prom_install_exporter: Deploys exporters (Redis, Postgres, Blackbox) with mandatory post-installation validation via prom_verify_exporter.
  • prom_upsert_rule_group: Authors and deploys PrometheusRule CRDs directly to target namespaces in k8s_crd storage mode.
  • prom_apply_probe: Configures endpoint synthetic monitoring with Blackbox probes.

2. Alertmanager Sub-Agent (alertmanager-operator)​

Automates on-call incident triage, silence lifecycle management, routing tree audits, and notification integration testing.

Read-Only & Triage Tools​

  • Resource URIs: alertmanager://system/backends, alertmanager://alerts/firing, alertmanager://silences/active, alertmanager://routing/tree
  • Triage Tools: am_list_firing_alerts, am_describe_alert, am_audit_routing, am_simulate_routing

State-Modifying Workflows (HITL Gated)​

  • am_create_silence: Mandatory dry-run preview displaying matched alerts, author signature, and duration before creation.
  • am_extend_silence / am_expire_silence: Manages active silence lifecycles during maintenance.
  • am_push_test_alert: Pushes synthetic test alerts to verify downstream Slack/PagerDuty routing without triggering false production alarms.

3. OpenTelemetry Sub-Agent (opentelemetry-operator)​

Manages OpenTelemetry Collector topologies, workload auto-instrumentation, processor ordering, and telemetry sampling.

Key Capabilities​

  • Auto-Instrumentation Injection: Uses otel_lookup_instrumentation and otel_patch_instrumentation to configure Java, Python, Node.js, and Go auto-instrumentation CRDs.
  • Deployment Annotation: Invokes otel_annotate_deployment to inject init-containers into pods (with explicit warnings about rolling restarts).
  • Collector Provisioning: Deploys OpenTelemetry Collector instances via otel_provision_collector across Deployment, DaemonSet, or Sidecar modes.
  • Cardinality Auditing & Processor Ordering: Validates that k8sattributes processors execute after batch processors and detects runaway SpanMetrics attributes via otel_detect_cardinality.

4. Loki Sub-Agent (loki-operator)​

Navigates high-scale log aggregations, automatic pattern discovery, and cross-pillar trace-to-log correlation. (100% Read-Only).

Key Capabilities​

  • Schema & Label Discovery: Explores clusters via get_cluster_labels, get_label_values, and get_active_series before forming queries.
  • FinOps Cost Preflight: Mandatory get_query_stats preflight check to estimate processed chunk volume before executing heavy queries, preventing backend OOMs.
  • LogQL Pattern Extraction: Automatically suggests | pattern matchers for unstructured text logs via get_log_patterns.
  • Trace-Log Pivoting: Queries structured metadata ({namespace="prod"} | trace_id="1234abcd") to isolate log streams matching a specific Tempo trace span.

5. Tempo Sub-Agent (tempo-operator)​

Handles distributed tracing, TraceQL query execution, critical path latency summarization, and RED metrics derivation.

Key Capabilities​

  • Intelligent Trace Summarization: Uses tempo_summarize_trace to automatically isolate the critical path, lock contentions, and root cause exception spans within a distributed call tree.
  • RED Metrics Derivation: Derives P99 latency curves and request rates directly from span data using tempo_traceql_metrics_range.
  • Trace Comparison (A/B Diffing): Compares slow vs. fast traces with tempo_compare_traces to identify structural regressions between deployment releases.
  • Cross-Pillar Trace Linking: Resolves Prometheus exemplar metrics to full spans via tempo_get_exemplar_traces and generates alert expressions via tempo_generate_alerting_expression.

🔄 Cross-Pillar Telemetry Workflow​

k8s-autopilot seamlessly correlates across metrics, logs, and traces during active incident response: