Skip to main content

Workflow: Error Investigation & Root Cause

A structured, discovery-first methodology for isolating application errors and root causes in Grafana Loki using the TalkOps Loki MCP Server.

Step 1: Discover Label Taxonomy & Scoped Values​

Always verify cluster label names and exact service identifiers first to prevent query failures:

# 1. Discover all active cluster labels
get_cluster_labels()

# 2. Get exact service names in the production namespace
get_label_values(name="app", query='{namespace="production"}')

Step 2: Preflight Query Cost​

Ensure that the target search expression does not scan excessive chunks or exceed cluster byte limits:

get_query_stats(
  query='{namespace="production", app="api-gateway"} |= "level=error"',
  start="1h"
)

The tool reports total streams, chunks, and estimated byte volume to confirm the query is safe to execute.

Step 3: Fetch Sample Error Lines with JSON Parsing​

Retrieve the most recent error logs, extract structured JSON fields, and filter by status code:

execute_logql_query(
  query='{namespace="production", app="api-gateway"} |= "level=error" | json | status_code >= 500',
  limit=50,
  start="1h"
)

Step 4: Quantify Error Volume & Trend Over Time​

Transform the raw log stream into a metric to evaluate error rates across 5-minute intervals:

execute_logql_query(
  query='sum(rate({namespace="production", app="api-gateway"} |= "level=error" [5m])) by (status_code)',
  start="6h",
  step="5m"
)

Analyze the returned time-series matrix to determine whether the error is a sustained regression or an isolated spike.