Skip to main content

Workflow: Live Incident Response

Rapidly triage outages, scan for panic stack traces, and isolate service degradation during live production incidents.

Step 1: Scan for System-Wide Critical Exceptions​

Scan across the target cluster for panics, fatal errors, and out-of-memory crashes over the past 15 minutes:

execute_logql_query(
  query='{cluster="prod-east"} |~ "(?i)(panic|fatal|out of memory|oom|segfault)"',
  limit=100,
  start="15m"
)

Step 2: Rank Top Failing Services by Error Volume​

Identify which services are experiencing the steepest surge in error logs:

execute_logql_instant(
  query='topk(5, sum(rate({cluster="prod-east"} |= "error" [5m])) by (app))'
)

Step 3: Stream Live Logs into an Interactive A2UI Table​

Stream filtered logs from the focal service directly into an interactive frontend table for active diagnosis:

loki_query_a2ui(
  query='{app="payment-processor"} | json | response_time > 5.0 or status_code >= 500',
  limit=100
)

Review correlated stack traces, identify upstream connection drops, and share actionable findings with the incident response channel.