Skip to main content

Workflow: Pipeline Investigation & Processor Validation

Auditing OpenTelemetry collector configurations to prevent out-of-memory (OOM) crashes, data loss, and runaway log feedback loops.

Step 1: Inspect Collector Pipeline Topology​

Fetch the runtime pipeline configuration and raw YAML from the Kubernetes cluster:

otel_get_collector(name="otel-gateway", namespace="monitoring")

The tool returns the full graph of active receivers, processors, exporters, and connectors across all enabled telemetry signals (traces, metrics, logs).

Step 2: Validate Processor Execution Sequence​

Validate the processor ordering against OpenTelemetry community architecture standards:

otel_validate_k8sattributes_order(collector_name="otel-gateway", namespace="monitoring")

The Standard Golden Processor Order​

To guarantee stability and correct metadata enrichment, processors must follow this sequence:

  1. memory_limiter: MUST be first. Drops or halts data processing when collector heap usage approaches memory limits, preventing sudden OOM pod restarts.
  2. k8sattributes: Enriches spans and metrics with Pod, Namespace, Node, and ReplicaSet metadata extracted from the Kubernetes API.
  3. resourcedetection: Discovers cloud provider metadata (AWS EC2, GCP GKE, Azure VM) and host details.
  4. transform: Executes OTTL expressions to scrub sensitive PII, remove high-cardinality keys, or normalize attribute names.
  5. filter: Drops unwanted spans or healthy status codes based on expression rules.
  6. tail_sampling: Evaluates entire distributed traces before making retention decisions.
  7. batch: MUST be last (before exporters). Groups telemetry batches together to minimize network overhead and maximize throughput.

[!WARNING] Placing batch before memory_limiter or k8sattributes can cause memory spikes and prevents processors from operating on individual unbatched telemetry items.

Step 3: Check Filelog Receiver Safety​

When running node-level log collection via DaemonSets, audit the receiver configuration to avoid critical failure modes:

otel_check_filelog_safety(collector_name="otel-daemonset", namespace="monitoring")

The tool checks 3 critical safety criteria:

  1. Checkpoint Storage Extension: Verifies that storage: file_storage is enabled so log read positions survive pod restarts without re-ingesting duplicate gigabytes of historical logs.
  2. Self-Log Exclusion: Ensures that the collector's own log files (e.g. /var/log/pods/monitoring_otel-daemonset*) are excluded via regex to eliminate recursive log generation loops.
  3. Container Resource Enrichment: Confirms that container names and pod namespaces are extracted accurately from the file path structure.