Skip to main content

Workflow: Cardinality & SpanMetrics Audit

Preventing time-series explosions in Prometheus and Mimir when generating metrics from distributed trace spans.

Step 1: Detect High-Cardinality Dimensions​

Audit the spanmetrics connector configuration and active dimensions on your telemetry gateway:

otel_detect_cardinality(
  collector_name="otel-gateway",
  namespace="monitoring",
  threshold=5000
)

The tool analyzes:

  • Active Dimension Combinations: Identifies unbounded attributes such as http.url (with dynamic UUIDs), user.id, client.ip, or order.id.
  • Histogram Bucket Multiplier: Calculates the series multiplier (distinct_dimensions × bucket_count). For example, 1,000 URLs across 15 histogram buckets yields 15,000 active time series per pod.
  • Severity Rating: Classifies cardinality risk as LOW, MEDIUM, or CRITICAL.

Step 2: Generate Remediation OTTL Rules​

Use the tool to generate OpenTelemetry Transformation Language (OTTL) rules that drop the noisy dimensions before metrics are calculated or exported:

otel_gen_drop_attribute_rules(
  signal="metrics",
  attributes_to_drop=["http.url", "user.id", "client.ip"]
)

Generated OTTL Configuration Snippet:

processors:
  transform/strip_cardinality:
    metric_statements:
      - context: datapoint
        statements:
          - delete_key(attributes, "http.url")
          - delete_key(attributes, "user.id")
          - delete_key(attributes, "client.ip")

Step 3: Apply the Collector Patch & Validate​

Patch the collector CRD using dry-run verification first:

otel_patch_collector(
  name="otel-gateway",
  namespace="monitoring",
  config_yaml="...",
  dry_run=false
)

After rollout, verify that Prometheus / Mimir memory usage drops and active time-series counts stabilize.