Skip to main content

Workflow: Sampling Strategy Optimization

Balancing distributed trace visibility with network and storage costs by harmonizing head sampling in application SDKs with tail sampling in collector pipelines.

Step 1: Inspect Multi-Layer Sampling Configuration​

Cross-examine head sampling rules in Instrumentation CRDs against tail sampling rules in your gateway collectors:

otel_inspect_sampling_configuration(
  collector_name="otel-traces",
  namespace="monitoring"
)

What the Tool Detects:

  • Head Sampling Rate: The percentage of traces initiated at the application level (e.g. 5% probabilistic).
  • Tail Sampling Policies: Collector rules designed to capture 100% of HTTP 5xx errors or latency outliers (> 1.5s).
  • Sampling Conflicts: Detects whether aggressive head sampling is dropping spans at the SDK layer before the collector has a chance to evaluate error status codes in tail sampling.

Step 2: Switch to a Hybrid Tail-Biased Strategy​

Generate a balanced sampling configuration patch that enables 100% head generation while applying intelligent tail filtering in the collector:

otel_toggle_sampling_strategy(
  collector_name="otel-traces",
  namespace="monitoring",
  strategy="hybrid",
  tail_sampling_percentage=10.0,
  dry_run=true
)

Applied Tail Sampling Policy Structure:​

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    expected_new_traces_per_sec: 2000
    policies:
      # 1. Capture 100% of errors
      - name: error-policy
        type: status_code
        status_code: { status_codes: [ERROR] }

      # 2. Capture 100% of slow requests
      - name: latency-policy
        type: latency
        latency: { threshold_ms: 1500 }

      # 3. Probabilistic baseline for normal requests
      - name: baseline-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 10.0 }

Step 3: Apply & Validate Trace Ingestion​

Apply the configuration patch (dry_run=false). Verify that Tempo or Jaeger receives complete distributed trace graphs for all production incidents without overwhelming backend storage.