Tail sampling

​

Prerequisites

• An OpenTelemetry Collector Contrib build (otelcol-contrib) that receives every span of a trace.
• An ingest key in the Collector’s environment as SQUASHER_API_KEY. See Create an ingest key.

​

Configure the sampler

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    expected_new_traces_per_sec: 100
    policies:
      - name: error-status
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: recorded-exceptions
        type: ottl_condition
        ottl_condition:
          error_mode: ignore
          spanevent:
            - 'name == "exception"'
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
  batch: {}

exporters:
  otlphttp/squasher:
    endpoint: https://ingest.squasher.ai
    headers:
      x-squasher-key: "${env:SQUASHER_API_KEY}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/squasher]

​

Send every span to the sampler

• Sample all traces in the SDK, usually with the always_on sampler. Tail sampling cannot restore a trace that the SDK dropped.
• Do not sample at random in each service. That splits traces before the Collector sees them.
• Send all spans of one trace to the same Collector. With more than one Collector, route by trace ID with the OpenTelemetry load-balancing exporter. A round-robin load balancer splits traces.

​

What Squasher keeps

​

Verify

1. Send four traces: a fast success, a slow one, one with an ERROR span, and one with an exception event but no status.
2. In Squasher, select Traces in the sidebar. The last three traces are there.
3. Watch the Collector’s metrics for traces dropped before a decision, late spans, and export failures before you widen the rollout.

​

Limits

• The sampler keeps spans in memory until it decides. If it drops traces too early, increase num_traces and the memory, or reduce decision_wait.
• Tail sampling applies to traces only, not to logs or metrics.

​

Next steps

OpenTelemetry

See the endpoint and what Squasher does with each signal.

Durable workflows

Model retries and resumed work as linked traces.