Tail sampling

Tail sampling waits for spans from a trace and then makes one decision for the complete trace. It can keep traces with recorded error signals and slow traces even when the root span looked successful at the start. Run tail sampling in an OpenTelemetry Collector that you control before traces reach Squasher. This reduces Squasher ingest volume without a second hosted service.

Tail sampling is stateful. The Collector keeps spans in memory while it waits for a decision. All spans for one trace must reach the same Collector instance.

Start with a bounded policy

This example keeps traces when any span has ERROR status, when any span has a recorded exception event, or when the trace is slower than one second. It also keeps a 10 percent baseline sample of the remaining traces.

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    expected_new_traces_per_sec: 100
    decision_cache:
      sampled_cache_size: 100000
      non_sampled_cache_size: 100000
    policies:
      - name: error-status
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: recorded-exceptions
        type: ottl_condition
        ottl_condition:
          error_mode: ignore
          spanevent:
            - 'name == "exception"'
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
  batch: {}

exporters:
  otlphttp/squasher:
    endpoint: https://ingest.squasher.ai
    headers:
      x-squasher-key: "${env:SQUASHER_API_KEY}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/squasher]

The policies use OR behavior: a trace is kept when any keep policy matches. Tune num_traces, caches, and decision_wait for your trace rate and maximum normal trace duration. A larger buffer or longer wait uses more memory. Recording an exception adds a span event. It does not set the span status to ERROR. Instrumentation must record the exception event, set ERROR status, or both. Failures that produce neither signal can only enter through the baseline sample. Add policies for other error signals that your services use.

Configure SDK sampling

Tail sampling cannot restore a trace that an SDK dropped. Send all candidate traces from the SDK to the tail-sampling Collector, usually with an always_on head sampler. Keep a lower SDK head-sampling rate only when application overhead is more important than complete tail decisions. Do not apply independent random head-sampling decisions in each service. That can produce partial traces before the Collector sees them.

Scale safely

For one Collector instance, send every service in the trace to that instance. For several instances, route by trace ID through an OpenTelemetry load-balancing exporter before the tail-sampling layer. A normal round-robin load balancer can split one trace and cause an incorrect decision. Keep processors that need receiver context, such as Kubernetes attribute enrichment, before tail_sampling.

Monitor the Collector

Watch the Collector’s own metrics for:

traces removed before a decision
late spans
sampling-decision latency
exporter failures
accepted and exported spans

If traces are removed too early, reduce decision_wait, reduce incoming volume, or increase num_traces and available memory. Change one control at a time and compare the retained error and slow-trace counts.

Verify

Run the pipeline without tail sampling and record a short baseline.
Enable tail sampling for one service.
Send a successful fast trace, a slow trace, a trace with ERROR span status, and a trace with a recorded exception event but unset span status.
Confirm that the slow trace and both error-signal traces always arrive.
Compare Collector accepted/exported counts and Squasher trace volume.
Widen the rollout only after late-span and early-drop metrics stay healthy.

Reference

OpenTelemetry tail sampling processor

Agent handoff

Add customer-side OpenTelemetry tail sampling before Squasher for project <project_id>. Keep traces with ERROR span status, recorded exception events, or latency above the agreed threshold. Keep a small baseline sample, route every span in one trace to the same Collector instance, monitor early drops and late spans, and ask before changing shared SDK sampling.

Get Started

SDKs

Integrations

Migrations

Cloud Log Drains

Features

API Reference

Start with a bounded policy

Configure SDK sampling

Scale safely

Monitor the Collector

Verify

Reference

Agent handoff

​Start with a bounded policy

​Configure SDK sampling

​Scale safely

​Monitor the Collector

​Verify

​Reference

​Agent handoff

Start with a bounded policy

Configure SDK sampling

Scale safely

Monitor the Collector

Verify

Reference

Agent handoff