> ## Documentation Index
> Fetch the complete documentation index at: https://docs.squasher.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Tail sampling

> Keep error and slow traces while reducing routine trace volume in your OpenTelemetry Collector.

Tail sampling waits for spans from a trace and then makes one decision for the
complete trace. It can keep traces with recorded error signals and slow traces
even when the root span looked successful at the start.

Run tail sampling in an OpenTelemetry Collector that you control before traces
reach Squasher. This reduces Squasher ingest volume without a second hosted
service.

<Warning>
  Tail sampling is stateful. The Collector keeps spans in memory while it waits for a decision. All
  spans for one trace must reach the same Collector instance.
</Warning>

## Start with a bounded policy

This example keeps traces when any span has `ERROR` status, when any span has a
recorded `exception` event, or when the trace is slower than one second. It also
keeps a 10 percent baseline sample of the remaining traces.

```yaml otel-collector-config.yaml theme={null}
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    expected_new_traces_per_sec: 100
    decision_cache:
      sampled_cache_size: 100000
      non_sampled_cache_size: 100000
    policies:
      - name: error-status
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: recorded-exceptions
        type: ottl_condition
        ottl_condition:
          error_mode: ignore
          spanevent:
            - 'name == "exception"'
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10
  batch: {}

exporters:
  otlphttp/squasher:
    endpoint: https://ingest.squasher.ai
    headers:
      x-squasher-key: "${env:SQUASHER_API_KEY}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/squasher]
```

The policies use OR behavior: a trace is kept when any keep policy matches.
Tune `num_traces`, caches, and `decision_wait` for your trace rate and maximum
normal trace duration. A larger buffer or longer wait uses more memory.

Recording an exception adds a span event. It does not set the span status to
`ERROR`. Instrumentation must record the exception event, set `ERROR` status,
or both. Failures that produce neither signal can only enter through the
baseline sample. Add policies for other error signals that your services use.

## Configure SDK sampling

Tail sampling cannot restore a trace that an SDK dropped. Send all candidate
traces from the SDK to the tail-sampling Collector, usually with an `always_on`
head sampler. Keep a lower SDK head-sampling rate only when application
overhead is more important than complete tail decisions.

Do not apply independent random head-sampling decisions in each service. That
can produce partial traces before the Collector sees them.

## Scale safely

For one Collector instance, send every service in the trace to that instance.
For several instances, route by trace ID through an OpenTelemetry
load-balancing exporter before the tail-sampling layer. A normal round-robin
load balancer can split one trace and cause an incorrect decision.

Keep processors that need receiver context, such as Kubernetes attribute
enrichment, before `tail_sampling`.

## Monitor the Collector

Watch the Collector's own metrics for:

* traces removed before a decision
* late spans
* sampling-decision latency
* exporter failures
* accepted and exported spans

If traces are removed too early, reduce `decision_wait`, reduce incoming
volume, or increase `num_traces` and available memory. Change one control at a
time and compare the retained error and slow-trace counts.

## Verify

1. Run the pipeline without tail sampling and record a short baseline.
2. Enable tail sampling for one service.
3. Send a successful fast trace, a slow trace, a trace with `ERROR` span status,
   and a trace with a recorded exception event but unset span status.
4. Confirm that the slow trace and both error-signal traces always arrive.
5. Compare Collector accepted/exported counts and Squasher trace volume.
6. Widen the rollout only after late-span and early-drop metrics stay healthy.

## Reference

* [OpenTelemetry tail sampling processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor)

## Agent handoff

```text theme={null}
Add customer-side OpenTelemetry tail sampling before Squasher for project <project_id>. Keep traces with ERROR span status, recorded exception events, or latency above the agreed threshold. Keep a small baseline sample, route every span in one trace to the same Collector instance, monitor early drops and late spans, and ask before changing shared SDK sampling.
```