Durable workflow traces

Durable workflows can pause for days, resume in another process, and retry a step many times. Model each active attempt as a finite span, and link resumed work to the earlier attempt, instead of one span that stays open for the full workflow.

Prerequisites

A service that sends traces to Squasher with an OpenTelemetry SDK. See OpenTelemetry.

Name the workflow on each span

Put these attributes on the root span of each attempt and on important child spans:

Attribute	Value
`squasher.workflow.id`	Stable workflow name, such as `invoice-delivery`
`squasher.workflow.run_id`	ID of one workflow run
`squasher.workflow.step`	Stable step name
`squasher.workflow.attempt`	Attempt number, starting at `1`
`squasher.workflow.checkpoint_id`	ID of the checkpoint or resume token

These names are a convention. Squasher stores them as ordinary span attributes and gives them no special meaning. Do not put credentials, full payloads, or customer content in them.

Link a retry to the failed attempt

Use a parent span when work continues in one live call. Use a span link when a retry starts after the earlier trace ended, when a checkpoint resumes in a new process, or when one step starts several branches.

const retrySpan = tracer.startSpan("workflow.step", {
  attributes: {
    "squasher.workflow.id": "invoice-delivery",
    "squasher.workflow.run_id": workflowRunId,
    "squasher.workflow.step": "send-email",
    "squasher.workflow.attempt": 2,
  },
  links: [{ context: previousAttemptSpanContext, attributes: { "squasher.link.reason": "retry" } }],
});

previousAttemptSpanContext is the span context that you stored with the earlier attempt. If you did not store it, do not invent a parent: the workflow attributes still connect the attempts.

Record checkpoints and outcomes

Record a checkpoint during an active span as an event: span.addEvent("workflow.checkpoint", { "squasher.workflow.checkpoint_id": checkpointId }).
End each attempt span when the attempt stops, not when the workflow ends.
Set the span status to ERROR when an attempt fails. That creates an issue in Squasher.
Record error.type on the failed span. Squasher uses it to group the issues.
Record the final outcome as an event or a short completion span.

Verify

Run one workflow that pauses and resumes, and one that fails once and then succeeds.
In Squasher, select Traces in the sidebar.
Open the retry span. Its Links tab shows the earlier attempt, and its Attributes tab shows the workflow attributes.
Select Issues in the sidebar. The failed attempt is an issue.

Get started

Guides

Durable workflow traces

Prerequisites

Name the workflow on each span

Link a retry to the failed attempt

Record checkpoints and outcomes

Verify

Next steps

Tail sampling

Design queryable logs

​Prerequisites

​Name the workflow on each span

​Link a retry to the failed attempt

​Record checkpoints and outcomes

​Verify

​Next steps

Tail sampling

Design queryable logs

Prerequisites

Name the workflow on each span

Link a retry to the failed attempt

Record checkpoints and outcomes

Verify

Next steps