Monitors and incidents

A monitor checks one thing on a schedule: an endpoint, a cron job, or a query on your logs. When the check fails, the monitor opens an incident. When the check passes again, the incident resolves.

Monitor types

Type	Checks	Plan
`http`	An HTTP response: the status code, and optionally the time and a text in the body.	All plans
`query`	A number from your data against a threshold, for example error logs a minute. It can use logs, error events, request latency, ingest delay, metrics, or Web Vitals.	All plans
`heartbeat`	That a cron job or a worker calls its heartbeat URL in time.	Team and above
`keyword`	That a page contains, or does not contain, a word.	Team and above
`ssl`	That the TLS handshake works and the certificate is valid. See SSL certificate expiry.	Team and above
`tcp`	That a TCP port accepts a connection.	Team and above
`dns`	A DNS record of the type `A`, `AAAA`, `CNAME`, `MX`, or `TXT`.	Team and above
`domain`	That a domain resolves to an address, and optionally to the addresses that you expect.	Team and above
`smtp`, `pop3`, `imap`	A mail server.	Team and above

SSL certificate expiry

An ssl monitor reads the server certificate on each check and records when it expires and who issued it.

Certificate	Check result
Expired	Failed
Expires within `warningDays` (default 14)	Degraded
Expires later	Healthy

The certificate is read in a TLS 1.2 handshake. A server that offers only TLS 1.3 still gets the handshake check, but Squasher cannot read its expiry date. The SSL certificates report lists the expiry of every ssl monitor (see Reliability reports).

Create a monitor

In the dashboard:

Select Monitors in the sidebar, under Alerting, then select New monitor.
Enter a Name and choose the Type.
Enter the target: a URL for http and keyword, or a host name such as example.com for ssl, domain, dns, tcp, and the mail types. A heartbeat monitor needs no target; Squasher makes its token.
Choose the Check frequency and the Failed checks before an incident, then fill in the fields for the type, for example the expected status codes, the keyword, or the port.
Select Create monitor.

Types that your plan does not include show (upgrade required). Frequencies below the minimum of your plan show (paid plans). A heartbeat can also expect a call every 6, 12, or 24 hours. When the project has all the monitors that the plan includes, the form shows the limit. For a query monitor, the form also shows a Backtest panel. See Backtest a query monitor. With the CLI, sign in first with squasher login. The fastest way is a template. This command creates an HTTP monitor that checks for a 200 response every 2 minutes:

squasher monitors create-from-template --project "$SQUASHER_PROJECT_ID" http-health \
  --inputs '{"url":"https://example.com/health"}'

To list the templates, run squasher monitors templates --project "$SQUASHER_PROJECT_ID". The templates are http-health, error-burst, ingest-silent, internal-auth-failures, infrastructure-collector-offline, and infrastructure-high-cpu. To set every value yourself, use squasher monitors create with a JSON config:

squasher monitors create --project "$SQUASHER_PROJECT_ID" --name "API health" --kind http \
  --target https://example.com/health \
  --config '{"method":"GET","expectedStatusCodes":[200],"timeoutMs":8000,"intervalMinutes":5,"failureThreshold":2,"recoveryThreshold":1}'

Use squasher monitors upsert with the same flags to create or update a monitor by name. See Monitors API for every field.

How a monitor opens an incident

intervalSeconds sets how often the check runs: 30, 60, 120, 180, 300, 600, 900, 1,800, or 3,600 seconds. The shortest interval is 30 seconds on paid plans and 3 minutes on Free. Only http, keyword, tcp, dns, ssl, domain, and the mail types run more often than every 2 minutes; query and heartbeat monitors keep whole-minute schedules.
intervalMinutes, from 1 to 1,440, is the older form. intervalSeconds wins when you set both.
failureThreshold sets how many failed checks in a row open an incident, from 1 to 10.
recoveryThreshold sets how many passed checks in a row resolve it, from 1 to 10.
For http and keyword, degradedAtMs and failedAtMs mark a slow response as Degraded or failed.

You cannot run a check by hand.

Check from several locations

An http or keyword monitor can check from more than one location. Set regions to a list of location ids. Every location runs on each interval, so a monitor with three locations runs three checks per interval and uses three checks of your plan.

Location id	Area	Data centers near
`us-west`	Western North America	San Francisco, Los Angeles, Seattle
`us-east`	Eastern North America	Ashburn, New York, Chicago
`eu-west`	Western Europe	London, Amsterdam, Frankfurt
`eu-east`	Eastern Europe	Warsaw, Vienna, Stockholm
`asia-pacific`	Asia-Pacific	Singapore, Tokyo, Hong Kong
`oceania`	Oceania	Sydney, Melbourne, Auckland

A location is an area, not one city. Squasher runs the check in a data center in or near the area and records the data center code (for example FRA) on each check. The data center can change over time. A monitor without regions, or with regions: ["auto"], checks from us-east. Older region codes keep working and run from the location they name, for example iad from us-east, lhr from eu-west, and syd from oceania.

Plan	Locations per monitor
Free	1
Pro	3
Team	6
Scale	6

With more than one location, a monitor fails only when enough locations fail in the same interval. confirmation.requiredLocations sets how many; the default is 2. So with three locations, one failing location does not open an incident, and two do. A failure that moves from one location to another between intervals does not add up. With one location, a failure is checked again from a second location before the monitor fails.

{
  "method": "GET",
  "expectedStatusCodes": [200],
  "timeoutMs": 8000,
  "intervalSeconds": 60,
  "failureThreshold": 1,
  "recoveryThreshold": 1,
  "regions": ["us-east", "eu-west", "asia-pacific"],
  "confirmation": { "requiredLocations": 2 }
}

With the CLI, add --regions us-east,eu-west,asia-pacific --confirmation 2 to squasher monitors create, upsert, or update. On update without --config, the flags change the current config. After a downgrade, the monitor keeps checking from the first locations that the new plan allows. On Free, a round runs only when the monthly check allowance has room for every location of the round; without room the monitor pauses until the allowance resets.

Failure evidence

When an http or keyword check fails, Squasher keeps evidence with the check:

the DNS lookup time and the addresses it returned,
the TCP connect time and the TLS handshake time, and the certificate issuer and expiry,
the HTTP status and the response headers,
the first 2 KB of a text response body,
the location of the check and, for several locations, which locations failed.

In one interval, DNS, TCP, and TLS facts are collected for at most two failing locations; the others keep the HTTP evidence and show that the network facts were skipped. They are also skipped when the check has too little time left before its next run. Squasher removes credentials before it stores evidence: Set-Cookie, Cookie, Authorization, and headers that hold tokens, sessions, or keys, and tokens, passwords, emails, and long keys in the body. The result text of each check is cleaned the same way. It does not store binary bodies. Each network step has 1.5 seconds; a step that does not finish records its error. The TLS time is an estimate from a second connection. The Checks panel of a monitor shows the evidence of a failed check. The incident that the monitor opens links to the failed checks, and the incident evidence names the failing locations and the latest failure. To read checks from the API or the CLI:

squasher monitors checks --project "$SQUASHER_PROJECT_ID" <monitor_id> --failing-only

Checks that need other probes

Monitors run on an edge network that has no ICMP or UDP and no browser. So Squasher has no ping, UDP, or browser (scripted page) monitors. Use a tcp monitor for a port, and an http or keyword monitor for a page.

Backtest a query monitor

A backtest replays a draft query monitor over the last 24 hours or the last 7 days of real data. It uses the same evaluation as a live monitor and shows the alerts the rule would have opened. It saves nothing, so you can change the threshold and run it again. In the dashboard, open the query monitor form and select Run backtest in the Backtest panel. Choose 24 hours or 7 days. The chart shows the value of each check, the threshold (or the spike line), and shaded spans where an alert would have been open. Above the fields, the form states the rule in one sentence, for example “Alert when logs matching “level:error service:checkout-api” are above 50 per 5 minutes, 2 checks in a row.” To start from what you are looking at, select Create alert: in the Copy query menu on Logs (the current search), in the Copy query menu on Traces (the error logs of the same service), or next to the trend on Web Vitals (the selected metric). The form opens with the rule filled in. With the CLI, pass the draft config as JSON, as a file with @path, or with flags:

squasher monitors backtest --project "$SQUASHER_PROJECT_ID" --source logs \
  --query 'level:error' --aggregation count --comparator gt --threshold 100 --window 5m --range 7d

The command prints one line such as 2 alerts in the last 7 days (7 of 2016 checks failing), then a table of the alerts with their start, end, and peak value. Add --json for the full result. With the API, send the draft config to monitors.backtest. See Monitors API.

A backtest replays at most 7 days, and the rule window can be at most 7 days.
It uses the query budget of the project. A 503 response means the budget is busy; try again later.
A long range can read wider time buckets than the rule. Then the result marks the resolution as approximate and its notes say what changed.
A percentile over more than one time bucket is an estimate. The chart shows the estimated values, but the backtest does not count alerts for these rules.
Each read has a byte and time limit. Past it, the backtest stops and asks for a shorter range or a narrower query.
A backtest needs permission to read monitors and telemetry.
A check without enough data is marked not evaluated, with a reason. For example, data newer than the ingest watermark can still arrive, and a spike rule needs enough history for its baseline.

Alert on request latency

A query monitor with source traces alerts on how long your requests take. A request is an incoming span: a server span or a consumer span. The monitor measures the p50, p95, or p99 duration of the requests in its window, in milliseconds.

Field	Value
`source`	`traces`
`aggregation`	`p50`, `p95`, or `p99`
`serviceName`	Optional. Only the requests of this service.
`operation`	Optional. Only requests with this span name or HTTP route, like `/cart/{id}`.
`environment`	Optional. Only requests from this deployment environment.

Each percentile covers every request in the whole window. The live check, the backtest, and GET /v1/projects/{project_id}/trace-latency read the same numbers, so a backtest shows the alerts the monitor would really have opened. A window without requests has no data and follows the monitor’s noDataPolicy. The latency source is different: it measures ingest delay, the time from when an event happened to when Squasher received it.

squasher monitors backtest --project "$SQUASHER_PROJECT_ID" --source traces --aggregation p95 \
  --service-name checkout --operation '/cart/{id}' --comparator gt --threshold 800 --window 15m

In the dashboard, choose Request latency as the data source in the query monitor form.

Send heartbeats from a job

A heartbeat monitor waits for your job to call its URL. If no call arrives within the interval plus graceMinutes, the monitor fails.

curl -fsS https://api.squasher.ai/v1/heartbeats/<token>

Call /v1/heartbeats/<token>/fail to report a failed run, or /v1/heartbeats/<token>/<exit_code> to send the exit code of the job. A code other than 0 is a failure. GET and POST both work.

Watch your monitors

Select Monitors in the sidebar, under Alerting. The list shows the state of each monitor. Open a monitor to see its uptime, its latency, its failed checks, the Check history, and the linked incidents.

Respond to an incident

Select Incidents in the sidebar. The list has the views Active, Resolved, and All. An incident has four tabs:

Overview: the summary, then the root cause, the response, and the suggested controls when Squasher wrote them. Under them, Activity shows the comments and every event of the incident on one timeline, newest first. Each event shows who did it, an icon for what happened, and the status or severity it set. Identical automatic events in a row show as one line with a count. Select Add a comment above the timeline to write a comment.
Evidence: the issues, deploys, and queries attached to the incident. Collect evidence searches new issues, deploys, error logs, failed spans, and the monitor’s checks from 30 minutes before to 30 minutes after the start. It attaches only what is not attached yet, adds one timeline entry, and sends no notification. The tab then says what it added, or that it found nothing new, and which searches failed.
Communications: the status page the incident is on, who sees each update, a preview of the page and the subscriber email, the next public update, and every update that was posted.
Follow-ups: the Linear issues for the work that prevents the next incident, with the status, owner, and due date from Linear. Copy prompt asks your coding agent to draft follow-ups as an incident comment. The tab also shows the post-incident review link.

The right column shows:

Properties: the status and severity (select one to change it), when the incident started, and how long it is open or lasted.
People: the assignee, and whether the Squasher Agent works on the incident.
Links: Source, where the incident came from (the monitor, the alert integration with its alert ID and a link back to the alert, or the person who declared it), the status page, the post-incident review, and the follow-ups. Open in Datadog (or the name of your tool) opens the alert in the tool. When the tool sent only a link to another site, such as a runbook, the button names that site instead.
Timestamps and durations: when the incident started, was acknowledged, resolved, and last updated, and the time to acknowledge and to resolve.

More incident actions (⋯) in the header also opens the status page update, a new follow-up, and the evidence.

Change the status or reopen an incident

An incident is Investigating, Identified, Monitoring, or Resolved. Select Resolve in the header, select another status in More incident actions (⋯), or change the Status field. In the incidents list, select the status of a row to change it there, like the severity and the assignee. You can move the status forward or back. To open a resolved incident again, select Reopen in the header. The incident moves to Investigating. To reopen it as Identified or Monitoring, use ⋯ or the Status field. When you reopen an incident, Squasher:

clears the resolved time and the acknowledged time,
adds a Reopened event with your name to the activity,
sends the opened event again, so your notification routes post again and on-call paging starts a new page.

A monitor incident that you reopen can resolve again when its monitor recovers. A published incident shows as active on its status page again; post a public update to tell your users why. Comments support Markdown. Type @ to mention a teammate. To link another object, type its ID (iss_42 for an issue, inc_12 for an incident) or paste its Squasher link. The comment shows a link with the object’s icon. Links to an error, a monitor, or a status page, to a Slack channel or thread, to a GitHub pull request or issue, and to a Linear issue show the same way. Comments stay inside Squasher. An incident appears on a status page only when you publish it. See Status pages. To get incidents in Slack, PagerDuty, email, or another channel, set up notifications.

Get started

Guides

Monitors and incidents

Monitor types

SSL certificate expiry

Create a monitor

How a monitor opens an incident

Check from several locations

Failure evidence

Checks that need other probes

Backtest a query monitor

Alert on request latency

Send heartbeats from a job

Watch your monitors

Respond to an incident

Change the status or reopen an incident

Next steps

Set up notifications

Status pages

​Monitor types

​SSL certificate expiry

​Create a monitor

​How a monitor opens an incident

​Check from several locations

​Failure evidence

​Checks that need other probes

​Backtest a query monitor

​Alert on request latency

​Send heartbeats from a job

​Watch your monitors

​Respond to an incident

​Change the status or reopen an incident

​Comment and link related work

​Next steps

Set up notifications

Status pages

Monitor types

SSL certificate expiry

Create a monitor

How a monitor opens an incident

Check from several locations

Failure evidence

Checks that need other probes

Backtest a query monitor

Alert on request latency

Send heartbeats from a job

Watch your monitors

Respond to an incident

Change the status or reopen an incident

Comment and link related work

Next steps