Skip to content
lube

Incidents

Auto-detectTimelinePostmortemMaintenance

Detect, coordinate, and close every incident.

When a monitor fails, the incident is already open and deduped. Run the response on a live timeline with a lead and responders, then close it out with a governed postmortem and a scheduled maintenance window.

Checkout API degradation
p1major#231
leadA. RiveraresponderJ. Okafor
  • monitoringpublicjust now

    Error rate back to baseline. Watching before we resolve.

    A. Rivera

  • identifiedinternal6m ago

    Traced to a saturated connection pool on the checkout service.

    J. Okafor

  • investigatingpublic14m ago

    We are investigating elevated errors on checkout.

    A. Rivera

Live incident timeline

Its place in the loop

Incidents turn a detected outage into a coordinated response.

One connected loop, held on the stage this capability serves. The other stages stay as context so you can see what feeds in and what comes next.

BD
Discovery
Feedback
Product
Engineering
Release
Reliability
Evidence

Stage inventory

Reliability

Know whether it held.

Web performance

Review Lighthouse report evidence against configured targets.

Available now

Test intelligence

Investigate test outcomes and flakiness evidence across runs.

Available now

Coming later reflects product vision, not a delivery commitment.

It opens itself, once

Automatic detection, deduped into one incident.

A failing uptime monitor opens an incident on its own. Repeated failures from the same monitor fold into that single incident instead of paging you per check, and when the monitor recovers it auto-resolves.

checkout-api

failing
  • check failed · 50214:02:11
  • check failed · 50214:03:11
  • check failed · timeout14:04:11
deduped into one
Checkout API degradationopened

#231 · automated · one incident, not one page per check

on recovery it auto-resolves

Automatic detection with dedupe

Know what it touches

Every incident carries its blast radius.

The incident knows which monitors it affects and their current status, which public status pages it appears on, and how many people subscribed for updates. Scope it to the whole organization or to a specific list of monitors — this is impact accounting, not a guess at cause.

#231 impact

scoped monitors

affected monitors

  • checkout-apidown
  • payments-apidegraded
  • web-frontendup

shows on status pages

  • Public status/status
  • Partner status/partners
subscribers on this incident12
Affected monitors, status pages, subscribers

Coordinate the response

One live timeline, the rest is history.

Incident timeline

Only the incident still in flight moves.

Every incident sits on a time-ordered rail carrying its severity and priority. Resolved ones go quiet; only the active one pulses, so the thing that needs you is the thing you see.

p1 to p4critical to cosmeticreopen chain

incident history

  • Elevated 5xx on api#218 · p2 highresolved9d ago
  • Certificate renewal#224 · p4 lowresolved5d ago
  • Checkout API degradation#231 · p1 criticalactivenow
Incident history

Lifecycle

Open, work it, resolve — and reopen if it comes back.

An incident moves through a clear set of states. A resolved incident is never truly gone: if the same problem returns, it reopens as a linked follow-up so the history stays connected.

automated or manualresolvereopen as linked

incident state

  1. 1open
  2. 2acknowledged
  3. 3resolved
resolved → reopened as a linked follow-up
Incident lifecycle

Close it out

The write-up and the next window are here too.

Postmortem

A postmortem you have to approve before it ships.

Write the postmortem in markdown and move it through draft, approved, published, and withdrawn. Publishing needs approval first, and every save is versioned so two editors cannot quietly overwrite each other.

draft → approvedpublishedwithdrawn

Checkout API degradation — postmortem

rev 4
  1. draft
  2. approved
  3. publishedhere
  4. withdrawn

approval before publish · versioned saves

Governed postmortem lifecycle

Maintenance

Schedule the work, mute the noise, tell the world.

A maintenance window covers a fixed span and the monitors it touches. Switch on alert suppression so planned work does not page anyone, and show it on your status page so customers know it is planned.

scheduledsuppress alertson status page

DB migration

scheduled
Sat 02:00Sat 03:30
  • Suppress alerts
  • Show on status page

affected monitors

checkout-apidb:5432 (tcp)nightly-sync
Scheduled maintenance window

What this is

Incidents is the coordination record: detection, a timeline, responders, a governed postmortem, and maintenance windows. It does not guess at root cause or wire your outage to a deploy — it keeps repeated failures from the same monitor as one honest incident and gives your team one place to run it.

Every state, spelled out

The vocabulary of an incident.

The states, severities, and controls a real incident moves through — the same set the dashboard uses.

Severity & priority

Two independent scales.

criticalmajorminorcosmeticp1p2p3p4

Update timeline

Each update, public or internal.

investigatingidentifiedmonitoringresolvedpublicinternal

Response & scope

Who and what it covers.

leadrespondersubscribersorganizationspecific monitors

Postmortem

Governed and versioned.

draftapprovedpublishedwithdrawn

Maintenance

Planned, not surprising.

scheduledactivecompletedcancelledsuppress alerts

Source

How it started.

automatedmanualdedupedauto-resolvereopen

Around the loop

What feeds incidents, and what carries them out.

Be ready before the next one.

Start free

Connect a monitor and your incidents open themselves.