How the watcher decides a site is down
What it checks
Every five minutes the watcher asks each site for its health route, a small address that answers with the commit the site is running and whether its database and its queue respond. A site with no health route is asked for its home page instead, and a 200 from there counts as alive and nothing more: it says the server answered, not that the site works.
Each check records the status code, how long the answer took, and the commit it reported. The watcher runs from one region, so a slow answer can mean a slow site or a slow path between the two. It does not try to tell those apart on one check; the thresholds below look at several in a row before they say anything.
Checks that time out after ten seconds are recorded as no answer, which is a state of its own and not a zero. A chart of response times leaves a gap there, hatched, so a quiet line is never mistaken for a fast one.
The watcher also reads each site's deploys from its platform, so a failure that starts a few minutes after a deploy is shown beside that deploy, with a link to roll it back.
Thresholds
One failed check is not an outage. Networks drop packets and platforms restart workers, and a person woken for each of those would soon stop reading the alerts. A site is marked degraded after two failed checks in a row, or when its median response time over the last hour is more than three times its median over the last week.
A site is marked down after three failed checks in a row, which is ten to fifteen minutes depending on where in the cycle the trouble began. That delay is the price of not crying wolf, and the page says so: the time shown is when the first of the three failed, not when the third did.
Thresholds are the same for every site on purpose. A site that needs a different one is a site whose health route is measuring the wrong thing, and the fix belongs there.
Usage against a platform's free limits has its own thresholds: a warning at three quarters used, or sooner when the day's projection runs out before the reset, and critical at nine tenths.
When it alerts
A site going down sends one message, and a site coming back sends one more. Nothing repeats while the state holds, because a repeated alert says nothing new and teaches its reader to ignore the channel it arrives on.
Degraded sends nothing on its own. It shows on the overview, worst first, with its reason in plain words, and turns into an alert only if it becomes down. Notices, such as a dependency with an update waiting, never alert and are collapsed under one row until someone opens them.
Every alert names the site, what failed, when it started, and the one thing to do first, which is usually to open the site's detail and read the last three checks. An alert that cannot say what to do next is a bug in the alert.
Alerts stop at night only for warnings. A site that is down is down at any hour, and the person it wakes chose that when they added it.
What it keeps
Each check is kept for seven days, which is enough to see a pattern across a week and short enough that the store stays inside the free tier. A day's checks are then folded into one row per site: how many answered, the median and the slowest response, and the commits seen.
Folded rows are kept for a year. They draw the uptime strip on each site's page, one slot per day, with days the watcher itself was not running drawn hatched and labeled as no data rather than as up.
Nothing the watcher keeps identifies a visitor. It asks its own questions of each site and records the answers; it never reads a site's logs or its visitors' requests.
The record of each change a person makes through the Portal, a resumed job or a revoked key, is kept separately and for good, under that person's name.