Knowing It Broke Before Your Users Do: The Fourth Leg of a VPS Deploy

This series has taught three legs of running your own box: deploy it, migrate it, and restore it. The deploy guide even builds a health endpoint worth gating on — and then mentions, in one passing clause, that "any uptime monitor" might hit it. That clause is the fourth leg, and it deserves better than a clause, because an endpoint is an app answering a question; monitoring is the process outside the app that asks on a schedule and decides whether a human hears about it. To be precise about the gap: the deploy guide already covers the one-time checks (confirm the certbot timer, run the dry-run once). What no article here has covered is the recurring version — the thing that asks every five minutes, forever, without training you to delete its emails. Everything below runs in production on our own VPS, and this article quotes its logs, its incidents, and its fixes, with dates attached — because a monitoring guide with no scar tissue is a brochure.

The Shape: One Script, One Cron Line, Three Kinds of Probe

The watcher is a shell script on a five-minute cron tick — */5 * * * * /opt/monitoring/health_check.sh — and it asks three kinds of question, in increasing order of depth. Unit checks: is each service on a curated watch list (currently 20 units on our box) still active per systemd? Reachability probes for the stateful pair: can we actually open a connection to MariaDB and Redis and get sane numbers back (thread counts, memory) — because a unit can read active while the daemon inside it refuses connections, which is why the databases are deliberately excluded from the unit list and given the stronger probe instead. Resource floors: disk over 80% pages someone, since every other failure gets loud eventually but a full disk gets quiet first. The exclusions from the watch list are as principled as the inclusions: cron and ssh are futile to watch from inside (if cron dies, the watcher never runs to tell you), OS plumbing isn't ours to page on, and — a subtler point — the box's timer-driven oneshot units (the certbots and weekly jobs) mostly aren't even in the enabled-services census, which leads to the first trap.

The Two Ways a Service Watcher Lies to You

Lie one: the timer-driven oneshot. A unit with Type=oneshot runs, exits, and reads inactive almost all the time — that's its healthy state. Put one on a watch list that alarms on "not active" and it alarms forever. Our watch script carries a comment block naming its known oneshots precisely so nobody "helpfully" adds them back.

Lie two: the phantom unit, and this one has a scar. systemctl is-active some-name returns inactive for a service that is down and for a name that resolves to nothing at all — a typo, or a service that was decommissioned and removed while the watch list kept its name. On August 3rd, our own journal recorded 187 ALERT lines for a unit that did not exist — one every five minutes for roughly sixteen hours, faithfully reporting that a long-deleted service was "NOT running," while a real renamed service went unwatched. (Those alerts went to syslog only; email delivery and dedupe didn't exist until the rewrite that same day, which is the only reason the story is funny instead of expensive.) The fix is structural, not vigilance: our installer extracts the watch list back out of the installed script and asserts every entry both exists as a unit file and is not a oneshot. We re-ran that assertion while writing this, on August 13th: watched=20 unwatchable=0. A watch list you can't mechanically validate is a list of things you remembered to believe.

Alert Fatigue Is the Failure Mode, So Dedupe Is the Feature

A five-minute tick with naive alerting is 288 emails a day for one down service — which trains you to ignore the alert, the exact failure a monitor exists to prevent. So the watcher's real sophistication is in what it doesn't send: a per-service state file gates re-alerts behind a six-hour cooldown; multiple down services batch into one email; and the recovery path only sends "RECOVERED" if the state file proves an alert actually went out — an all-clear for an alarm nobody received is noise wearing a checkmark. The mechanism has a production receipt, again from the August 3rd journal: fail2ban went down for real, the log recorded four ALERT ticks across seventeen minutes — and exactly one email went out, followed by one recovery mail when the service returned. Four ticks, one mail, one all-clear. (A note on those receipts: the journal on this box is a rolling window of about a week, so the August 3rd lines have since rotated out — we quote them with their dates from the incident record, which is itself a monitoring lesson: logs are evidence with a shelf life; write the dates down while they're still in the window.) There's also a --selftest flag that exercises alert → email → recovery end to end without faking an outage, and the cron line passes no arguments, so the self-test can never fire on a schedule.

The Cron Trap That Ate an Afternoon: The Thin PATH

Here's the one that will bite you first, stated precisely because the precision is the trap. On Debian, a user crontab (what you get from crontab -e) with no PATH= line runs your jobs under cron's compiled-in default — on our box, literally the string /usr/bin:/bin inside the cron binary. That is not the richer PATH you see in /etc/crontab, which applies only to /etc/crontab and /etc/cron.d entries. The consequence, measured on our box: of everything our monitoring calls, exactly one binary lives outside /usr/bin — nginx, at /usr/sbin/nginx. So a script that worked perfectly in an interactive shell failed only under cron, and only at its nginx step — the single most confusing kind of failure, because everything else succeeded. The production before/after sits in one day's journal: the Monday 06:00 certificate check failed from cron on August 3rd, and after pinning PATH at the top of the scripts (and teaching the installer to verify its work under env -i PATH=/usr/bin:/bin, cron's actual world), the same check ran clean that afternoon — same box, same certificates, nothing renewed in between. Pin the PATH in the script, not in your memory.

Certificates: The Recurring Check, and the Edge Your Disk Can't See

The deploy guide's one-time confirmation (certbot timer exists, dry-run passes) is necessary and insufficient, because certificate failure is a slow failure: certbot renews at 30 days out and retries twice daily, so a cert that has slipped under our alert threshold of 21 days has already survived roughly eighteen failed renewal attempts — that's the arithmetic that justifies the threshold. The weekly check walks every lineage that actually exists on disk (driven by the directory, not a hardcoded list — 17 lineages on our box as of August 13th), alarms under the threshold, retries the monthly dry-run once before alarming (a transient failure that passes on attempt two is logged, not paged — and our logs hold exactly that case from August 6th), and takes certbot's global lock as "skip, don't alarm."

Then there's the check a local scan structurally cannot do: your disk holds the certificate you have; only a network probe shows the certificate you're serving — and when a CDN or edge sits in front of you, those can differ. The check resolves each public hostname's A records through public DNS and handshakes each IP individually with SNI and full verification. The first time we ran it — August 6th, minutes after installing it, not yet from cron — it caught a real one: one of our six public hostnames was serving an expired certificate at two separate edge IPs while every local lineage on disk read 35+ days healthy. Origin fine, edge expired, invisible to every local check we had. (Incident closed on the 9th; the hostname now serves a certificate with weeks of margin, verified by the same probe.)

What the Watcher Watches Least Well: Its Own Blind Spots

Honesty section, because a monitoring article that implies total coverage is selling something. The sharpest lesson this box taught us is that artifact checks are not content checks: for months, our backup verifier confirmed each nightly dump was fresh, non-trivial in size, and gzip-valid — and all three of those gates pass for a dump that died mid-table. In early August we found exactly that: a truncated dump, megabytes large and gzip-clean, missing eight tables and its completion trailer, wearing a green checkmark. The fix (shipped August 9th) checks content: the dump's -- Dump completed trailer and its CREATE TABLE count against the live database's own table count — and it was proven by deliberately truncating a dump and watching three alerts fire. The same push added the leg this box had genuinely lacked: an encrypted off-host copy, shipped nightly over a write-only key and verified on the destination, with a restore drill run from a third machine. As of this writing the push has run every night since (26 encrypted dumps a night), and the newest local dump reads complete — all tables, trailer present, checked August 13th. What's still true and stays said: both boxes live with the same hosting provider, so this protects against host loss and compromise, not against the provider or the region — and the restore doctrine from the third leg still governs: a backup nobody has restored is a belief, and so is a monitor nobody has watched fail.

The Fourth-Leg Checklist

Deploy, migrate, restore, watch. The fourth leg is the cheapest of the four — a few hundred lines of shell and Python on a cron tick — and it's the one that converts the other three from hopes into facts on a schedule. Every mechanism above earned its place by failing first: the phantom unit paged nobody about something and everybody about nothing, the thin PATH broke exactly one binary, the edge served an expired cert over a healthy origin, and the greenest checkmark on the box was guarding a truncated file. The box will teach you the same lessons if you let it. Cheaper to read them here, with the dates attached.