Our fail2ban was installed. It was configured, with four jails covering SSH and three application surfaces. It was enabled in systemd, which is the box's promise to start it at every boot. And for 95 days it was not running, and nobody noticed, because every check a person would casually run looked exactly the way it looks when everything is fine.
That is not a story about carelessness. It is a story about a class of check that cannot fail loudly — and the reason this guide exists is that the fix is not "check more often," it is check something else. This is the fifth leg of a series about the unglamorous half of running your own box: migrate, deploy, restore, watch, and now prove. Every one of them turns on the same discipline: the artifact that says a thing is configured is never the artifact that says it works.
What 95 Days of Silence Looks Like From the Outside
Here is what we can prove from the box itself, re-read this morning. It rebooted at a known moment in the spring — uptime -s and the reboot ledger agree on the timestamp to the second. The fail2ban unit is UnitFileState=enabled, so systemd started it on that boot. And systemd records the unit entering its inactive state eighteen seconds after that boot began, and not leaving it until an afternoon in early August. Log to log, systemd to systemd, both measurements land in the same place: 95 days, not "a few months."
What the box cannot tell us is why. There is not one ERROR or CRITICAL line anywhere in fail2ban's own log across its whole history — the failed start never got far enough to write about itself. We have an internal note guessing at a cause; it is a guess, it is undated, and it is not in this article, because a guess in a guide about verification would be a small joke at our own expense. The event is proven. The cause is not.
Why Nothing Caught It
The service was enabled, which reads like a health claim and is not one — it describes what systemd intends at boot, not what survived it. Its restart counter sat at zero, which looks reassuring and only means nothing tried again. And you cannot go looking in the obvious place: fail2ban on this box writes to its own log file rather than the journal, so journalctl -u fail2ban returns “-- No entries --” for every date you ask about — not because nothing happened, but because nothing was ever there. (The journal also keeps roughly a week here anyway, which is its own lesson from the monitoring guide and the reason that guide exists.) An unhelpful command returning empty output looks identical to good news. That is the whole trap.
The Check That Isn't One
Most hardening guides end with a verification step that goes roughly: grep your firewall for the f2b-<jail> chain and confirm it's there. We believed that too, and then we tested it, and the test disproved the advice.
Running the drill below against a jail that had never banned anyone, we found no chain at all — and then watched fail2ban create the chain and its jump from the main input path at the exact moment of the first ban. After the unban, the chain and the jump stayed; only the rejection rule went away. So the firewall's shape tells you a jail's history, not its readiness:
- No chain means this jail has never banned anybody. That is a completely normal resting state for a jail watching a surface nobody has attacked.
- A chain with nothing but a RETURN rule means this jail has banned someone before and nobody is banned right now.
- Neither one tells you whether the jail can act. The only way to know a jail can act is to make it act.
The same reasoning applies to the jail counters. Three of our four jails report zero matches ever — and that is not a fault. Their filters watch surfaces that have not been probed. Reading an empty counter as breakage is how people end up rewriting a working filter at midnight.
The Drill: Four Commands That Settle It
Everything above converts into one short, reversible exercise you can run on your own box in under a minute. It is the only readiness test we now trust, and we re-ran it this morning against production before publishing this sentence.
# 1. Look at the chain BEFORE, so you can see the change rather than guess at it
iptables -S f2b-sshd
# 2. Ban a documentation address (RFC 5737 TEST-NET-1 — can never be a real client)
fail2ban-client set sshd banip 192.0.2.1
# 3. Confirm it reached BOTH systems: fail2ban's own view, and the firewall
fail2ban-client get sshd banned
iptables -S f2b-sshd
# 4. Put it back, then prove nothing of yours is left anywhere
fail2ban-client unban 192.0.2.1
iptables -S | grep -c '192.0.2.'What you should see, and what we saw today: the chain starts with a bare RETURN rule; the ban command answers 1; the chain gains a REJECT rule for that address, placed above the RETURN; fail2ban's own banned list shows the address; the unban command answers 1; the REJECT rule vanishes; and the final residue count is 0. That sequence is the proof, because it exercises the entire path a real ban travels — the client, the server, the action, and the kernel's own rule table — instead of asking any one of them to vouch for itself.
Two honest details about the drill. The cumulative “total banned” counter for the jail does not decrement on unban, so your test permanently adds one to a number you may later read as attack volume — ours are in there, and this article is where we say so. And on a jail that has never banned anyone, the drill creates the chain and the jump, which then remain: it is reversible in effect, not in every artifact. Both are harmless. Both are the kind of thing you would rather read here than discover in six months while trying to reconstruct what a number means.
Why 192.0.2.1
RFC 5737 reserves three ranges purely for documentation, and TEST-NET-1 is the one used above. No real host can ever legitimately hold that address, which makes it the safest possible thing to ban on a production machine: if you fumble the unban, the worst case is that you have blocked an address nobody can be using. Never rehearse with a real address, and never with your own.
Read the Right Clocks
When you do want to know whether the service has been up the whole time, one timestamp is not enough — ours told a clean-sounding story that was wrong. Read five, and read them together:
uptime -s # when the box last booted
last -x reboot # the boot ledger, independent of systemd
systemctl show fail2ban -p UnitFileState \
-p ActiveEnterTimestamp -p ActiveExitTimestamp \
-p InactiveEnterTimestamp -p InactiveExitTimestamp -p NRestartsAn enabled unit whose inactive-enter timestamp lands seconds after a boot, and whose inactive-exit timestamp lands months later, is a boot-time start that failed and was never retried — which is precisely the shape our silence had. If ActiveEnterTimestamp matches the last boot, the service has been up as long as the machine has. If it does not, the gap between those two clocks is your outage, and no amount of “it says enabled” will close it.
A Note on Where the Config Actually Lives
One practical thing that trips people up, because nearly every fail2ban guide on the internet says otherwise. Those guides tell you to edit jail.local. This box does not have one — its configuration lives entirely in drop-in files under jail.d/, which is a perfectly standard arrangement and means a merchant following the standard advice would be editing a file that has no effect because it does not exist. Before you tune anything, find out which file your box is actually reading. And whatever your thresholds are, write them where a person can find them: the counts and windows are a policy decision, and a policy nobody can locate is not one.
Our jails are deliberately configured to ban only — no whois lookups, no email per ban. That is not minimalism for its own sake; it is a division of labour. Telling a human that something is happening is the monitoring rail's job, and it does it with deduplication and a memory, which a per-ban email cannot. Ban here; notify there.
The Ladder, Bottom to Top
The five legs of this series all end up in the same place, so here it is stated once, in the order of how much each claim is actually worth:
- It's installed. Worth almost nothing.
- It's enabled. Worth nothing about right now — ours was enabled for all 95 silent days.
- It's active. Worth something, and still not proof that any rule fires.
- Its jails are listed and its filters match. Better — but an untouched surface reports zero forever.
- You banned an address and watched the rule appear in the kernel, then removed it and watched it go. That is the only rung that proves the machinery works, and it is four commands.
Same shape as the backup that has never been restored: the config file is a statement of intent, and intent is not the thing you needed. Run the drill on a calm afternoon. If it works, you have spent a minute. If it does not, you have just found out on a Tuesday instead of during an incident — and that difference is the entire product of this series.
The Rest of the Series
Migrate, deploy, restore, watch, prove — five guides on the unglamorous half of running your own infrastructure, each one built around a check you can actually run.
Knowing It Broke Before Your Users Do