I had just finished setting up the Telegram notification system (see previous chapter) and was feeling pretty good about the server. Monitoring in place, services checked every 5 minutes, VPN connections logged. I went to bed thinking "finally, everything is under control."

Then my phone started buzzing.

9 PM: It Begins

The first Telegram notification came in around 9 PM:

🚫 fail2ban: Banned [ATTACKER_IP] from sshd for 600 seconds

Cool, fail2ban doing its job. Some bot found my server and tried to brute-force SSH. Nothing new — this is the background noise of running anything on the internet. I'd seen it before (remember Chapter 2?). The IP gets banned for 10 minutes, the bot moves on. No big deal.

Except this bot didn't move on.

The Spam

Ten minutes later:

✅ fail2ban: Unbanned [ATTACKER_IP] from sshd

And immediately after:

🚫 fail2ban: Banned [ATTACKER_IP] from sshd for 600 seconds

The same IP. Banned, waited exactly for the ban to expire, came right back, got banned again. Over and over and over. My Telegram was lighting up every 10 minutes like clockwork. By midnight I had dozens of ban/unban notifications from the exact same address.

This wasn't a random scan anymore. This was a bot that had found my server and was stubbornly hammering it, 24/7, waiting out every ban and trying again immediately.

"The definition of insanity is doing the same thing over and over and expecting different results. Bots don't know about insanity."

Am I Actually at Risk?

Short answer: no. But the investigation revealed something I should have caught much earlier.

  • SSH requires key-only authentication — there's no password to brute-force. Even if the bot tried every password in every dictionary in every language, it would never get in because passwords are disabled entirely.
  • SSH should be behind the VPN — wait, then how is the bot even reaching SSH? The whole point was that we never forwarded port 22 on the router... right?

I checked my router's port forwarding table. And there it was.

Port 22 was forwarded.

I have no idea when that happened. Maybe I added it during the initial setup when I was still figuring things out. Maybe I added it during a debugging session and forgot to remove it. Either way, SSH had been exposed to the entire internet this whole time. Every bot, every scanner, every script kiddie could reach my SSH port directly.

The good news: password authentication was disabled and SSH keys were required, so nobody actually got in. fail2ban was also doing its job banning the attempts. But still — the door was supposed to be hidden behind a wall, and instead it was right there on the street with a big neon sign saying "TRY ME."

The Immediate Fix

Removed the port 22 forwarding rule from the router. Done. The only forwarded ports now are:

  • 80 TCP — HTTP (website)
  • 443 TCP — HTTPS (website)
  • [VPN_PORT] UDP — OpenVPN

Port 22 is still open in UFW because I need SSH accessible from the local network and through the VPN tunnel. But without the router forwarding it, the internet can't reach it anymore. The bots can knock all they want — there's no door to knock on.

After removing the forwarding rule, the brute-force attempts stopped immediately. My Telegram went silent. The problem was never fail2ban's ban duration — it was that SSH was exposed in the first place.

Lesson Learned

Always check your router. You can have the best firewall, the strongest SSH keys, the most aggressive fail2ban config — but if your router is forwarding a port you forgot about, you're fighting a battle that shouldn't exist. I spent hours setting up recidive jails when the real fix was deleting one line from the router config.

The Extra Layer: Recidive Jail

Removing the port forwarding fixed the immediate problem, but I still wanted better fail2ban behavior for the future. The default 10-minute ban was clearly too short — if anything ever does get through to a service, I want escalating bans, not a revolving door.

fail2ban has a built-in feature for exactly this: the recidive jail. It monitors fail2ban's own log and catches IPs that keep getting banned repeatedly.

Updated /etc/fail2ban/jail.local:

[DEFAULT]
action = %(action_)s
         telegram
bantime = 1h
findtime = 10m
maxretry = 3

[sshd]
enabled = true

[recidive]
enabled = true
bantime = 1w
findtime = 1d
maxretry = 3

What This Does

Two layers of banning:

  1. First offense: 3 failed attempts in 10 minutes → banned for 1 hour (upgraded from 10 minutes)
  2. Repeat offender: 3 bans in 24 hours → recidive jail kicks in → banned for 1 week

The escalation path looks like this:

Attempt What Happens Ban Duration
1st ban sshd jail catches it 1 hour
2nd ban Bot comes back, banned again 1 hour
3rd ban Recidive jail triggers 1 week
After 1 week If they come back, cycle repeats Another week

After restarting fail2ban, the persistent bot got caught by recidive within a few hours and was silenced for a week. My Telegram went quiet. Sweet, sweet silence.

The Nuclear Option

For truly persistent IPs that you never want to hear from again, there's always the manual route:

sudo ufw deny from [ATTACKER_IP]

This adds a permanent firewall rule — the IP is dropped at the network level before it even reaches fail2ban or any service. I didn't end up needing this since recidive handled it, but it's good to know it's there for extreme cases.

What I Learned

A few takeaways from my first real "attack" (if you can even call a dumb bot an attack):

  • Check your router. This is the biggest one. I had all these layers of security — SSH keys, fail2ban, VPN-only access — but port 22 was forwarded on the router the whole time. The real fix was one click in the router admin panel. Always verify what's actually exposed before hardening what's behind it.
  • The notification system works. I knew about the attack within seconds. Before the Telegram bot, this would have gone unnoticed in the logs for weeks.
  • Default fail2ban settings aren't enough. A 10-minute ban is a joke for persistent bots. Bump it to at least 1 hour and enable recidive.
  • Notifications can become the problem. The irony of building a monitoring system is that the monitoring itself can overwhelm you. The recidive jail doesn't just protect the server — it protects my sanity by reducing the notification spam.
  • Defense in depth saves you. Even with SSH exposed, the bot couldn't get in. SSH keys meant there was no password to brute-force. Every security layer is independent — if one fails (or in this case, if you forget to close a port), the others still hold.
  • Bots are dumb but relentless. They don't give up, they don't get bored, they don't sleep. Your defenses need to be equally tireless — which is why automation (fail2ban, recidive) beats manual blocking every time.

Current Security Stack

After this incident, here's the full defense setup:

  • UFW firewall — only necessary ports open
  • fail2ban — 1-hour bans after 3 attempts
  • recidive jail — 1-week bans for repeat offenders
  • SSH key-only — no passwords to brute-force
  • SSH behind VPN — not directly exposed to internet
  • Non-default VPN port — hard to find
  • tls-crypt-v2 — VPN server invisible without the key
  • UDP rate limiting — flood protection on VPN port
  • Cloudflare proxy — real IP hidden for web traffic
  • Nginx rate limiting — 10 req/s per IP
  • Telegram notifications — real-time alerts for everything
  • UptimeRobot — external monitoring when server is offline
  • Custom MOTD — service status at a glance on login

For a laptop in a basement, that's a pretty solid fortress. The bot that wouldn't quit was the first real test of the whole system, and everything held. I'll take that as a win.

What's Next

  • Containerization with Docker
  • UPS (power outages are still the #1 enemy)
  • Self-hosted drive with NAS
  • Mail server
  • Local LLM (still dreaming)