Ok so, the server kept crashing. Even after the C-State fix, even after removing the battery, even after setting CriticalPowerAction=Ignore... it still crashed. Not as often, but enough that I couldn't trust it. Every time I opened my phone to check if the site was still up, there was this little anxiety in my stomach. I was tired of debugging a system that was held together with duct tape and prayers.

So I decided to do the most drastic thing possible: nuke everything and start from zero.

New OS. New web server. New VPN. New security. Everything.

"Sometimes the best debugging technique is deleting everything and starting over with lessons learned."

Why I Decided to Rebuild

The old stack was Ubuntu Server + Apache + OpenVPN + Cloudflare (proxied). The crashes were the main trigger, but the real problem was deeper: the system had accumulated so much patching over the months that I wasn't even sure what was installed anymore. Every fix introduced a new config file, every crash left behind orphaned services, and the whole thing felt like a house of cards. I wanted something lean, intentional, and properly hardened from day one.

The New Stack

After some research (and a very long conversation with an AI that had strong opinions about Caddy), here's what I landed on:

  • OS: Debian 13 (Trixie) — minimal install, no desktop environment, just SSH and standard utilities from the installer. Leaner than Ubuntu, no snaps, no bloat, longer support cycle.
  • Web Server: Nginx — replaced Apache. More lightweight, better performance, and the config files are actually readable by humans.
  • VPN: OpenVPN — kept this one, it works well for my use case (connecting from restrictive networks abroad).
  • Firewall: UFW — simple, effective, no reason to overthink it.
  • TLS: Certbot with the Nginx plugin — automatic Let's Encrypt certificates.
  • DNS: Cloudflare — but this time, smarter (more on this below).

Part 1: The Install

Installed Debian 13 from a USB stick. During the tasksel screen I only selected:

  • ✅ SSH server
  • ✅ Standard system utilities

Everything else unchecked. No desktop, no web server (Apache comes with that option and I wanted Nginx), no print server. The result? A system using barely any RAM, ready to be configured exactly how I wanted it.

Fun fact: Debian minimal is SO minimal that sudo isn't even installed. Neither is curl. I had to:

su -
apt install sudo curl wget net-tools gnupg2 ca-certificates lsb-release
usermod -aG sudo [YOUR_USERNAME]

Then log out and back in for the group to take effect. Classic Debian experience ahahah.

Part 2: Firewall First

Before exposing anything to the internet, I locked down the firewall. This is the thing I should have done FIRST on the old server instead of fixing it after finding brute-force attempts in the logs (see Chapter 2...).

sudo apt install ufw
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow 22/tcp
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow [VPN_PORT]/udp
sudo ufw enable

Notice the port [VPN_PORT]/udp — that's OpenVPN on a non-default port. On the old server I used the standard 1194, which meant every script kiddie on the internet knew exactly where to poke. This time, random high port. Security through obscurity isn't security by itself, but it does reduce noise significantly.

Part 3: Nginx + Static Site

sudo apt install nginx
sudo systemctl enable nginx
sudo mkdir -p /var/www/mosearc.eu

Created the Nginx config at /etc/nginx/sites-available/mosearc.eu:

server {
    listen 80;
    listen [::]:80;
    server_name mosearc.eu www.mosearc.eu;

    root /var/www/mosearc.eu;
    index index.html;

    location / {
        try_files $uri $uri/ =404;
    }
}

Then enabled it and removed the default config:

sudo ln -s /etc/nginx/sites-available/mosearc.eu /etc/nginx/sites-enabled/
sudo rm /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl reload nginx

Clean. Simple. No .htaccess files scattered everywhere like Apache. Just one config file that does exactly what it says.

Part 4: HTTPS with Certbot

This was embarrassingly easy:

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d mosearc.eu -d www.mosearc.eu

That's it. Certbot modifies the Nginx config automatically, sets up the certificate, and creates a systemd timer that renews it before expiration. On the old server with Apache I had to mess around with virtual hosts and SSL modules. This was... just two commands.

Part 5: The Cloudflare Strategy

This is where I got smarter compared to the old setup. On the previous server, I had Cloudflare proxy (orange cloud) on my main domain. This was great for hiding my IP and DDoS protection, but it caused a problem: OpenVPN couldn't connect through Cloudflare's proxy because Cloudflare only handles HTTP/HTTPS traffic, not UDP.

The new strategy:

  • mosearc.eu → A record → Proxied (orange cloud) — website traffic goes through Cloudflare, my real IP is hidden.
  • [redacted].mosearc.eu → A record → DNS only (grey cloud) — VPN traffic goes directly to my IP, but the subdomain name is a random string that nobody can guess.

I also set Cloudflare SSL mode to Full (Strict) since I have a real Certbot certificate, and added the Cloudflare IP ranges to Nginx so it logs real visitor IPs instead of Cloudflare's:

# /etc/nginx/conf.d/cloudflare.conf
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 104.16.0.0/13;
# ... (all Cloudflare ranges)
real_ip_header CF-Connecting-IP;

Dynamic DNS

My ISP gives me a dynamic IP (thanks TIM), so I wrote a bash script that checks my public IP and updates both Cloudflare A records using their API. It runs once a day via cron. The script is smart enough to skip the API call if the IP hasn't changed:

IP=$(curl -s ifconfig.me)
OLD_IP=$(cat /tmp/cf-ddns-ip 2>/dev/null)

if [ "$IP" = "$OLD_IP" ]; then
    exit 0
fi

# ... update both A records via Cloudflare API ...

echo "$IP" > /tmp/cf-ddns-ip
logger "cf-ddns: Updated domains to $IP"

Part 6: OpenVPN (The Right Way)

Used the angristan/openvpn-install script which handles all the PKI, certificate generation, and iptables rules. On the old server I set up OpenVPN semi-manually and had certificate issues for weeks. This time:

curl -O https://raw.githubusercontent.com/angristan/openvpn-install/master/openvpn-install.sh
chmod +x openvpn-install.sh
sudo ./openvpn-install.sh

Changed the port to [VPN_PORT] in /etc/openvpn/server/server.conf after install, and updated the .ovpn client files to point to my secret subdomain instead of the raw IP.

The best part? Adding new clients for different devices is just re-running the script. Each device gets its own certificate, so if I lose my phone I can revoke just that one client without affecting everything else.

Part 7: SSH Behind the VPN

This is the biggest security improvement over the old setup. On the previous server, SSH was exposed to the internet (remember the brute-force attack from Chapter 2?). This time, SSH is not port-forwarded on the router at all. The only way to reach it is:

  1. Connect to OpenVPN (which requires a valid certificate)
  2. Then SSH to the local IP [SERVER_LOCAL_IP]

Two layers of authentication before you even see a login prompt. And I set up SSH key-only authentication too, so even if someone somehow gets through the VPN, they still can't brute-force a password because passwords are disabled:

# /etc/ssh/sshd_config
PermitRootLogin no
PasswordAuthentication no

Compare this to the old server where I discovered someone hammering root from [ATTACKER_IP] and had to scramble to fix it. Night and day.

Part 8: Hardening

The finishing touches:

  • fail2ban — installed and enabled. Watches SSH and Nginx logs, auto-bans IPs that try funny stuff.
  • unattended-upgrades — security patches install automatically. I don't want to wake up to another vulnerability because I forgot to run apt upgrade.
  • Rate limiting in Nginx — added limit_req_zone to throttle requests. 10 requests per second per IP, with a burst of 20. Enough for normal browsing, stops basic flooding.

Part 9: The Laptop Problem (Again)

Of course, since this is still laptop hardware, I had to deal with the classics:

  • Lid close: Set HandleLidSwitch=ignore and all its variants in /etc/systemd/logind.conf. Also masked sleep.target, suspend.target, hibernate.target, and hybrid-sleep.target.
  • Network on lid close: The Ethernet interface was dying when the lid closed. Fixed it with ethtool -s [ETH_INTERFACE] wol g and a systemd service to persist it across reboots.
  • Dual network: Both Ethernet and Wi-Fi are configured with different metrics (Ethernet preferred at metric 10, Wi-Fi as fallback at metric 100). The Ethernet port is a bit loose so having Wi-Fi as backup gives me peace of mind.
  • C-State freeze: Applied the same GRUB fix from Chapter 2 — processor.max_cstate=1 intel_idle.max_cstate=1. Not making that mistake twice.
  • Battery: Already removed it during the previous chapter. CriticalPowerAction=Ignore in UPower config as backup. That ghost battery is NOT haunting me again.

The Result

The new server feels completely different. It boots fast, uses minimal resources, and I actually understand every single thing that's installed on it. No mystery services, no leftover configs from months of patching.

Here's the before and after:

Old Server New Server
OS Ubuntu Server Debian 13 (minimal)
Web Server Apache Nginx
VPN OpenVPN (port 1194) OpenVPN (random high port)
SSH Exposed to internet Behind VPN only
Firewall Added after the hack First thing configured
Auto Updates Nope unattended-upgrades
DNS Cloudflare proxy on everything Proxy for web, DNS-only for VPN

What's Next

The to-do list from the last chapter is still alive. Now that the foundation is solid, I can actually start building on top of it:

  • Containerization with Docker — isolate services properly
  • UPS — because the power outage problem hasn't gone away, I just got better at recovering from it
  • Notification system — I want a Telegram bot or something that pings me when the server reboots or a service goes down
  • Self-hosted drive with NAS
  • Mail server (the ambitious one)
  • Local LLM... if this old laptop can handle it (spoiler: it probably can't, but it'll be fun to try)

For now though, the server is up, the site is live, the VPN works from anywhere, and I can finally close my phone without that little anxiety in my stomach. That's a win.