diff --git a/_posts/homelab/2026-01-01-homelab-part2.md b/_posts/homelab/2026-01-01-homelab-part2.md new file mode 100644 index 0000000..6aab0b0 --- /dev/null +++ b/_posts/homelab/2026-01-01-homelab-part2.md @@ -0,0 +1,343 @@ +--- +layout: post +title: 'ThinkCentre M720q + Debian: 24/7 Server Setup Step 2' +date: 2026-01-01 23:08:00 -0400 +categories: + - homelab +highlight: true +--- + +[[2025-12-31-homelab]] + +We need secure SSH access to our home servers (`Node 1`, `Node 2`, `Node 3`) from anywhere in the world. However, +our ISP does not provide a Static IP, and we want to avoid exposing ports on our home router or using third-party +DDNS services. So we implement a VPN, specifically a hub-and-spoke VPN using an EC2 server (with a static IPV4): + +- **The Hub (EC2):** A tiny Amazon Linux instance (`t4g.nano`) acts as a Relay and a static Public IP. +- **The Spokes (Nodes):** Home nodes initiate _outbound_ connections to the EC2 Hub. we bypass the need for home + port forwarding. +- **The Client (Personal Mac):** Connects to the EC2 Hub to reach the home nodes. +- **Security:** Zero trust. SSH ports on home nodes are closed to the local network and only listen on the VPN + interface. Access requires both the specific WireGuard Private Key and the SSH Key. + +## 2. Network Map + +| Device | Role | VPN IP (`wg0`) | Physical IP (LAN) | Public IP | +| ------------- | ------- | -------------- | ----------------- | ------------------- | +| **EC2 Proxy** | **HUB** | `10.100.0.1` | `10.0.x.x` | `3.99.x.x` (Static) | +| **MacBook** | Client | `10.100.0.2` | (Dynamic) | (Dynamic) | +| **Node 1** | Server | `10.100.0.10` | `192.168.2.250` | (Hidden) | +| **Node 2** | Server | `10.100.0.11` | `192.168.2.251` | (Hidden) | +| **Node 3** | Server | `10.100.0.12` | `192.168.2.252` | (Hidden) | + +--- + +## 3. Phase 1: The EC2 Relay (The Hub) + +_OS: Amazon Linux 2023_ + +### 3.1 Initial Setup + +1. Launch a `t4g.nano` instance. +2. **Security Group:** Allow UDP Port `51820` from `0.0.0.0/0`. +3. **Source/Dest Check:** Go to EC2 Console > Actions > Networking > **Change Source/Destination check** > **Stop** ( + Required for routing). + +### 3.2 Installation & Forwarding + +```bash +sudo dnf update -y +sudo dnf install wireguard-tools -y +sudo dnf install iptables -y + +# Enable IP Forwarding Permanently +echo "net.ipv4.ip_forward=1" | sudo tee /etc/sysctl.d/99-wireguard-forward.conf +sudo sysctl -p /etc/sysctl.d/99-wireguard-forward.conf + +# Generate Keys (Save these) +priv=$(wg genkey); pub=$(echo "$priv" | wg pubkey); echo "Private: $priv"; echo "Public: $pub" +``` + +### 3.3 Configure WireGuard (`/etc/wireguard/wg0.conf`) (EC2 proxy) + +_We will replace `` with actual values as we set up the nodes._ + +```ini +[Interface] +Address = 10.100.0.1/24 +ListenPort = 51820 +PrivateKey = + +# Allow traffic to flow between peers (Mac <-> Nodes) +PostUp = iptables -A FORWARD -i %i -j ACCEPT +PostDown = iptables -D FORWARD -i %i -j ACCEPT + +# Peer: MacBook +[Peer] +PublicKey = +AllowedIPs = 10.100.0.2/32 + +# Peer: Node 1 +[Peer] +PublicKey = +AllowedIPs = 10.100.0.10/32 + +# Peer: Node 2 +[Peer] +PublicKey = +AllowedIPs = 10.100.0.11/32 + +# Peer: Node 3 +[Peer] +PublicKey = +AllowedIPs = 10.100.0.12/32 +``` + +**Start Service:** `sudo wg-quick up wg0` + +--- + +## 4. Phase 2: Node Setup (The Spokes) + +_OS: Debian (Headless). Repeat for each node._ + +### 4.1 Root Access & Users + +```bash +# Switch to root +su - + +# Create generic user if needed (optional) or just rely on root +# Verify sudo is installed +apt update && apt install sudo -y +``` + +### 4.2 Temporary IP Setup + +_Avoids conflicts with Home Router DHCP immediately._ + +```bash +# Example for Node 2 (.251) - Adjust for .250 or .252 +ip addr flush dev enp0s31f6 +ip addr add 192.168.2.251/24 dev enp0s31f6 +ip link set enp0s31f6 up +ip route add default via 192.168.2.1 +``` + +### 4.3 SSH Hardening & Firewall (Temporary Access) + +_We allow local SSH temporarily to copy-paste keys._ + +```bash +apt install ufw openssh-server -y + +# 1. Firewall Basics +ufw default deny incoming +ufw default allow outgoing + +# 2. Allow Local SSH (Temporary Rule) +# Replace 192.168.2.147 with your Mac's current local IP +ufw allow from 192.168.2.147 to any port 22 proto tcp +ufw enable + +# 3. SSH Config +nano /etc/ssh/sshd_config +# Change: +# PermitRootLogin yes +# PasswordAuthentication no + +systemctl restart ssh +``` + +### 4.4 Keys & Permanent Access (On Mac) + +1. **Generate Key:** `ssh-keygen -t ed25519 -f ~/.ssh/home-server` +2. **Copy to Node:** `ssh-copy-id -i ~/.ssh/home-server root@192.168.2.251` +3. **Test:** `ssh root@192.168.2.251` + +### 4.5 Permanent Network Config + +_Make the Static IP survive a reboot._ + +`nano /etc/network/interfaces` + +```bash +# The Primary Network Interface +auto enp0s31f6 +iface enp0s31f6 inet static + address 192.168.2.251 # <--- change per node + netmask 255.255.255.0 + gateway 192.168.2.1 + dns-nameservers 1.1.1.1 8.8.8.8 +``` + +### 4.6 WireGuard Setup (nodes) + +1. **Install:** `apt install wireguard -y` +2. **Generate Keys:** + + ```bash + priv=$(wg genkey); pub=$(echo "$priv" | wg pubkey); echo "Private: $priv"; echo "Public: $pub" + ``` + +3. **Configure (`/etc/wireguard/wg0.conf`):** + + ```ini + [Interface] + Address = 10.100.0.11/24 # <--- change per node (.10, .11, .12) + PrivateKey = + + [Peer] + PublicKey = + Endpoint = :51820 + AllowedIPs = 10.100.0.0/24 + PersistentKeepalive = 25 + ``` + +4. **Enable:** `systemctl enable --now wg-quick@wg0` + +(to update: `sudo systemctl restart wg-quick@wg0`) + +--- + +## 5. Phase 3: Client Setup (MacBook) + +**App:** Official WireGuard Client +**Config:** + +```ini +[Interface] +PrivateKey = +Address = 10.100.0.2/24 +DNS = 1.1.1.1 + +[Peer] +PublicKey = +Endpoint = :51820 +# Route VPN traffic AND Home Network traffic through EC2 +AllowedIPs = 10.100.0.0/24, 192.168.2.0/24 +PersistentKeepalive = 25 +``` + +--- + +Once WireGuard works, and you can SSH using `ssh root@10.100.0.11` (or the local IP via the tunnel): + +1. **Delete the Temporary SSH Rule on Nodes:** + + ```bash + ufw status numbered + ufw delete + ``` + +2. **Add the VPN-only SSH Rule:** + + ```bash + # Only allow SSH if it comes from the VPN Tunnel (EC2/Mac) + ufw allow in on wg0 to any port 22 proto tcp + ``` + +Now: + +- Connecting via `192.168.2.251` from home Wi-Fi is **blocked**. +- Connecting via `10.100.0.11` (+VPN active) is **allowed**. + +**SSH Shortcuts (`~/.ssh/config` on Mac):** +This allows us to ssh to the servers without the need to mention their hostname or add (`-i`) the ssh key + +```ssh +Host ec2-proxy + HostName 10.100.0.1 + User ec2-user + IdentityFile ~/.ssh/ec2-proxy + +Host node1 + HostName 10.100.0.10 + User root + IdentityFile ~/.ssh/home-server + +Host node2 + HostName 10.100.0.11 + User root + IdentityFile ~/.ssh/home-server + +Host node3 + HostName 10.100.0.12 + User root + IdentityFile ~/.ssh/home-server +``` + +This is a solid, production-ready guide. The logic flows correctly from infrastructure (EC2) to nodes, then to clients, +and finally to hardening. + +I have **one critical correction** for your SSH config snippet before you publish: + +> **Correction in `~/.ssh/config**`: +You have a copy-paste error for `Host node3`. It is currently pointing to `.11`(Node 2's IP). It should be`.12`. + +Here is the **Troubleshooting** section you requested, written in the same Markdown format to append to the end of your +post. + +--- + +## 6. Troubleshooting + +### 1. Check the Handshakes + +On every machine (Mac, EC2, Nodes), run: + +```bash +sudo wg show +``` + +We want to see `latest handshake: X seconds ago`. If "Handshake: None": + +- Check that the **Public Key** in peer A's config matches the **Private Key** on peer B. +- Ensure AWS Security Group allows inbound **UDP 51820** from `0.0.0.0/0` and all traffic for outbound. +- Ensure the Nodes have the correct EC2 Public IP in their `Endpoint` field. + +### 2. "Destination Host Unreachable" + +If you can ping EC2 (`10.100.0.1`) but not the Nodes (`10.100.0.11`): + +- Go to the EC2 Console, select your instance, and ensure **Actions > Networking > Change Source/Destination Check** is + set to **STOP**. +- Run `sysctl net.ipv4.ip_forward` on EC2. It must be `1`. +- Ensure `iptables` is installed on Amazon Linux (`sudo dnf install iptables -y`) and that your `PostUp` rules in + `wg0.conf` are active. + +### 3. Connection Stalls (MTU Issues) + +If SSH connects but "hangs" or freezes, or if ping works but SSH doesn't, you likely have an MTU (Packet Size) issue. +This is common with home ISPs (PPPoE). + +- **Fix:** Lower the MTU on the Node interface. +- Edit `/etc/wireguard/wg0.conf` on the Node: + +```ini +[Interface] +... +MTU = 1280 +``` + +- Restart WireGuard: `sudo systemctl restart wg-quick@wg0` + +### 4. SSH "Permission Denied" (Public Key) + +If you see a handshake but SSH rejects you: + +- Run `ssh -v root@10.100.0.11` (with `-v`) to see exactly which key your Mac is offering. +- On the Node, run `tail -f /var/log/ufw.log`. If you see blocks on Port 22, your UFW rule `ufw allow in on wg0` might + be missing. + +### 5. "Remote Host Identification Has Changed" + +Since we moved from a direct connection to a VPN, your Mac might think the IP `10.100.0.x` is being spoofed. + +- **Fix:** Clear the old fingerprint from your Mac: + +```bash +ssh-keygen -R 10.100.0.1 +ssh-keygen -R 10.100.0.10 +ssh-keygen -R 10.100.0.11 +```