added homelab part 2

2026-01-01 23:34:45 -05:00
parent 1df3fb90f4
commit e7ea793b84
1 changed files with 343 additions and 0 deletions
--- a/_posts/homelab/2026-01-01-homelab-part2.md
+++ b/_posts/homelab/2026-01-01-homelab-part2.md
@@ -0,0 +1,343 @@
+---
+layout: post
+title: 'ThinkCentre M720q + Debian: 24/7 Server Setup Step 2'
+date: 2026-01-01 23:08:00 -0400
+categories:
+  - homelab
+highlight: true
+---
+
+[[2025-12-31-homelab]]
+
+We need secure SSH access to our home servers (`Node 1`, `Node 2`, `Node 3`) from anywhere in the world. However,
+our ISP does not provide a Static IP, and we want to avoid exposing ports on our home router or using third-party
+DDNS services. So we implement a VPN, specifically a hub-and-spoke VPN using an EC2 server (with a static IPV4):
+
+- **The Hub (EC2):** A tiny Amazon Linux instance (`t4g.nano`) acts as a Relay and a static Public IP.
+- **The Spokes (Nodes):** Home nodes initiate _outbound_ connections to the EC2 Hub. we bypass the need for home
+  port forwarding.
+- **The Client (Personal Mac):** Connects to the EC2 Hub to reach the home nodes.
+- **Security:** Zero trust. SSH ports on home nodes are closed to the local network and only listen on the VPN
+  interface. Access requires both the specific WireGuard Private Key and the SSH Key.
+
+## 2. Network Map
+
+| Device        | Role    | VPN IP (`wg0`) | Physical IP (LAN) | Public IP           |
+| ------------- | ------- | -------------- | ----------------- | ------------------- |
+| **EC2 Proxy** | **HUB** | `10.100.0.1`   | `10.0.x.x`        | `3.99.x.x` (Static) |
+| **MacBook**   | Client  | `10.100.0.2`   | (Dynamic)         | (Dynamic)           |
+| **Node 1**    | Server  | `10.100.0.10`  | `192.168.2.250`   | (Hidden)            |
+| **Node 2**    | Server  | `10.100.0.11`  | `192.168.2.251`   | (Hidden)            |
+| **Node 3**    | Server  | `10.100.0.12`  | `192.168.2.252`   | (Hidden)            |
+
+---
+
+## 3. Phase 1: The EC2 Relay (The Hub)
+
+_OS: Amazon Linux 2023_
+
+### 3.1 Initial Setup
+
+1. Launch a `t4g.nano` instance.
+2. **Security Group:** Allow UDP Port `51820` from `0.0.0.0/0`.
+3. **Source/Dest Check:** Go to EC2 Console > Actions > Networking > **Change Source/Destination check** > **Stop** (
+   Required for routing).
+
+### 3.2 Installation & Forwarding
+
+```bash
+sudo dnf update -y
+sudo dnf install wireguard-tools -y
+sudo dnf install iptables -y
+
+# Enable IP Forwarding Permanently
+echo "net.ipv4.ip_forward=1" | sudo tee /etc/sysctl.d/99-wireguard-forward.conf
+sudo sysctl -p /etc/sysctl.d/99-wireguard-forward.conf
+
+# Generate Keys (Save these)
+priv=$(wg genkey); pub=$(echo "$priv" | wg pubkey); echo "Private: $priv"; echo "Public: $pub"
+```
+
+### 3.3 Configure WireGuard (`/etc/wireguard/wg0.conf`) (EC2 proxy)
+
+_We will replace `<KEYS>` with actual values as we set up the nodes._
+
+```ini
+[Interface]
+Address = 10.100.0.1/24
+ListenPort = 51820
+PrivateKey = <EC2_PRIVATE_KEY>
+
+# Allow traffic to flow between peers (Mac <-> Nodes)
+PostUp = iptables -A FORWARD -i %i -j ACCEPT
+PostDown = iptables -D FORWARD -i %i -j ACCEPT
+
+# Peer: MacBook
+[Peer]
+PublicKey = <MAC_PUBLIC_KEY>
+AllowedIPs = 10.100.0.2/32
+
+# Peer: Node 1
+[Peer]
+PublicKey = <NODE_1_PUBLIC_KEY>
+AllowedIPs = 10.100.0.10/32
+
+# Peer: Node 2
+[Peer]
+PublicKey = <NODE_2_PUBLIC_KEY>
+AllowedIPs = 10.100.0.11/32
+
+# Peer: Node 3
+[Peer]
+PublicKey = <NODE_3_PUBLIC_KEY>
+AllowedIPs = 10.100.0.12/32
+```
+
+**Start Service:** `sudo wg-quick up wg0`
+
+---
+
+## 4. Phase 2: Node Setup (The Spokes)
+
+_OS: Debian (Headless). Repeat for each node._
+
+### 4.1 Root Access & Users
+
+```bash
+# Switch to root
+su -
+
+# Create generic user if needed (optional) or just rely on root
+# Verify sudo is installed
+apt update && apt install sudo -y
+```
+
+### 4.2 Temporary IP Setup
+
+_Avoids conflicts with Home Router DHCP immediately._
+
+```bash
+# Example for Node 2 (.251) - Adjust for .250 or .252
+ip addr flush dev enp0s31f6
+ip addr add 192.168.2.251/24 dev enp0s31f6
+ip link set enp0s31f6 up
+ip route add default via 192.168.2.1
+```
+
+### 4.3 SSH Hardening & Firewall (Temporary Access)
+
+_We allow local SSH temporarily to copy-paste keys._
+
+```bash
+apt install ufw openssh-server -y
+
+# 1. Firewall Basics
+ufw default deny incoming
+ufw default allow outgoing
+
+# 2. Allow Local SSH (Temporary Rule)
+# Replace 192.168.2.147 with your Mac's current local IP
+ufw allow from 192.168.2.147 to any port 22 proto tcp
+ufw enable
+
+# 3. SSH Config
+nano /etc/ssh/sshd_config
+# Change:
+# PermitRootLogin yes
+# PasswordAuthentication no
+
+systemctl restart ssh
+```
+
+### 4.4 Keys & Permanent Access (On Mac)
+
+1. **Generate Key:** `ssh-keygen -t ed25519 -f ~/.ssh/home-server`
+2. **Copy to Node:** `ssh-copy-id -i ~/.ssh/home-server root@192.168.2.251`
+3. **Test:** `ssh root@192.168.2.251`
+
+### 4.5 Permanent Network Config
+
+_Make the Static IP survive a reboot._
+
+`nano /etc/network/interfaces`
+
+```bash
+# The Primary Network Interface
+auto enp0s31f6
+iface enp0s31f6 inet static
+    address 192.168.2.251  # <--- change per node
+    netmask 255.255.255.0
+    gateway 192.168.2.1
+    dns-nameservers 1.1.1.1 8.8.8.8
+```
+
+### 4.6 WireGuard Setup (nodes)
+
+1. **Install:** `apt install wireguard -y`
+2. **Generate Keys:**
+
+   ```bash
+   priv=$(wg genkey); pub=$(echo "$priv" | wg pubkey); echo "Private: $priv"; echo "Public: $pub"
+   ```
+
+3. **Configure (`/etc/wireguard/wg0.conf`):**
+
+   ```ini
+   [Interface]
+   Address = 10.100.0.11/24  # <--- change per node (.10, .11, .12)
+   PrivateKey = <NODE_PRIVATE_KEY>
+
+   [Peer]
+   PublicKey = <EC2_PUBLIC_KEY>
+   Endpoint = <EC2_PUBLIC_IP>:51820
+   AllowedIPs = 10.100.0.0/24
+   PersistentKeepalive = 25
+   ```
+
+4. **Enable:** `systemctl enable --now wg-quick@wg0`
+
+(to update: `sudo systemctl restart wg-quick@wg0`)
+
+---
+
+## 5. Phase 3: Client Setup (MacBook)
+
+**App:** Official WireGuard Client
+**Config:**
+
+```ini
+[Interface]
+PrivateKey = <MAC_PRIVATE_KEY>
+Address = 10.100.0.2/24
+DNS = 1.1.1.1
+
+[Peer]
+PublicKey = <EC2_PUBLIC_KEY>
+Endpoint = <EC2_PUBLIC_IP>:51820
+# Route VPN traffic AND Home Network traffic through EC2
+AllowedIPs = 10.100.0.0/24, 192.168.2.0/24
+PersistentKeepalive = 25
+```
+
+---
+
+Once WireGuard works, and you can SSH using `ssh root@10.100.0.11` (or the local IP via the tunnel):
+
+1. **Delete the Temporary SSH Rule on Nodes:**
+
+   ```bash
+   ufw status numbered
+   ufw delete <number_of_local_ssh_rule>
+   ```
+
+2. **Add the VPN-only SSH Rule:**
+
+   ```bash
+   # Only allow SSH if it comes from the VPN Tunnel (EC2/Mac)
+   ufw allow in on wg0 to any port 22 proto tcp
+   ```
+
+Now:
+
+- Connecting via `192.168.2.251` from home Wi-Fi is **blocked**.
+- Connecting via `10.100.0.11` (+VPN active) is **allowed**.
+
+**SSH Shortcuts (`~/.ssh/config` on Mac):**
+This allows us to ssh to the servers without the need to mention their hostname or add (`-i`) the ssh key
+
+```ssh
+Host ec2-proxy
+    HostName 10.100.0.1
+    User ec2-user
+    IdentityFile ~/.ssh/ec2-proxy
+
+Host node1
+    HostName 10.100.0.10
+    User root
+    IdentityFile ~/.ssh/home-server
+
+Host node2
+    HostName 10.100.0.11
+    User root
+    IdentityFile ~/.ssh/home-server
+
+Host node3
+    HostName 10.100.0.12
+    User root
+    IdentityFile ~/.ssh/home-server
+```
+
+This is a solid, production-ready guide. The logic flows correctly from infrastructure (EC2) to nodes, then to clients,
+and finally to hardening.
+
+I have **one critical correction** for your SSH config snippet before you publish:
+
+> **Correction in `~/.ssh/config**`:
+You have a copy-paste error for `Host node3`. It is currently pointing to `.11`(Node 2's IP). It should be`.12`.
+
+Here is the **Troubleshooting** section you requested, written in the same Markdown format to append to the end of your
+post.
+
+---
+
+## 6. Troubleshooting
+
+### 1. Check the Handshakes
+
+On every machine (Mac, EC2, Nodes), run:
+
+```bash
+sudo wg show
+```
+
+We want to see `latest handshake: X seconds ago`. If "Handshake: None":
+
+- Check that the **Public Key** in peer A's config matches the **Private Key** on peer B.
+- Ensure AWS Security Group allows inbound **UDP 51820** from `0.0.0.0/0` and all traffic for outbound.
+- Ensure the Nodes have the correct EC2 Public IP in their `Endpoint` field.
+
+### 2. "Destination Host Unreachable"
+
+If you can ping EC2 (`10.100.0.1`) but not the Nodes (`10.100.0.11`):
+
+- Go to the EC2 Console, select your instance, and ensure **Actions > Networking > Change Source/Destination Check** is
+  set to **STOP**.
+- Run `sysctl net.ipv4.ip_forward` on EC2. It must be `1`.
+- Ensure `iptables` is installed on Amazon Linux (`sudo dnf install iptables -y`) and that your `PostUp` rules in
+  `wg0.conf` are active.
+
+### 3. Connection Stalls (MTU Issues)
+
+If SSH connects but "hangs" or freezes, or if ping works but SSH doesn't, you likely have an MTU (Packet Size) issue.
+This is common with home ISPs (PPPoE).
+
+- **Fix:** Lower the MTU on the Node interface.
+- Edit `/etc/wireguard/wg0.conf` on the Node:
+
+```ini
+[Interface]
+...
+MTU = 1280
+```
+
+- Restart WireGuard: `sudo systemctl restart wg-quick@wg0`
+
+### 4. SSH "Permission Denied" (Public Key)
+
+If you see a handshake but SSH rejects you:
+
+- Run `ssh -v root@10.100.0.11` (with `-v`) to see exactly which key your Mac is offering.
+- On the Node, run `tail -f /var/log/ufw.log`. If you see blocks on Port 22, your UFW rule `ufw allow in on wg0` might
+  be missing.
+
+### 5. "Remote Host Identification Has Changed"
+
+Since we moved from a direct connection to a VPN, your Mac might think the IP `10.100.0.x` is being spoofed.
+
+- **Fix:** Clear the old fingerprint from your Mac:
+
+```bash
+ssh-keygen -R 10.100.0.1
+ssh-keygen -R 10.100.0.10
+ssh-keygen -R 10.100.0.11
+```