Severed/Severed-Blog
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c0c062df8cfbe0a641cea7f1c17c4b227fd051e1
Severed-Blog/_posts/homelab/2026-01-01-homelab-part2.md
wboughattas c0c062df8c added part 3
2026-01-03 23:05:40 -05:00

12 KiB
Raw Blame History

layout, title, date, categories, highlight
layout title date categories highlight
post ThinkCentre Kubernetes Home Server Step 2 (Debian, SSH, Firewall, hub-and-spoke VPN, AWS) 2026-01-01 23:08:00 -0400
homelab
true

2025-12-31-homelab-part1

We need secure SSH access to our home servers (Node 1, Node 2, Node 3) from anywhere in the world. However, our ISP does not provide a Static IP, and we want to avoid exposing ports on our home router or using third-party DDNS services. So we implement a VPN, specifically a hub-and-spoke VPN using an EC2 server (with a static IPV4):

  • The Hub (EC2): A tiny Amazon Linux instance (t4g.nano) acts as a Relay and a static Public IP.
  • The Spokes (Nodes): Home nodes initiate outbound connections to the EC2 Hub. we bypass the need for home port forwarding.
  • The Client (Personal Mac): Connects to the EC2 Hub to reach the home nodes.
  • Security: SSH ports on home nodes are closed to the local network and only listen on the VPN interface. Access requires both the specific WireGuard Private Key and the SSH Key.

2. Network Map

Device / Resource Type Role VPN IP (wg0) Physical IP (LAN) Public IP
EC2 Proxy To be created VPN Hub 10.100.0.1 10.0.x.x 3.99.x.x (Static)
MacBook Existing Client 10.100.0.2 (Dynamic) (Hidden)
Node 1 Existing K3s Master 10.100.0.10 192.168.2.250 (Hidden)
Node 2 Existing K3s Worker 10.100.0.11 192.168.2.251 (Hidden)
Node 3 Existing K3s Worker 10.100.0.12 192.168.2.252 (Hidden)
NAS-Server Existing Backup Storage N/A 192.168.2.135 (Hidden)
Home Router Existing Gateway / DHCP N/A 192.168.2.1 (Dynamic ISP)
Kube-VIP To be created App LoadBalancer N/A 192.168.2.240 (Hidden)
AdGuard Home To be created DNS LoadBalancer N/A 192.168.2.241 (Hidden)

3. Phase 1: EC2 Setup (The Hub)

OS: Amazon Linux 2023

3.1 Initial Setup

  1. Launch a t4g.nano instance.
  2. Security Group: Allow inbound UDP Port 51820 from 0.0.0.0/0 . All traffic for outbound.
  3. Source/Dest Check: Go to EC2 Console > Actions > Networking > Change Source/Destination check > Stop ( Required for routing).

3.2 Installation & Forwarding

sudo dnf update -y
sudo dnf install wireguard-tools -y
sudo dnf install iptables -y

# Enable IP Forwarding Permanently
echo "net.ipv4.ip_forward=1" | sudo tee /etc/sysctl.d/99-wireguard-forward.conf
sudo sysctl -p /etc/sysctl.d/99-wireguard-forward.conf

# Generate Keys (Save these)
priv=$(wg genkey); pub=$(echo "$priv" | wg pubkey); echo "Private: $priv"; echo "Public: $pub"

3.3 Configure WireGuard (EC2 proxy)

sudo nano /etc/wireguard/wg0.conf

We will replace <KEYS> with actual values as we set up the nodes.

[Interface]
Address = 10.100.0.1/24
ListenPort = 51820
PrivateKey = <EC2_PRIVATE_KEY>

# Allow traffic to flow between peers (Mac <-> Nodes)
PostUp = iptables -A FORWARD -i %i -j ACCEPT
PostDown = iptables -D FORWARD -i %i -j ACCEPT

# Peer: MacBook
[Peer]
PublicKey = <MAC_PUBLIC_KEY>
AllowedIPs = 10.100.0.2/32

# Peer: Node 1 (192.168.2.240/32 is the the floating IP of the kubernetes cluster, we will need it for step 3)
[Peer]
PublicKey = <NODE_1_PUBLIC_KEY>
AllowedIPs = 10.100.0.10/32, 192.168.2.240/32

# Peer: Node 2
[Peer]
PublicKey = <NODE_2_PUBLIC_KEY>
AllowedIPs = 10.100.0.11/32

# Peer: Node 3
[Peer]
PublicKey = <NODE_3_PUBLIC_KEY>
AllowedIPs = 10.100.0.12/32

Start Service: sudo wg-quick up wg0 Update Service sudo wg-quick down wg0 && sudo wg-quick up wg0

4. Phase 2: Node Setup (The Spokes)

OS: Debian (Headless). Repeat for each node.

4.1 Root Access & Users

# Switch to root
su -

apt update

4.2 Temporary IP Setup

Even though we will set a static IP on the Debian node itself, your router might still try to claim that IP for another device via DHCP. Ensure you go into your router settings and reserve those IPs by marking them as "Static" in the device list AND by shrinking the DHCP IPs range to 192.168.2.2 -> 192.168.2.239. Reboot the nodes to apply the changes.

ip link show

Remember the interface name, in my case enp0s31f6.

# Example for Node 2 (.251) - Adjust for .250 or .252
sudo ip addr flush dev enp0s31f6
sudo ip link set enp0s31f6 up
sudo ip addr add 192.168.2.251/24 dev enp0s31f6
sudo ip route add default via 192.168.2.1

ping -c 3 192.168.2.1
ip addr show

4.3 SSH Setup & Firewall (Temporary Access)

We allow temporary local SSH temporarily to copy-paste keys.

apt install ufw openssh-server resolvconf -y
# ssh server auto started
# we will update ssh config in the next step

# 1. Firewall Basics
ufw default deny incoming
ufw default allow outgoing

# 2. Allow Local SSH (Temporary Rule)
# Replace 192.168.2.147 with your Mac's current local IP
ufw allow from 192.168.2.147 to any port 22 proto tcp
ufw enable

4.4 Keys & Permanent Access (On Mac)

  1. Generate Key: ssh-keygen -t ed25519 -f ~/.ssh/home-server

  2. Copy to Node: ssh-copy-id -i ~/.ssh/home-server root@192.168.2.251

  3. Update SSH rules:

    # 3. SSH Config
nano /etc/ssh/sshd_config
# Change:
# PermitRootLogin yes
# PasswordAuthentication no
# Find the line AcceptEnv LANG LC_* and comment it out

systemctl restart ssh

  4. Test: ssh root@192.168.2.251 which should not work without ssh key and ssh -i ~/.ssh/home-server root@192.168.2.251 which should work.

4.5 Permanent Network Config

Make the Static IP survive a reboot. Before that, we must disable NetworkManager to prevent it from fighting with our manual configuration, and enable resolvconf to handle DNS.

# 1. Stop the automated manager
sudo systemctl stop NetworkManager
sudo systemctl disable NetworkManager

# 2. Enable the DNS helper
sudo systemctl enable --now resolvconf

Now, let's edit the interface file to make the IP survive a reboot,

sudo nano /etc/network/interfaces

Append at the end,

# The Primary Network Interface
auto enp0s31f6
iface enp0s31f6 inet static
    address 192.168.2.251
    netmask 255.255.255.0
    gateway 192.168.2.1
    dns-nameservers 1.1.1.1 8.8.8.8 192.168.2.1

Test the syntax before rebooting,

sudo ifup -v --no-act enp0s31f6

and reboot to apply the changes,

reboot

On mac,

ping 192.168.2.251

4.6 WireGuard Setup (nodes)

  1. Install: apt install wireguard -y

  2. Generate Keys:

    priv=$(wg genkey); pub=$(echo "$priv" | wg pubkey); echo "Private: $priv"; echo "Public: $pub"

  3. Configure (sudo nano /etc/wireguard/wg0.conf):

    [Interface]
Address = 10.100.0.11/24  # <--- change per node (.10, .11, .12)
PrivateKey = <NODE_PRIVATE_KEY>
MTU = 1280

[Peer]
PublicKey = <EC2_PUBLIC_KEY>
Endpoint = <EC2_PUBLIC_IP>:51820
AllowedIPs = 10.100.0.0/24
PersistentKeepalive = 25

  4. Enable: systemctl enable --now wg-quick@wg0

(to update: sudo systemctl restart wg-quick@wg0)

5. Phase 3: Client Setup (MacBook)

App: Official WireGuard Client Config:

[Interface]
PrivateKey = <MAC_PRIVATE_KEY>
Address = 10.100.0.2/24
DNS = 192.168.2.241, 192.168.2.1
MTU = 1280

[Peer]
PublicKey = <EC2_PUBLIC_KEY>
Endpoint = <EC2_PUBLIC_IP>:51820
# Route VPN traffic AND Home Network traffic through EC2
AllowedIPs = 10.100.0.0/24
PersistentKeepalive = 25

Once WireGuard works, and you can SSH using ssh root@10.100.0.11 (or the local IP via the tunnel):

  1. Add the VPN-only SSH Rule:

    # Only allow SSH if it comes from the VPN Tunnel (EC2/Mac)
ufw allow in on wg0 to any port 22 proto tcp

    SSHing with 10.100.0.11 should work now.

  2. Delete the Temporary SSH Rule on Nodes:

    ufw status numbered
ufw delete <number_of_local_ssh_rule>

    Connecting via 192.168.2.251 from home Wi-Fi is now blocked.

We are done. Final checks:

root@node-2:~# ufw numbered
Status: active

To                         Action      From
--                         ------      ----
22/tcp on wg0              ALLOW       Anywhere
22/tcp (v6) on wg0         ALLOW       Anywhere (v6)

root@node-2:~# sudo wg show
interface: wg0
  public key: ...
  private key: (hidden)
  listening port: 38547

peer: ...
  endpoint: 3.99.xx.xxx:51820
  allowed ips: 10.100.0.0/24
  latest handshake: Now
  transfer: 84.52 KiB received, 57.66 KiB sent
  persistent keepalive: every 25 seconds

root@node-2:~# ip addr show wg0
... 10.100.0.11/24 ...

root@node-2:~# sudo systemctl status networking
# must be enabled

root@node-2:~# sudo systemctl status NetworkManager
# must be disabled

root@node-2:~# sudo systemctl status ssh
# must be enabled

root@node-2:~# sudo systemctl status resolvconf
# must be enabled

root@node-2:~# cat /etc/resolv.conf
# nameserver 1.1.1.1
# nameserver 8.8.8.8
# nameserver 192.168.2.1

root@node-2:~# ufw numbered
# Status: active
#
# To                         Action      From
# --                         ------      ----
# 22/tcp on wg0              ALLOW       Anywhere
# 22/tcp (v6) on wg0         ALLOW       Anywhere (v6)

SSH Shortcuts (~/.ssh/config on Mac): This allows us to ssh to the servers without the need to mention their hostname or add (-i) the ssh key

Host ec2-proxy
    HostName 10.100.0.1
    User ec2-user
    IdentityFile ~/.ssh/ec2-proxy

Host home-node1
    HostName 10.100.0.10
    User root
    IdentityFile ~/.ssh/home-server

Host home-node2
    HostName 10.100.0.11
    User root
    IdentityFile ~/.ssh/home-server

Host home-node3
    HostName 10.100.0.12
    User root
    IdentityFile ~/.ssh/home-server

6. Troubleshooting

Check the Handshakes

On every node, run:

sudo wg show

We want to see latest handshake: X seconds ago. If "Handshake: None":

  • Check that the Public Key in peer A's config matches the Private Key on peer B.
  • Ensure AWS Security Group allows inbound UDP 51820 from 0.0.0.0/0 and all traffic for outbound.
  • Ensure the Nodes have the correct EC2 Public IP in their Endpoint field.

Destination Host Unreachable

If you can ping EC2 (10.100.0.1) but not the Nodes (10.100.0.11):

  • Go to the EC2 Console, select your instance, and ensure Actions > Networking > Change Source/Destination Check is set to STOP.
  • Run sysctl net.ipv4.ip_forward on EC2. It must be 1.
  • Ensure iptables is installed on Amazon Linux (sudo dnf install iptables -y) and that your PostUp rules in wg0.conf are active.

SSH "Permission Denied" (Public Key)

If you see a handshake but SSH rejects you:

  • Run ssh -v root@10.100.0.11 (with -v) to see exactly which key your Mac is offering.
  • On the Node, run tail -f /var/log/ufw.log. If you see blocks on Port 22, your UFW rule ufw allow in on wg0 might be missing.

"Remote Host Identification Has Changed"

Since we moved from a direct connection to a VPN, your Mac might think the IP 10.100.0.x is being spoofed.

  • Fix: Clear the old fingerprint from your Mac:
ssh-keygen -R 10.100.0.1
ssh-keygen -R 10.100.0.10
ssh-keygen -R 10.100.0.11
ssh-keygen -R 10.100.0.12

In next steps, we will automate the infrastructure using Ansible and deploy Kubernetes, with all services running in the cluster and HPA rules properly configured. The expected traffic flows will be as follows:

  • Admin Traffic: Admin → EC2 Bastion (WireGuard) → Home Cluster (complete)
  • Web Traffic: User → AWS CloudFront (WAF) → Cloudflare Tunnel → Home Cluster (next steps)

Our total cost is approximately 10$ (CAD) per month to cover the EC2 instance (on demand pricing ~ 4$/mo), elastic IP allocation (~5$/mo), and the domain (~1$/mo). Cloudflare will be used on the free tier.

2026-01-02-homelab-part3

Reference in New Issue View Git Blame Copy Permalink