From 74a8527e2ccfa7a4da87aec81a8cf2310ca8290f Mon Sep 17 00:00:00 2001 From: wboughattas Date: Fri, 2 Jan 2026 15:55:39 -0500 Subject: [PATCH] refactor + added more notes --- _posts/homelab/2025-12-31-homelab-part1.md | 68 ++++---- _posts/homelab/2026-01-01-homelab-part2.md | 183 +++++++++++++++------ 2 files changed, 173 insertions(+), 78 deletions(-) diff --git a/_posts/homelab/2025-12-31-homelab-part1.md b/_posts/homelab/2025-12-31-homelab-part1.md index c73685c..d9341c5 100644 --- a/_posts/homelab/2025-12-31-homelab-part1.md +++ b/_posts/homelab/2025-12-31-homelab-part1.md @@ -31,53 +31,52 @@ vPro and ThinkCentre BIOS, which is essential for the advanced BIOS settings tha For this setup, we will not take advantage of vPro's features such as accessing the BIOS over the network, but we will take advantage of features that come with ThinkCentre BIOS such as the auto-power on after a power loss. -# ThinkCentre M720q: Node Configuration +I already created bootable USB drive for debian 13 ISO using rufus. -## 1. Initial Verification +# ThinkCentre M720q: Node Configuration - **Enter BIOS:** Press repeatedly the **F1** key. - **Hardware Audit:** Confirm the CPU, RAM, and Storage match what is expected. Reset BIOS to default settings if it's the first time. -- **Set the Standard:** Set the system clock to **UTC** if it's not the case. This is necessary for Kubernetes because - it ensures that logs from different nodes align perfectly regardless of local time zones. - -## 2. Hardware Decoupling We want to strip away desktop features to save power and reduce the potential attack surface. -| Menu Path | Setting | Action | Why | -| :-------------------------- | :------------------ | :-------------------------- | :--------------------------------------------------------- | -| **Devices > Audio Setup** | Integrated Audio | **Disabled** | Servers don't need sound | -| **Devices > Network Setup** | Wi-Fi / BT / PXE | **Disabled** | Forces the node to rely on the Onboard Ethernet. | -| **Devices > USB Setup** | USB Legacy Support | **Disabled** | Prevents the use of less secure USB protocols during boot. | -| **Advanced > CPU Setup** | Virtualization | **Enabled** | Needed for container runtimes. | -| **Advanced > CPU Setup** | VT-d | **Enabled** | IO magic. | -| **Power** | After Power Loss | **Power On** | The Auto-Restart rule. | -| **Power** | Intelligent Cooling | **Performance or Acoustic** | Either to prevent thermal throttling or lower noise. | +| Menu Path | Setting | Action | Why | +| :-------------------------- | :--------------------------------- | :------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------- | +| **Devices > Audio Setup** | Integrated Audio | **Disabled** | Servers don't need sound | +| **Devices > Network Setup** | Wi-Fi / PXE | **Disabled** | Forces the node to rely on the Onboard Ethernet. | +| **Devices > USB Setup** | USB Legacy Support | **Disabled** | Prevents the use of less secure USB protocols during boot. | +| **Advanced > CPU Setup** | Virtualization | **Enabled** | Needed for container runtimes. | +| **Advanced > CPU Setup** | VT-d | **Enabled** | IO magic. | +| **Power** | After Power Loss | **Power On** | The Auto-Restart rule. | +| **Power** | Intelligent Cooling | **Performance or Acoustic** | Either to prevent thermal throttling or lower noise. | +| **Security** | Administrator Password | **Set** | Prevents tampering with the BIOS settings. | +| **Security** | Windows UEFI Update | **Disabled** | We are replacing Windows with Debian. | +| **Security** | Password for F1/F12 | **Yes** | Requires your admin password to boot from an unauthorized USB. | +| **Security** | Require POP on System Boot/Restart | **No** | Overkill. | +| **Security** | POP Changeable by User | **No** | Requires your admin to change his password. | +| **Security** | Secure Boot | **Enabled** | Verifies the Debian kernel signature before allowing it to boot. | +| **Startup** | Boot Sequence | **Move the drive(s) to the #1 spot (prioritize the one storing the OS Bootloader). Exclude everything else.** | | -## 3. Security Governance +Debian installation is straightforward. Set most settings to default. -| Menu Path | Setting | Action | Why | -| :----------- | :--------------------- | :----------- | :--------------------------------------------------------------- | -| **Security** | Administrator Password | **Set** | Prevents tampering with the BIOS settings. | -| **Security** | Windows UEFI Update | **Disabled** | We are replacing Windows with Debian. | -| **Security** | Password for F1/F12 | **Yes** | Requires your admin password to boot from an unauthorized USB. | -| **Security** | POP Changeable by User | **No** | Requires your admin to change his password. | -| **Security** | Secure Boot | **Enabled** | Verifies the Debian kernel signature before allowing it to boot. | - -## 4. Boot Sequence - -**Startup > Boot Sequence:** Move the drive(s) to the #1 spot (prioritize the one storing the OS Bootloader). Exclude -everything else. -**Startup > CSM:** must be disabled to restrict non-UEFI operating systems. - -## 5. Post-Install +## Post-Install Once Debian is running, we need to tell the OS to take ownership of the hardware management. In a better world, we prefer to take full ownership ourselves using CoreBoot. But we cannot because Intel Boot Guard is supported on the hardware level. Since Lenovo ships these with Intel Boot Guard (Verified Boot) enabled, we must work within the manufacturer's ecosystem. +### Disable the GUI on boot + +```bash +# Switch to root +su - + +sudo systemctl set-default multi-user.target +reboot +``` + ### Synchronize the Hardware Clock This command tells Debian to treat the motherboard's clock as UTC (the server standard). @@ -86,6 +85,13 @@ This command tells Debian to treat the motherboard's clock as UTC (the server st sudo timedatectl set-local-rtc 0 ``` +Add missing locales. In my case, `en_CA.UTF-8`. + +```bash +sudo dpkg-reconfigure locales +# Select en_CA.UTF-8 (or your preference) -> OK +``` + ### Enable Linux-Native Firmware Updates We disabled the Windows update hook in the BIOS; now we replace it with the **Linux Vendor Firmware Service (LVFS)**. diff --git a/_posts/homelab/2026-01-01-homelab-part2.md b/_posts/homelab/2026-01-01-homelab-part2.md index aa7e6f3..f7cefdf 100644 --- a/_posts/homelab/2026-01-01-homelab-part2.md +++ b/_posts/homelab/2026-01-01-homelab-part2.md @@ -14,11 +14,11 @@ our ISP does not provide a Static IP, and we want to avoid exposing ports on our DDNS services. So we implement a VPN, specifically a hub-and-spoke VPN using an EC2 server (with a static IPV4): - **The Hub (EC2):** A tiny Amazon Linux instance (`t4g.nano`) acts as a Relay and a static Public IP. -- **The Spokes (Nodes):** Home nodes initiate _outbound_ connections to the EC2 Hub. we bypass the need for home +- **The Spokes (Nodes):** Home nodes initiate outbound connections to the EC2 Hub. we bypass the need for home port forwarding. - **The Client (Personal Mac):** Connects to the EC2 Hub to reach the home nodes. -- **Security:** Zero trust. SSH ports on home nodes are closed to the local network and only listen on the VPN - interface. Access requires both the specific WireGuard Private Key and the SSH Key. +- **Security:** SSH ports on home nodes are closed to the local network and only listen on the VPN interface. Access + requires both the specific WireGuard Private Key and the SSH Key. ## 2. Network Map @@ -30,16 +30,14 @@ DDNS services. So we implement a VPN, specifically a hub-and-spoke VPN using an | **Node 2** | Server | `10.100.0.11` | `192.168.2.251` | (Hidden) | | **Node 3** | Server | `10.100.0.12` | `192.168.2.252` | (Hidden) | ---- +## 3. Phase 1: EC2 Setup (The Hub) -## 3. Phase 1: The EC2 Relay (The Hub) - -_OS: Amazon Linux 2023_ +OS: Amazon Linux 2023 ### 3.1 Initial Setup 1. Launch a `t4g.nano` instance. -2. **Security Group:** Allow UDP Port `51820` from `0.0.0.0/0`. +2. **Security Group:** Allow inbound UDP Port `51820` from `0.0.0.0/0` . All traffic for outbound. 3. **Source/Dest Check:** Go to EC2 Console > Actions > Networking > **Change Source/Destination check** > **Stop** ( Required for routing). @@ -58,9 +56,13 @@ sudo sysctl -p /etc/sysctl.d/99-wireguard-forward.conf priv=$(wg genkey); pub=$(echo "$priv" | wg pubkey); echo "Private: $priv"; echo "Public: $pub" ``` -### 3.3 Configure WireGuard (`/etc/wireguard/wg0.conf`) (EC2 proxy) +### 3.3 Configure WireGuard (EC2 proxy) -_We will replace `` with actual values as we set up the nodes._ +```bash +sudo nano /etc/wireguard/wg0.conf +``` + +We will replace `` with actual values as we set up the nodes. ```ini [Interface] @@ -93,13 +95,13 @@ PublicKey = AllowedIPs = 10.100.0.12/32 ``` -**Start Service:** `sudo wg-quick up wg0` +Start Service: `sudo wg-quick up wg0` --- ## 4. Phase 2: Node Setup (The Spokes) -_OS: Debian (Headless). Repeat for each node._ +OS: Debian (Headless). Repeat for each node. ### 4.1 Root Access & Users @@ -107,29 +109,42 @@ _OS: Debian (Headless). Repeat for each node._ # Switch to root su - -# Create generic user if needed (optional) or just rely on root -# Verify sudo is installed -apt update && apt install sudo -y +apt update ``` ### 4.2 Temporary IP Setup -_Avoids conflicts with Home Router DHCP immediately._ +Even though we will set a static IP on the Debian node itself, your router might still try to claim that IP for another +device via DHCP. Ensure you go into your router settings and reserve those IPs by marking them as "Static" in the +device list or by shrinking the DHCP IPs range to .. Reboot the nodes to apply the changes. + +```bash +ip link show +``` + +Remember the interface name, in my case `enp0s31f6`. ```bash # Example for Node 2 (.251) - Adjust for .250 or .252 -ip addr flush dev enp0s31f6 -ip addr add 192.168.2.251/24 dev enp0s31f6 -ip link set enp0s31f6 up -ip route add default via 192.168.2.1 +sudo ip addr flush dev enp0s31f6 +sudo ip link set enp0s31f6 up +sudo ip addr add 192.168.2.251/24 dev enp0s31f6 +sudo ip route add default via 192.168.2.1 ``` -### 4.3 SSH Hardening & Firewall (Temporary Access) +```bash +ping -c 3 192.168.2.1 +ip addr show +``` -_We allow local SSH temporarily to copy-paste keys._ +### 4.3 SSH Setup & Firewall (Temporary Access) + +We allow temporary local SSH temporarily to copy-paste keys. ```bash -apt install ufw openssh-server -y +apt install ufw openssh-server resolvconf -y +# ssh server auto started +# we will update ssh config in the next step # 1. Firewall Basics ufw default deny incoming @@ -139,36 +154,76 @@ ufw default allow outgoing # Replace 192.168.2.147 with your Mac's current local IP ufw allow from 192.168.2.147 to any port 22 proto tcp ufw enable - -# 3. SSH Config -nano /etc/ssh/sshd_config -# Change: -# PermitRootLogin yes -# PasswordAuthentication no - -systemctl restart ssh ``` ### 4.4 Keys & Permanent Access (On Mac) -1. **Generate Key:** `ssh-keygen -t ed25519 -f ~/.ssh/home-server` -2. **Copy to Node:** `ssh-copy-id -i ~/.ssh/home-server root@192.168.2.251` -3. **Test:** `ssh root@192.168.2.251` +1. Generate Key: `ssh-keygen -t ed25519 -f ~/.ssh/home-server` +2. Copy to Node: `ssh-copy-id -i ~/.ssh/home-server root@192.168.2.251` +3. Update SSH rules: + + ```bash + # 3. SSH Config + nano /etc/ssh/sshd_config + # Change: + # PermitRootLogin yes + # PasswordAuthentication no + # Find the line AcceptEnv LANG LC_* and comment it out + + systemctl restart ssh + ``` + +4. Test: `ssh root@192.168.2.251` which should not work without ssh key and + `ssh -i ~/.ssh/home-server root@192.168.2.251` which should work. ### 4.5 Permanent Network Config -_Make the Static IP survive a reboot._ - -`nano /etc/network/interfaces` +Make the Static IP survive a reboot. Before that, we must disable `NetworkManager` to prevent it from fighting with our +manual configuration, and enable `resolvconf` to handle DNS. ```bash +# 1. Stop the automated manager +sudo systemctl stop NetworkManager +sudo systemctl disable NetworkManager + +# 2. Enable the DNS helper +sudo systemctl enable --now resolvconf +``` + +Now, let's edit the interface file to make the IP survive a reboot, + +```bash +sudo nano /etc/network/interfaces +``` + +Append at the end, + +```text # The Primary Network Interface auto enp0s31f6 iface enp0s31f6 inet static - address 192.168.2.251 # <--- change per node + address 192.168.2.251 netmask 255.255.255.0 gateway 192.168.2.1 - dns-nameservers 1.1.1.1 8.8.8.8 + dns-nameservers 1.1.1.1 8.8.8.8 192.168.2.1 +``` + +Test the syntax before rebooting, + +```bash +sudo ifup -v --no-act enp0s31f6 +``` + +and reboot to apply the changes, + +```bash +reboot +``` + +On mac, + +```bash +ping 192.168.2.251 ``` ### 4.6 WireGuard Setup (nodes) @@ -180,7 +235,7 @@ iface enp0s31f6 inet static priv=$(wg genkey); pub=$(echo "$priv" | wg pubkey); echo "Private: $priv"; echo "Public: $pub" ``` -3. **Configure (`/etc/wireguard/wg0.conf`):** +3. **Configure (`sudo nano /etc/wireguard/wg0.conf`):** ```ini [Interface] @@ -299,12 +354,45 @@ Host node3 ## 6. Troubleshooting -### 1. Check the Handshakes +### Check the Handshakes -On every machine (Mac, EC2, Nodes), run: +On every node, run: ```bash sudo wg show +# must enabled and can see latest handshake and transfer + +sudo systemctl status networking +# must be enabled + +sudo systemctl status NetworkManager +# must be disabled + +sudo systemctl status ssh +# must be enabled + +sudo systemctl status resolvconf +# must be enabled + +cat /etc/resolv.conf +# nameserver 1.1.1.1 +# nameserver 8.8.8.8 +# nameserver 192.168.2.1 + +ufw numbered +# Status: active +# +# To Action From +# -- ------ ---- +# 22/tcp on wg0 ALLOW Anywhere +# 22/tcp (v6) on wg0 ALLOW Anywhere (v6) +``` + +On mac and EC2, run + +```bash +sudo wg show +# must enabled and can see latest handshake and transfer ``` We want to see `latest handshake: X seconds ago`. If "Handshake: None": @@ -313,7 +401,7 @@ We want to see `latest handshake: X seconds ago`. If "Handshake: None": - Ensure AWS Security Group allows inbound **UDP 51820** from `0.0.0.0/0` and all traffic for outbound. - Ensure the Nodes have the correct EC2 Public IP in their `Endpoint` field. -### 2. "Destination Host Unreachable" +### Destination Host Unreachable If you can ping EC2 (`10.100.0.1`) but not the Nodes (`10.100.0.11`): @@ -323,13 +411,13 @@ If you can ping EC2 (`10.100.0.1`) but not the Nodes (`10.100.0.11`): - Ensure `iptables` is installed on Amazon Linux (`sudo dnf install iptables -y`) and that your `PostUp` rules in `wg0.conf` are active. -### 3. Connection Stalls (MTU Issues) +### Connection Stalls (MTU Issues) If SSH connects but "hangs" or freezes, or if ping works but SSH doesn't, you likely have an MTU (Packet Size) issue. This is common with home ISPs (PPPoE). - **Fix:** Lower the MTU on the Node interface. -- Edit `/etc/wireguard/wg0.conf` on the Node: +- Edit `sudo nano /etc/wireguard/wg0.conf` on the Node: ```ini [Interface] @@ -339,7 +427,7 @@ MTU = 1280 - Restart WireGuard: `sudo systemctl restart wg-quick@wg0` -### 4. SSH "Permission Denied" (Public Key) +### SSH "Permission Denied" (Public Key) If you see a handshake but SSH rejects you: @@ -347,7 +435,7 @@ If you see a handshake but SSH rejects you: - On the Node, run `tail -f /var/log/ufw.log`. If you see blocks on Port 22, your UFW rule `ufw allow in on wg0` might be missing. -### 5. "Remote Host Identification Has Changed" +### "Remote Host Identification Has Changed" Since we moved from a direct connection to a VPN, your Mac might think the IP `10.100.0.x` is being spoofed. @@ -357,4 +445,5 @@ Since we moved from a direct connection to a VPN, your Mac might think the IP `1 ssh-keygen -R 10.100.0.1 ssh-keygen -R 10.100.0.10 ssh-keygen -R 10.100.0.11 +ssh-keygen -R 10.100.0.12 ```