added homelab article

_posts/homelab/2025-12-31-homelab.md

@@ -0,0 +1,112 @@
+---
+layout: post
+title: 'ThinkCentre M720q + Debian: 24/7 Server Setup'
+date: 2025-12-31 22:00:00 -0400
+categories:
+  - homelab
+  - debian
+highlight: true
+---
+
+> _I recently picked up 3 refurbished Lenovo ThinkCentre M720q Tiny to migrate my Kubernetes cluster from AWS and slash
+> my EKS costs. These machines are great for being silent and power-efficient, but out of the box, they're tuned as office
+> desktops, not as high-availability server racks. Here's how I configured the hardware and OS settings to make this
+> 3-node production-ready cluster._
+
+Not the cleanest setup, no cable management, servers upside down, stickers still on, no room for ventilation, and they have too much wiggle room in the rack, but the software is sound and that's 90% of the work done :D
+![server.png](assets/server.png)
+
+# The Hardware Stack
+
+Before diving into the configuration, here is the spec sheet for each node in the cluster. These units feature Intel
+vPro, which is essential for the advanced BIOS security we are about to toggle:
+
+- CPU: Intel(R) Core(TM) i7-7700 @ 3.60GHz (4 Cores / 8 Threads)
+- RAM: 16GB DDR4
+- Storage: 500GB SSD
+- Network: Dedicated 5-port 1Gbps Ethernet Switch
+
+# ThinkCentre M720q: Node Configuration
+
+## 1. Initial Verification
+
+- **Enter BIOS:** Press repeatedly the **F1** key.
+- **Hardware Audit:** Confirm the CPU, RAM, and Storage match what is expected. Reset BIOS to default settings if it's
+  the first time.
+- **Set the Standard:** Set the system clock to **UTC** if it's not the case. This is necessary for Kubernetes because
+  it ensures that logs from different nodes align perfectly regardless of local time zones.
+
+## 2. Hardware Decoupling
+
+We want to strip away desktop features to save power and reduce the potential attack surface.
+
+| Menu Path                   | Setting             | Action          | Why                                                        |
+| :-------------------------- | :------------------ | :-------------- | :--------------------------------------------------------- |
+| **Devices > Audio Setup**   | Integrated Audio    | **Disabled**    | Servers don't need sound                                   |
+| **Devices > Network Setup** | Wi-Fi / BT / PXE    | **Disabled**    | Forces the node to rely on the Onboard Ethernet.           |
+| **Devices > USB Setup**     | USB Legacy Support  | **Disabled**    | Prevents the use of less secure USB protocols during boot. |
+| **Power**                   | After Power Loss    | **Power On**    | The Auto-Restart rule.                                     |
+| **Power**                   | Intelligent Cooling | **Performance** | To prevent thermal throttling.                             |
+
+## 3. Security Governance
+
+| Menu Path    | Setting             | Action       | Why                                                              |
+| :----------- | :------------------ | :----------- | :--------------------------------------------------------------- |
+| **Security** | Supervisor Password | **Set**      | Prevents tampering with the BIOS settings.                       |
+| **Security** | Windows UEFI Update | **Disabled** | We are replacing Windows with Debian.                            |
+| **Security** | Secure Boot         | **Enabled**  | Verifies the Debian kernel signature before allowing it to boot. |
+| **Security** | Password for F12    | **Yes**      | Requires your admin password to boot from an unauthorized USB.   |
+
+## 4. Boot Sequence
+
+**Startup > Boot Sequence:** Move the drive(s) to the #1 spot (prioritize the one storing the OS Bootloader). Exclude
+everything else.
+
+## 5. Post-Install
+
+Once Debian is running, we need to tell the OS to take ownership of the hardware management. In a better world, we
+prefer to take full ownership ourselves using CoreBoot. But we cannot because Intel Boot Guard is supported on the
+hardware level. Since Lenovo ships these with Intel Boot Guard (Verified Boot) enabled, we must work within the
+manufacturer's ecosystem.
+
+### Synchronize the Hardware Clock
+
+This command tells Debian to treat the motherboard's clock as UTC (the server standard).
+
+```bash
+sudo timedatectl set-local-rtc 0
+```
+
+### Enable Linux-Native Firmware Updates
+
+We disabled the Windows update hook in the BIOS; now we replace it with the **Linux Vendor Firmware Service (LVFS)**.
+This allows you to update your BIOS directly from the Debian terminal.
+
+```bash
+sudo apt update && sudo apt install fwupd -y
+```
+
+`fwupd` does not download and auto-install updates like Windows. To list new available firmware from Lenovo/Intel, run:
+
+```bash
+fwupdmgr refresh && fwupdmgr get-updates
+```
+
+**Note:** You should only run `fwupdmgr update` during a scheduled maintenance window. BIOS updates take 2–5 minutes,
+during which the node is completely unresponsive and will reboot automatically.
+
+### OS Updates & Cluster Maintenance
+
+While `fwupd` handles the hardware, `apt` handles the OS. Debian 13 point releases (e.g., 13.2 to 13.3) are
+security-focused and will **not** auto-install by default.
+
+For a Kubernetes cluster, you should always perform **"Rolling Updates."** This means you drain one node at a time to
+ensure your pods are rescheduled elsewhere before you perform the upgrade and reboot.
+
+During a scheduled maintenance window:
+
+1. **Drain the node:** `kubectl drain <node-name> --ignore-daemonsets`
+2. **Upgrade OS:** `sudo apt update && sudo apt upgrade -y`
+3. **Upgrade Firmware:** `fwupdmgr update` (if applicable)
+4. **Reboot:** `sudo reboot`
+5. **Bring it back:** `kubectl uncordon <node-name>`