This repository contains the Infrastructure-as-Code (IaC) and manifest definitions for Severed, a modern blog and observability stack running on Kubernetes (K3d). It demonstrates how to decouple configuration from code, automate observability, and secure internal services—all running locally on your laptop.

Architecture

The stack is designed to mimic a real-world AWS/Cloud environment but optimized for local development using K3d (k3s in Docker).

The Stack:

Cluster: K3d (Lightweight Kubernetes).
Ingress Controller: Traefik (Routing *.localhost domains).
Application: Jekyll Static Site served via Nginx (ConfigMap injected).
Observability (LGTM Stack):
Loki (Logs).
Grafana (Visualizations & Dashboards-as-Code).
Tempo (Tracing - Planned).
Monitoring / Prometheus (Metrics).
Agent: Grafana Alloy (OpenTelemetry Collector) running as a DaemonSet.

Repository Structure

Severed-Infra/
├── apps/                           # Application Manifests
│   ├── severed-blog.yaml           # Deployment + Service
│   ├── severed-blog-config.yaml    # Nginx ConfigMap (Decoupled Config)
│   └── severed-ingress.yaml        # Routing Rules (blog.localhost)
├── infra/                          # Infrastructure & Observability
│   ├── alloy-agent.yaml            # DaemonSet for Metrics/Logs Collection
│   ├── alloy-env.yaml              # Environment Variables
│   └── observer/                   # The Observability Stack
│       ├── loki.yaml               # Log Aggregation
│       ├── prometheus.yaml         # Metric Storage
│       ├── grafana.yaml            # Dashboard UI (Stateless)
│       └── dashboard-json.yaml     # "Cluster Health" Dashboard as Code
└── namespaces.yaml

Quick Start

1. Prerequisites

Ensure you have the following installed:

Docker Desktop
K3d (brew install k3d)
kubectl (brew install kubectl)

2. Boot the Cluster

Create a cluster with port mapping for the Ingress controller:

k3d cluster create severed-cluster -p "8080:80@loadbalancer"

3. Deploy Infrastructure (Observability)

Spin up the database backends (Loki/Prometheus) and the UI (Grafana).

# 1. Create the secret for Grafana Admin
kubectl create secret generic grafana-secrets \
  --namespace monitoring \
  --from-literal=admin-user=admin \
  --from-literal=admin-password=severed_secure_password

# 2. Deploy the stack
kubectl apply -f infra/observer/

# 3. Deploy the Collector Agent (Alloy)
kubectl apply -f infra/alloy-setup.yaml

4. Deploy the Application

Deploy the blog and its routing rules.

kubectl apply -f apps/

Access Points

Service	URL	Credentials / Notes
Severed Blog	http://blog.localhost:8080	Public Access
Grafana	http://grafana.localhost:8080	User: `admin` / Pass: `severed_secure_password` (Anonymous View Access Enabled)
Prometheus	Internal Only	Accessed via Alloy/Grafana
Loki	Internal Only	Accessed via Alloy/Grafana
K8s Dashboard	`https://localhost:8443`	Auth: Token-based. Access via `kubectl port-forward svc/kubernetes-dashboard-kong-proxy 8443:443 -n kubernetes-dashboard`.

Observability Features

This stack uses Grafana Alloy to automatically scrape metrics and tail logs from all pods.

Cluster Health Dashboard: Pre-provisioned "Infrastructure as Code." No manual setup required.
Real-time CPU/Memory usage per node.
Disk Usage monitoring (filtering out overlay/tmpfs noise).
RPS & Error Rates derived directly from Nginx logs using LogQL.
Log Relabeling: Alloy automatically promotes hidden K8s metadata (like app=severed-blog) into searchable Loki labels.

Engineering Decisions

ConfigMaps vs. Rebuilds: The Nginx configuration is injected via a ConfigMap (apps/blog-config.yaml). We can tweak caching headers or routing rules without rebuilding the Docker image.
Host Networking Fix: Alloy runs with hostNetwork: true to scrape node metrics but uses dnsPolicy: ClusterFirstWithHostNet to ensure it can still resolve internal K8s services (loki.monitoring.svc).
Security: Grafana admin credentials are stored in Kubernetes Secrets, not plaintext YAML. Anonymous access is restricted to Viewer role only.

Future Roadmap

Add Cert-Manager for TLS (HTTPS).
Implement ArgoCD for automated GitOps syncing.
Move to a physical Home Server.

todos/bugfixes

[ ] Automate Dashboard Auth: Rotate/retrieve the admin-user token to avoid manually create token every session.
[ ] External Secret Management: Replace generic secrets with HashiCorp Vault to encrypt grafana-secrets and dashboard tokens.
[ ] Ingress Hardening: Resolve the localhost 401 loop using Cert-Manager with self-signed certificates, which allows Kong to see valid HTTPS traffic and accept session cookies natively.
[ ] Persistence Layer: Deploy a Local Path Provisioner or HostPath storage class for Loki and Prometheus so that metrics and dashboard configurations survive a k3d cluster stop.
[ ] Resource Quotas: Define resources: requests/limits for the LGTM stack.