This repository contains the Infrastructure-as-Code (IaC) and manifest definitions for Severed, a modern blog and observability stack running on Kubernetes (K3d).

Architecture

The stack is designed to mimic a real-world AWS/Cloud environment but optimized for local development using K3d (k3s in Docker).

The Stack:

Cluster: K3d (Lightweight Kubernetes).
Ingress Controller: Traefik (Routing *.localhost domains).
Application: Jekyll Static Site served via Nginx (ConfigMap injected).
Observability (LGTM Stack):
Loki (Logs).
Grafana (Visualizations & Dashboards-as-Code).
Tempo (Tracing - Planned).
Monitoring / Prometheus (Metrics).
Agent: Grafana Alloy (OpenTelemetry Collector) running as a DaemonSet.

Repository Structure

Severed-Infra/
├── apps/                           # Application Manifests
│   ├── severed-blog.yaml           # Deployment + Service
│   ├── severed-blog-config.yaml    # Nginx ConfigMap (Decoupled Config)
│   └── severed-ingress.yaml        # Routing Rules (blog.localhost)
├── infra/                          # Infrastructure & Observability
│   ├── alloy-agent.yaml            # DaemonSet for Metrics/Logs Collection
│   ├── alloy-env.yaml              # Environment Variables
│   └── observer/                   # The Observability Stack
│       ├── loki.yaml               # Log Aggregation
│       ├── prometheus.yaml         # Metric Storage
│       ├── grafana.yaml            # Dashboard UI (Stateless)
│       └── dashboard-json.yaml     # "Cluster Health" Dashboard as Code
└── namespaces.yaml

Quick Start

1. Prerequisites

Ensure you have the following installed:

Docker Desktop
K3d (brew install k3d)
kubectl (brew install kubectl)

2. Boot the Cluster

Create a cluster with port mapping for the Ingress controller:

k3d cluster create severed-cluster -p "8080:80@loadbalancer"

3. Deploy Infrastructure (Observability)

Spin up the database backends (Loki/Prometheus) and the UI (Grafana).

# 1. Create the secret for Grafana Admin
kubectl create secret generic grafana-secrets \
  --namespace monitoring \
  --from-literal=admin-user=admin \
  --from-literal=admin-password=severed_secure_password

# 2. Deploy the stack
kubectl apply -f infra/observer/

# 3. Deploy the Collector Agent (Alloy)
kubectl apply -f infra/alloy-setup.yaml

4. Deploy the Application

Deploy the blog and its routing rules.

kubectl apply -f apps/

Access Points

Service	URL	Credentials / Notes
Severed Blog	http://blog.localhost:8080	Public Access
Grafana	http://grafana.localhost:8080	User: `admin` / Pass: `severed_secure_password` (Anonymous View Access Enabled)
Prometheus	Internal Only	Accessed via Alloy/Grafana
Loki	Internal Only	Accessed via Alloy/Grafana
K8s Dashboard	`https://localhost:8443`	Auth: Token-based. Access via `kubectl port-forward svc/kubernetes-dashboard-kong-proxy 8443:443 -n kubernetes-dashboard`.

Observability Features

This stack uses Grafana Alloy to automatically scrape metrics and tail logs from all pods.

Cluster Health Dashboard: Pre-provisioned "Infrastructure as Code." No manual setup required.
Real-time CPU/Memory usage per node.
Disk Usage monitoring (filtering out overlay/tmpfs noise).
RPS & Error Rates derived directly from Nginx logs using LogQL.
Log Relabeling: Alloy automatically promotes hidden K8s metadata (like app=severed-blog) into searchable Loki labels.

Engineering Decisions

ConfigMaps vs. Rebuilds: The Nginx configuration is injected via a ConfigMap (apps/blog-config.yaml). We can tweak caching headers or routing rules without rebuilding the Docker image.
Host Networking Fix: Alloy runs with hostNetwork: true to scrape node metrics but uses dnsPolicy: ClusterFirstWithHostNet to ensure it can still resolve internal K8s services (loki.monitoring.svc).
Security: Grafana admin credentials are stored in Kubernetes Secrets, not plaintext YAML. Anonymous access is restricted to Viewer role only.

Future Roadmap

Add Cert-Manager for TLS (HTTPS).
Implement ArgoCD for automated GitOps syncing.
Move to a physical Home Server.

todos/bugfixes

[ ] Automate Dashboard Auth: Rotate/retrieve the admin-user token to avoid manually create token every session.
[ ] External Secret Management: Replace generic secrets with HashiCorp Vault to encrypt grafana-secrets and dashboard tokens.
[ ] Ingress Hardening: Resolve the localhost 401 loop using Cert-Manager with self-signed certificates, which allows Kong to see valid HTTPS traffic and accept session cookies natively.
[ ] Persistence Layer: Deploy a Local Path Provisioner or HostPath storage class for Loki and Prometheus so that metrics and dashboard configurations survive a k3d cluster stop.
[ ] Resource Quotas: Define resources: requests/limits for the LGTM stack.