# Monitoring stack on k8s (VM: 178.18.243.51)

Migrated from docker-compose (`/root/monitoring`, `/root/CI-CD/sonarqube` on
the VM — left `docker compose stop`'d, not removed, as a rollback path) into
the VM's own k3s cluster, with real data migrated (not a fresh start):
Prometheus TSDB, Grafana dashboards/DB, SonarQube data+DB (confirmed intact:
the real `management-platform` SonarQube project and Grafana's existing
admin password both survived).

## Apply order
```
kubectl apply -f monitoring-pvcs.yaml
# migrate data into the PVCs via a loader pod per volume (see chat history —
# not scripted, was a one-time manual migration)
kubectl apply -f monitoring-apps.yaml
kubectl apply -f monitoring-ingress.yaml
```

## Domains — READ THIS BEFORE TOUCHING DNS
- `grafanna.nav.ovh` (note: **two Ns** — `grafana.nav.ovh` without the typo
  is a DIFFERENT, unrelated, pre-existing server at 109.199.127.74. Do not
  point anything at plain `grafana.nav.ovh` or touch that record.)
- `sonar.nav.ovh`
- `prom.nav.ovh`

All three already existed as A records pointing at 178.18.243.51 before this
work; only `nginx-vhost.conf` + a Let's Encrypt cert (via certbot, HTTP-01,
auto-renews) make them actually resolve to something.

## Why nginx is in front of Traefik, not Traefik directly on 80/443
This VM already runs an unrelated app ("nqks") on nginx, owning ports 80/443.
k3s's bundled Traefik defaults to a LoadBalancer Service, which on a
single-node k3s claims host 80/443 via iptables DNAT (klipper-lb) —
this silently hijacked nqks's traffic the first time (found and fixed
during this session). Traefik's Service is patched to ClusterIP-only via
a HelmChartConfig (`kubectl get helmchartconfig -n kube-system traefik`);
nginx reverse-proxies the 3 domains to Traefik's ClusterIP instead, and
handles TLS + BasicAuth itself.

## Access control
- Browser access to all 3 domains: HTTP Basic Auth (`/etc/nginx/.htpasswd-monitoring`
  on the VM, user `admin`) — ask whoever ran this migration for the password,
  it's not in git.
- **SonarQube is also reachable directly at `178.18.243.51:9000`, no auth** —
  this is deliberate, not an oversight. Jenkins' SonarQube server config
  (main cluster, `hudson.plugins.sonar.SonarGlobalConfiguration.xml`) is
  hardcoded to that exact address and authenticates via its own token, not
  BasicAuth (which would break the scanner). `sonarqube-jenkins-proxy.service`
  (systemd, socat) forwards host:9000 -> the sonarqube Service's ClusterIP;
  ufw restricts port 9000 to the main server's IP only, not the open
  internet. Do not add BasicAuth in front of this specific path.
  - Fragility note: the socat unit hardcodes the Service's ClusterIP. If the
    `sonarqube` Service in the `monitoring` namespace is ever deleted and
    recreated (not just the pod — pod recreates keep the same ClusterIP),
    update the `TCP:<ip>:9000` target in
    `/etc/systemd/system/sonarqube-jenkins-proxy.service` on the VM to match.

## Gotchas hit building this (context for next time)
- Don't add a k8s Service of type LoadBalancer on this cluster without first
  patching it to ClusterIP/setting a HelmChartConfig — same hijack risk as
  Traefik's default.
- Don't add `hostPort` to a pod that also needs normal in-cluster (Service/
  Ingress) traffic to reach it — it breaks pod-to-pod routing to that pod
  (confirmed directly: Traefik -> sonarqube hung/504'd the whole time
  hostPort was present, while direct host-to-pod and host-to-ClusterIP both
  worked fine — the asymmetry was the tell). Use a host-level proxy (socat)
  or a NodePort in the 30000-32767 range instead.
- SonarQube's DB was mid-upgrade (`DB_MIGRATION_NEEDED`) after the version
  jump — needed one `POST /api/system/migrate_db` (admin/admin was already
  changed on the real instance; used the actual admin session) — and the
  post-migration rule re-registration OOM'd on the compose file's original
  256MB web heap; bumped `SONAR_WEB_JAVAADDITIONALOPTS` to `-Xms512m -Xmx1536m`.