a307607a99
Real data migrated, not a fresh start: 1.2GB Prometheus TSDB, Grafana's
existing dashboards/DB (its admin password was already changed from
default — confirmed real, in-use data), SonarQube's data/logs/extensions
and its own Postgres DB (confirmed: the actual `management-platform`
SonarQube project survived, matching the Jenkinsfile's own
-Dsonar.projectKey). Old docker-compose services stopped, not removed —
left as a rollback path.
Real problems hit and fixed along the way (see k8s/monitoring/README.md
for the full detail, worth reading before touching any of this again):
- k3s's bundled Traefik defaults to a LoadBalancer Service, which
immediately hijacked ports 80/443 via iptables DNAT (klipper-lb) out
from under this VM's existing unrelated nginx-served app ("nqks") the
moment k3s was installed. Patched Traefik to ClusterIP via a
HelmChartConfig; nginx now reverse-proxies the 3 new domains to
Traefik's ClusterIP instead of Traefik touching host ports at all.
- Caught my own mistake before it caused damage: "corrected" what I
assumed was a typo (grafanna.nav.ovh -> grafana.nav.ovh) — the two-N
version is what the user actually specified, and is what points at this
VM; plain grafana.nav.ovh is a different, unrelated, pre-existing live
server at 109.199.127.74. Renamed everything back before requesting any
cert or touching that domain.
- Adding hostPort to the SonarQube pod (for Jenkins' hardcoded
178.18.243.51:9000) broke pod-to-pod routing to that pod entirely —
Traefik couldn't reach it (hung/504) while direct host-to-pod and
host-to-ClusterIP both worked fine the whole time, which is what gave
it away. Replaced with a host-level socat proxy (systemd unit) to the
Service's ClusterIP instead, which touches nothing pod-level. Verified
Jenkins' exact address still works AND Traefik routing to the same pod
works simultaneously.
- SonarQube needed a DB migration after the version jump
(DB_MIGRATION_NEEDED), then OOM'd during post-migration rule
re-registration on the compose file's original 256MB web heap — bumped
SONAR_WEB_JAVAADDITIONALOPTS.
TLS: real Let's Encrypt certs via certbot (HTTP-01, nginx already owns
port 80 so no DNS-01/OVH-webhook replication needed on this second
cluster), auto-renewing. Access: HTTP Basic Auth in front of all three
domains for browser access; SonarQube also stays reachable unauthenticated
at 178.18.243.51:9000 specifically for Jenkins (which authenticates via
its own token, not BasicAuth — that path would break the scanner),
restricted to the main server's IP via ufw, not open to the internet.
Verified end-to-end: all three https://*.nav.ovh domains return 401
without credentials and load correctly with them; Jenkins' exact
SonarQube address (no auth) still returns 200; real data confirmed
present in Grafana, Prometheus, and SonarQube post-migration.
102 lines
3.1 KiB
Plaintext
102 lines
3.1 KiB
Plaintext
server {
|
|
server_name grafanna.nav.ovh;
|
|
location / {
|
|
auth_basic "Restricted";
|
|
auth_basic_user_file /etc/nginx/.htpasswd-monitoring;
|
|
proxy_pass http://10.43.200.43;
|
|
proxy_set_header Host grafanna.nav.ovh;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Upgrade $http_upgrade;
|
|
proxy_set_header Connection "upgrade";
|
|
}
|
|
|
|
listen 443 ssl; # managed by Certbot
|
|
ssl_certificate /etc/letsencrypt/live/grafanna.nav.ovh/fullchain.pem; # managed by Certbot
|
|
ssl_certificate_key /etc/letsencrypt/live/grafanna.nav.ovh/privkey.pem; # managed by Certbot
|
|
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
|
|
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
|
|
|
|
}
|
|
|
|
server {
|
|
server_name sonar.nav.ovh;
|
|
location / {
|
|
auth_basic "Restricted";
|
|
auth_basic_user_file /etc/nginx/.htpasswd-monitoring;
|
|
proxy_pass http://10.43.200.43;
|
|
proxy_set_header Host sonar.nav.ovh;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
|
proxy_http_version 1.1;
|
|
}
|
|
|
|
listen 443 ssl; # managed by Certbot
|
|
ssl_certificate /etc/letsencrypt/live/grafanna.nav.ovh/fullchain.pem; # managed by Certbot
|
|
ssl_certificate_key /etc/letsencrypt/live/grafanna.nav.ovh/privkey.pem; # managed by Certbot
|
|
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
|
|
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
|
|
|
|
}
|
|
|
|
server {
|
|
server_name prom.nav.ovh;
|
|
location / {
|
|
auth_basic "Restricted";
|
|
auth_basic_user_file /etc/nginx/.htpasswd-monitoring;
|
|
proxy_pass http://10.43.200.43;
|
|
proxy_set_header Host prom.nav.ovh;
|
|
proxy_set_header X-Real-IP $remote_addr;
|
|
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
|
proxy_set_header X-Forwarded-Proto $scheme;
|
|
proxy_http_version 1.1;
|
|
}
|
|
|
|
listen 443 ssl; # managed by Certbot
|
|
ssl_certificate /etc/letsencrypt/live/grafanna.nav.ovh/fullchain.pem; # managed by Certbot
|
|
ssl_certificate_key /etc/letsencrypt/live/grafanna.nav.ovh/privkey.pem; # managed by Certbot
|
|
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
|
|
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
|
|
|
|
}
|
|
server {
|
|
if ($host = grafanna.nav.ovh) {
|
|
return 301 https://$host$request_uri;
|
|
} # managed by Certbot
|
|
|
|
|
|
listen 80;
|
|
server_name grafanna.nav.ovh;
|
|
return 404; # managed by Certbot
|
|
|
|
|
|
}
|
|
|
|
server {
|
|
if ($host = sonar.nav.ovh) {
|
|
return 301 https://$host$request_uri;
|
|
} # managed by Certbot
|
|
|
|
|
|
listen 80;
|
|
server_name sonar.nav.ovh;
|
|
return 404; # managed by Certbot
|
|
|
|
|
|
}
|
|
|
|
server {
|
|
if ($host = prom.nav.ovh) {
|
|
return 301 https://$host$request_uri;
|
|
} # managed by Certbot
|
|
|
|
|
|
listen 80;
|
|
server_name prom.nav.ovh;
|
|
return 404; # managed by Certbot
|
|
|
|
|
|
} |