Compare commits

...

4 Commits

Author SHA1 Message Date
root
de0b1ee4db Fix backups Audit/Details for k8s-format archives + broken local access
Two separate bugs found while testing the Audit/Details UI against a
real k8s-format backup (myapps-k8s-backup-*):

1. Format-hardcoded checks. audit_backup()'s Internal Structure and
   Volume Count checks assumed the legacy Docker-era archive layout
   (volumes/*.tar.gz, compose-files/) and would fail every k8s-format
   backup (per-app dirs with manifests.yaml/db-dump.sql.gz/pvc-data.tar.gz)
   even when it's perfectly healthy - same root cause as the
   myapps-backup-*/myapps-k8s-backup-* prefix bug fixed earlier, just in
   the audit checks instead of the listing/delete glob. cloud_backup.py's
   r2_audit_backup() had the identical unpatched filename regex, and
   app.py's /api/backups/details route had its own, which outright
   400'd any k8s-format filename before even looking at it.

2. A bigger, separate bug this surfaced: get_local_backups(),
   get_vm_backups(), and _resolve_archive_path() all branch on
   RUNNING_ON_MAIN_SERVER to decide between direct filesystem access and
   SSH-to-self - and from inside the management-platform pod that flag's
   hostname check can be wrong for the pod's own ephemeral hostname
   depending on how it's evaluated, sending these down the SSH branch
   using a topology (separate warm-standby-on-the-VM SSH path) that
   doesn't apply to a pod that already has /root hostPath-mounted in.
   Fixed by trying direct filesystem access first wherever the archive
   would already be locally reachable, falling back to the existing SSH
   paths otherwise - this preserves the original warm-standby-on-VM
   failover design (a real second deployment of this platform on the VM
   host, kept reachable via SSH when the main server is down) completely
   unchanged; it only adds the fast local path for the case where the
   caller already has direct filesystem access.

Verified against both a real k8s-format backup (odoo, single-app) and a
real legacy-format backup on disk - both audit correctly now, with
format-appropriate checks and labels.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
2026-08-21 02:00:07 +02:00
root
a8f996422c Rewrite Application Sites page to reflect real k8s state
modules/sites.py was still a fully Docker-era hardcoded registry:
container names (odoo-clean-odoo-1, frappe-erpnext, nextcloud-app,
mautic-app, n8n-app) that no longer exist post-migration, checked via
`docker inspect` over SSH, plus hardcoded domain/port fields -
nextcloud and mautic even had domain: None, silently falling back to
dead Docker host-port URLs while their real Ingress domains
(next.cloud.nav.ovh, mautics.nav.ovh) sat unused.

Now sources everything live from the cluster:
- Domain + TLS from the actual Ingress object per app (via the new
  get_ingress_info() in modules/kubernetes.py), not a static guess.
  Odoo/Nextcloud's Ingress lives in `default` (see prior commit for the
  RBAC this needed); n8n/mautic/erpnext's lives in their own namespace.
- App/DB/cache/worker/scheduler status from real Deployment state
  (list_deployments_for_namespace(), new in modules/kubernetes.py)
  instead of `docker inspect`.
- Health probe now hits the real domain directly from the pod (it has
  normal internet egress) instead of SSH-ing back out to the host to
  curl a public HTTPS URL, which is what the RUNNING_ON_MAIN_SERVER
  branch would have done for a pod whose hostname never matches the
  main-server hostname check.

Same API/template contract as before (get_sites_list/get_site_health
return the same field shapes) - templates/pages/sites.html needs no
changes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
2026-08-21 01:43:38 +02:00
root
6b0402ff8b Add narrow ingress-read RBAC for management-platform-sa in default ns
Odoo's and Nextcloud's Ingress objects live in the default namespace
(historical, from the original Docker->k8s migration) rather than their
own app namespace like n8n/mautic/erpnext. management-platform-sa had
no binding there at all. Granting the full backup ClusterRole in
default would be way too broad (default also hosts unrelated live
services, and that role includes secrets/pod writes) - so this adds a
separate ClusterRole scoped to just ingresses get/list/watch, bound
only in default. Needed for the Application Sites page fix (next
commit) to read real domain/TLS state instead of guessing.

Applied and verified live: can read ingresses in default, still denied
secrets/pod-delete there.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
2026-08-21 01:43:37 +02:00
root
49b68cf820 Lower backup/restore RAM safety threshold 1536MB -> 1024MB
Investigated a resource-safety abort during Odoo restore testing:
available RAM was hovering ~1.3-1.5GB even under normal conditions on
this host (k3s + Jenkins + Docker monitoring/wazuh stack + dev tooling
baseline), not a leak from any single process. 1536MB was tripping on
ordinary single-app backups; 1024MB still leaves real headroom above
what a pg_dump/mysqldump/tar step actually needs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
2026-08-21 01:43:37 +02:00
8 changed files with 459 additions and 291 deletions

View File

@@ -156,8 +156,14 @@ check_resources() {
# management-platform pod's minimal image, /proc/meminfo always is.
avail_mb=$(awk '/^MemAvailable:/{printf "%d", $2/1024}' /proc/meminfo)
avail_gb=$(df -BG / | awk 'NR==2{gsub("G","",$4); print $4}')
if [ "$avail_mb" -lt 1536 ]; then
echo " ❌ RAM available ${avail_mb}MB < 1536MB threshold — aborting."
# Threshold lowered 1536->1024MB 2026-08-21: this host's steady-state
# available RAM sits ~1.3-1.5GB even when idle (k3s + Jenkins + Docker
# monitoring/wazuh stack + dev tooling baseline) — investigated, no
# single runaway process, just genuine multi-service baseline. 1536MB
# was tripping on ordinary single-app backups. 1024MB still leaves
# real headroom above what a pg_dump/mysqldump/tar op actually needs.
if [ "$avail_mb" -lt 1024 ]; then
echo " ❌ RAM available ${avail_mb}MB < 1024MB threshold — aborting."
log_status "FAILED" "$BACKUP_NAME" "resource_safety_ram_${avail_mb}mb"
exit 1
fi

View File

@@ -122,7 +122,10 @@ check_resources() {
# management-platform pod's minimal image, /proc/meminfo always is.
avail_mb=$(awk '/^MemAvailable:/{printf "%d", $2/1024}' /proc/meminfo)
avail_gb=$(df -BG / | awk 'NR==2{gsub("G","",$4); print $4}')
if [ "$avail_mb" -lt 1536 ] || [ "$avail_gb" -lt 10 ]; then
# Threshold lowered 1536->1024MB 2026-08-21 — see backup-k8s-apps.sh
# check_resources() comment for the investigation (baseline host RAM,
# not a leak).
if [ "$avail_mb" -lt 1024 ] || [ "$avail_gb" -lt 10 ]; then
echo " ❌ Resource safety threshold hit (RAM=${avail_mb}MB, Disk=${avail_gb}GB) — aborting."
exit 1
fi

View File

@@ -0,0 +1,32 @@
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: management-platform-ingress-reader-role
labels: {owner: ameni-boukattaya, app.kubernetes.io/managed-by: navitrends-pfe-migration}
rules:
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get", "list", "watch"]
---
# Odoo's and Nextcloud's Ingress objects live in the `default` namespace
# (historical — created there during the original Docker->k8s migration,
# unlike n8n/mautic/erpnext whose Ingress lives in their own app namespace).
# management-platform-sa has no binding at all in `default` otherwise, and
# `default` also hosts unrelated live services (cloudops-ingress, jenkins-
# ingress, stage-navitrends-ingress, etc.) — so this grants ONLY ingress
# read there, not the full backup ClusterRole, to keep blast radius to
# exactly what the Application Sites page needs (real domain/TLS lookup).
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: management-platform-ingress-reader-binding
namespace: default
labels: {owner: ameni-boukattaya, app.kubernetes.io/managed-by: navitrends-pfe-migration}
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: management-platform-ingress-reader-role
subjects:
- kind: ServiceAccount
name: management-platform-sa
namespace: management-platform

View File

@@ -461,7 +461,9 @@ def api_backup_details():
backup_file = data.get('backup_file', '').strip()
source = data.get('source', 'local').strip()
if not re.match(r'^myapps-backup-\d{8}_\d{6}\.tar\.gz$', backup_file):
# Accepts both the legacy Docker-era lineage (myapps-backup-*) and the
# k8s-native one (myapps-k8s-backup-*, from backup-k8s-apps.sh).
if not re.match(r'^myapps-(k8s-)?backup-\d{8}_\d{6}\.tar\.gz$', backup_file):
return jsonify({'error': 'Invalid filename'}), 400
if source == 'local':
@@ -484,10 +486,15 @@ def api_backup_details():
}
# ── size + mtime ──────────────────────────────────────────────
# Try direct stat first regardless of RUNNING_ON_MAIN_SERVER: the
# management-platform pod has /root hostPath-mounted from the main
# server, so /root/backups is already right here even though the
# hostname check that flag relies on never matches inside a pod (it
# used to always take the SSH branch, using a key/topology this
# in-pod deployment doesn't have, and silently come back empty).
if source == 'local':
if RUNNING_ON_MAIN_SERVER:
# direct stat on this machine
if os.path.exists(archive_path):
# direct stat on this machine
st = os.stat(archive_path)
result['size_bytes'] = st.st_size
result['size_human'] = _human_bytes(st.st_size)
@@ -517,9 +524,8 @@ def api_backup_details():
result['sha256'] = sha_out.strip()
elif source == 'vm':
if not RUNNING_ON_MAIN_SERVER:
# we ARE the VM — direct stat
if os.path.exists(archive_path):
# direct stat when reachable (VM's own local disk, or a mount)
st = os.stat(archive_path)
result['size_bytes'] = st.st_size
result['size_human'] = _human_bytes(st.st_size)

View File

@@ -55,10 +55,34 @@ def _ssh_main(remote_cmd, timeout=30):
# BACKUPS
# ────────────────────────────────────────────────────────────────
def _list_local_dir(local_dir):
"""Direct filesystem listing, newest first, both lineages."""
files = (
glob.glob(f'{local_dir}/myapps-backup-*.tar.gz')
+ glob.glob(f'{local_dir}/myapps-k8s-backup-*.tar.gz')
)
files.sort(key=os.path.getmtime, reverse=True)
return [os.path.basename(f) for f in files[:20]]
def get_local_backups():
# myapps-backup-* is the legacy Docker-era lineage; myapps-k8s-backup-*
# is the k8s-native one (backup-k8s-apps.sh). Listed together, sorted
# by mtime, so history from both stays visible in the UI.
#
# Try direct filesystem access first: the management-platform pod has
# /root hostPath-mounted from the main server (see k8s/management-
# platform-deployment.yaml), so /root/backups is already right here.
# RUNNING_ON_MAIN_SERVER's hostname check never matches inside a pod
# (its hostname is the pod name, not the host's), so this used to
# always fall through to the SSH-to-self branch below — which also
# doesn't actually work in this topology (wrong key for this deploy
# model) — silently returning an empty list. Falling back to SSH only
# when there's genuinely no local mount keeps this working for any
# future non-mounted deployment too.
if os.path.isdir('/root/backups'):
return _list_local_dir('/root/backups')
stdout, _ = _ssh_main(
"ls -t /root/backups/myapps-backup-*.tar.gz "
"/root/backups/myapps-k8s-backup-*.tar.gz 2>/dev/null | head -20"
@@ -73,8 +97,16 @@ def get_local_backups():
def get_vm_backups():
# Same fix as get_local_backups(): try direct access first (covers the
# case where this process's filesystem actually is the VM's), fall
# back to SSH to the VM host otherwise — which is the real case for
# the management-platform pod, since only /root is mounted in, not
# anything from the separate VM backup host.
local_dir = '/backups/cloudproject'
if os.path.isdir(local_dir):
return _list_local_dir(local_dir)
vm_backups = []
if RUNNING_ON_MAIN_SERVER:
try:
cmd = (
f"ssh -i {VM_KEY} -p {VM_PORT} "
@@ -92,15 +124,6 @@ def get_vm_backups():
vm_backups.append(os.path.basename(line))
except Exception as e:
print(f"[backups] VM backup fetch error: {e}")
else:
backup_dir = '/backups/cloudproject'
if os.path.exists(backup_dir):
files = (
glob.glob(f'{backup_dir}/myapps-backup-*.tar.gz')
+ glob.glob(f'{backup_dir}/myapps-k8s-backup-*.tar.gz')
)
files.sort(key=os.path.getmtime, reverse=True)
vm_backups = [os.path.basename(f) for f in files[:20]]
return vm_backups
@@ -115,16 +138,19 @@ def get_vm_backups():
# test, easy to extend without pushing complexity back up.
def _resolve_archive_path(backup_file, source, add):
"""Figure out the local path to the archive, pulling it from the main
server first if needed. Returns the path, or None if it could not be
resolved (in which case an error result dict has already been added
via `add` and the caller should bail out)."""
"""Figure out the local path to the archive, pulling it over SSH only
if it's genuinely not reachable on this filesystem. Returns the path,
or None if it could not be resolved (in which case an error result
dict has already been added via `add` and the caller should bail out).
Tries direct access first regardless of RUNNING_ON_MAIN_SERVER: the
management-platform pod has /root hostPath-mounted from the main
server, so /root/backups is already right here even though the
hostname check that flag relies on never matches inside a pod."""
if source == 'local':
archive_path = f"/root/backups/{backup_file}"
else:
archive_path = f"/backups/cloudproject/{backup_file}"
if not RUNNING_ON_MAIN_SERVER and source == 'local':
if os.path.exists(archive_path):
return archive_path
tmp_path = f"/tmp/audit_{backup_file}"
if not os.path.exists(tmp_path):
pull_cmd = (
@@ -137,9 +163,24 @@ def _resolve_archive_path(backup_file, source, add):
if not os.path.exists(tmp_path):
add('File Access', 'fail', f'Could not pull from main server: {err}')
return None
archive_path = tmp_path
return tmp_path
else:
archive_path = f"/backups/cloudproject/{backup_file}"
if os.path.exists(archive_path):
return archive_path
tmp_path = f"/tmp/audit_vm_{backup_file}"
if not os.path.exists(tmp_path):
pull_cmd = (
f"scp -i {VM_KEY} -P {VM_PORT} "
f"-o StrictHostKeyChecking=no -o ConnectTimeout=15 "
f"{VM_USER}@{VM_HOST}:/backups/cloudproject/{backup_file} "
f"{tmp_path}"
)
_out, err = _run(pull_cmd, timeout=120)
if not os.path.exists(tmp_path):
add('File Access', 'fail', f'Could not pull from VM: {err}')
return None
return tmp_path
def _check_file_size(archive_path, add):
@@ -216,13 +257,54 @@ def _list_archive_members(archive_path):
return []
def _check_internal_structure(members, add):
# The two backup lineages have completely different internal layouts:
# - legacy (myapps-backup-*, from backup-myapps.sh): volumes/*.tar.gz,
# compose-files/, backup-info.txt at the archive root.
# - k8s-native (myapps-k8s-backup-*, from backup-k8s-apps.sh): per-app
# subdirs (n8n/, odoo/, mautic/, nextcloud/, frappe/) each with
# manifests.yaml, secret.yaml, db-dump.sql.gz, pvc-data.tar.gz, plus
# backup-info.txt at the archive root.
# Auditing a k8s-format archive against the legacy layout would always
# fail it ("volumes/ missing") even when it's a perfectly good backup —
# same root cause as the myapps-backup-*/myapps-k8s-backup-* prefix bug
# already fixed elsewhere in this file, just in the audit checks instead
# of the listing/delete glob.
_K8S_APP_NAMES = ('n8n', 'odoo', 'mautic', 'nextcloud', 'frappe')
def _is_k8s_format(backup_file):
return os.path.basename(backup_file).startswith('myapps-k8s-backup-')
def _check_internal_structure(members, backup_file, add):
if not members:
add('Internal Structure', 'warn', 'Could not inspect archive members')
return
has_volumes = any('volumes/' in m for m in members)
has_info = any('backup-info.txt' in m for m in members)
if _is_k8s_format(backup_file):
app_dirs = sorted({
parts[1] for m in members
if len(parts := m.split('/')) > 1 and parts[1] in _K8S_APP_NAMES
})
has_app_data = any(
m.endswith(('manifests.yaml', 'db-dump.sql.gz', 'pvc-data.tar.gz'))
for m in members
)
issues = []
if not has_app_data:
issues.append('no per-app manifests/db-dump/pvc-data found')
if not has_info:
issues.append('backup-info.txt missing')
if issues:
add('Internal Structure', 'fail', ' · '.join(issues))
return
add('Internal Structure', 'pass',
f'backup-info.txt ✓ apps: {", ".join(app_dirs) or "?"}')
return
has_volumes = any('volumes/' in m for m in members)
has_compose = any('compose-files/' in m for m in members)
vol_count = len([m for m in members if '/volumes/' in m and m.endswith('.tar.gz')])
@@ -302,7 +384,20 @@ def _check_executable_scripts(archive_path, add):
add('Executable Scripts', 'pass', 'No unexpected executable scripts found')
def _check_volume_count(members, add):
def _check_volume_count(members, backup_file, add):
if _is_k8s_format(backup_file):
pvc_archives = [m for m in members if m.endswith('/pvc-data.tar.gz')]
v = len(pvc_archives)
if v == 0:
add('App Data Archives', 'fail', 'No pvc-data.tar.gz found in backup')
elif v < 5:
add('App Data Archives', 'warn',
f'{v} app(s) backed up — fine if this was a --apps filtered run, '
f'otherwise expected 5 for a full backup')
else:
add('App Data Archives', 'pass', f'{v} app data archives present')
return
vol_archives = [m for m in members if 'volumes/' in m and m.endswith('.tar.gz')]
v = len(vol_archives)
if v == 0:
@@ -379,10 +474,10 @@ def audit_backup(backup_file, source='local'):
_check_archive_integrity(archive_path, add)
members = _list_archive_members(archive_path)
_check_internal_structure(members, add)
_check_internal_structure(members, backup_file, add)
_check_security_scan(members, add)
_check_executable_scripts(archive_path, add)
_check_volume_count(members, add)
_check_volume_count(members, backup_file, add)
score = _score_checks(checks)
has_fails = any(c['status'] == 'fail' for c in checks)

View File

@@ -345,10 +345,12 @@ def r2_audit_backup(key: str) -> dict:
else:
report["checks"].append(f"✅ File size looks healthy ({_human_size(size)})")
# Check 5: Filename format
# Check 5: Filename format — accepts both the legacy Docker-era
# lineage (myapps-backup-*) and the k8s-native one
# (myapps-k8s-backup-*, from backup-k8s-apps.sh).
import re
filename = key.replace("backups/", "")
if re.match(r"myapps-backup-\d{8}_\d{6}\.tar\.gz", filename):
if re.match(r"myapps-(k8s-)?backup-\d{8}_\d{6}\.tar\.gz", filename):
report["checks"].append("✅ Filename format is valid")
else:
report["warnings"].append(f"⚠️ Unexpected filename format: {filename}")

View File

@@ -21,11 +21,12 @@ NAMESPACES = [
_core_v1 = None
_apps_v1 = None
_custom_objects = None
_networking_v1 = None
K8S_AVAILABLE = False
def _init_client():
global _core_v1, _apps_v1, _custom_objects, K8S_AVAILABLE
global _core_v1, _apps_v1, _custom_objects, _networking_v1, K8S_AVAILABLE
if _core_v1 is not None or not _K8S_IMPORT_OK:
return
try:
@@ -33,6 +34,7 @@ def _init_client():
_core_v1 = client.CoreV1Api()
_apps_v1 = client.AppsV1Api()
_custom_objects = client.CustomObjectsApi()
_networking_v1 = client.NetworkingV1Api()
K8S_AVAILABLE = True
except Exception as e:
print(f"[kubernetes] in-cluster config unavailable: {e}")
@@ -181,6 +183,54 @@ def list_deployments():
return deployments
def get_ingress_info(name, namespace):
"""Real domain/TLS for one Ingress, or None if unavailable/not found.
Used by modules/sites.py instead of the old hardcoded Docker-era
domain fields — reflects actual live k8s Ingress state."""
_init_client()
if not K8S_AVAILABLE:
return None
try:
ing = _networking_v1.read_namespaced_ingress(name, namespace)
except Exception:
return None
host = None
if ing.spec and ing.spec.rules:
for rule in ing.spec.rules:
if rule.host:
host = rule.host
break
tls = bool(ing.spec and ing.spec.tls)
return {'host': host, 'tls': tls}
def list_deployments_for_namespace(ns):
"""Targeted single-namespace deployment list (vs. list_deployments()'s
fixed NAMESPACES sweep) — used by modules/sites.py per app."""
_init_client()
if not K8S_AVAILABLE:
return []
try:
resp = _apps_v1.list_namespaced_deployment(ns)
except Exception as e:
print(f"[kubernetes] list deployments failed for {ns}: {e}")
return []
out = []
for dep in resp.items:
desired = dep.spec.replicas or 0
ready = dep.status.ready_replicas or 0
containers = dep.spec.template.spec.containers or []
image = containers[0].image if containers else '—'
out.append({
'name': dep.metadata.name,
'desired': desired,
'ready': ready,
'health': _deploy_health(desired, ready),
'image': image,
})
return out
def get_pod_metrics():
"""Best-effort CPU/mem per pod via metrics-server. Returns {} if it's not
installed or unreachable — callers must treat an empty dict as 'no data',

View File

@@ -1,15 +1,27 @@
# modules/sites.py — Managed application sites registry + live status
import re
#
# REWRITTEN 2026-08-21: this used to be a fully Docker-era hardcoded registry
# (container names like odoo-clean-odoo-1/frappe-erpnext/nextcloud-app that
# no longer exist since the k8s migration, checked via `docker inspect` over
# SSH) with hardcoded domain/port fields — nextcloud and mautic even had
# `domain: None`, silently falling back to dead Docker host-port URLs while
# their real k8s Ingress domains (next.cloud.nav.ovh, mautics.nav.ovh) sat
# unused. Now sources everything live from the cluster: Deployment/pod
# status via modules/kubernetes.py's in-cluster client, and domain/TLS via
# the actual Ingress objects (not a static guess) — see
# management-platform-sites-rbac.yaml for the narrow ingress-read grant this
# needed in the `default` namespace (odoo/nextcloud's Ingress historically
# live there, not in their app namespace).
import time
from urllib.request import urlopen, Request
from urllib.error import URLError
import urllib.request
from config import MAIN_SERVER_IP, RUNNING_ON_MAIN_SERVER
from modules.backups import _ssh_main, get_all_root_containers, get_container_status
from modules.kubernetes import get_ingress_info, list_deployments_for_namespace
# ────────────────────────────────────────────────────────────────
# STATIC SITE REGISTRY (source of truth for Application Sites UI)
# STATIC SITE REGISTRY — identity/namespace/naming only. Live state
# (domain, TLS, deployment/pod health) is always fetched fresh below,
# never hardcoded.
# ────────────────────────────────────────────────────────────────
PLATFORM = {
@@ -35,20 +47,21 @@ SITES = [
'category': 'ERP',
'icon': 'erpnext',
'brand_color': '#0089FF',
'compose_dir': 'frappe-setup',
'main_container': 'frappe-erpnext',
'containers': [
{'name': 'frappe-erpnext', 'role': 'App', 'image': 'frappe/erpnext:latest'},
{'name': 'frappe-mariadb', 'role': 'Database', 'image': 'mariadb:10.6'},
{'name': 'frappe-redis', 'role': 'Cache / Queue', 'image': 'redis:alpine'},
],
'port': 8080,
'internal_port': 8000,
'domain': 'erpnext.navitrends.ovh',
'domain_protocol': 'http',
'ssl_configured': False,
'namespace': 'erpnext',
'main_deployment': 'erpnext-web',
'service_port': 8000,
'ingress_name': 'erpnext-ingress',
'ingress_namespace': 'erpnext',
'health_path': '/',
'volumes': ['frappe-setup_frappe-sites', 'frappe-setup_mariadb-data'],
'pvcs': ['erpnext-sites', 'erpnext-mariadb-data'],
'roles': {
'erpnext-web': 'App',
'erpnext-worker': 'Worker',
'erpnext-scheduler': 'Scheduler',
'erpnext-mariadb': 'Database',
'erpnext-redis-cache': 'Cache',
'erpnext-redis-queue': 'Queue',
},
},
{
'id': 'odoo',
@@ -57,19 +70,18 @@ SITES = [
'category': 'ERP',
'icon': 'odoo',
'brand_color': '#714B67',
'compose_dir': 'odoo-clean',
'main_container': 'odoo-clean-odoo-1',
'containers': [
{'name': 'odoo-clean-odoo-1', 'role': 'App', 'image': 'odoo:18'},
{'name': 'odoo-clean-db-1', 'role': 'Database', 'image': 'postgres:15'},
],
'port': 8069,
'internal_port': 8069,
'domain': 'odooo.nav.ovh',
'domain_protocol': 'https',
'ssl_configured': True,
'namespace': 'odoo',
'main_deployment': 'odoo',
'service_port': 8069,
# Historical: created in `default` during migration, not the `odoo` ns.
'ingress_name': 'odoo-ingress',
'ingress_namespace': 'default',
'health_path': '/web',
'volumes': ['odoo-clean_db-data', 'odoo-clean_odoo-etc'],
'pvcs': ['odoo-data', 'odoo-postgres-data'],
'roles': {
'odoo': 'App',
'odoo-postgres': 'Database',
},
},
{
'id': 'nextcloud',
@@ -78,19 +90,18 @@ SITES = [
'category': 'Storage',
'icon': 'nextcloud',
'brand_color': '#0082C9',
'compose_dir': 'nextcloud-setup',
'main_container': 'nextcloud-app',
'containers': [
{'name': 'nextcloud-app', 'role': 'App', 'image': 'nextcloud:latest'},
{'name': 'nextcloud-postgres', 'role': 'Database', 'image': 'postgres:15'},
],
'port': 8082,
'internal_port': 80,
'domain': 'next.cloud.nav.ovh',
'domain_protocol': 'https',
'ssl_configured': True,
'namespace': 'nextcloud',
'main_deployment': 'nextcloud',
'service_port': 80,
# Historical: created in `default` during migration, not the `nextcloud` ns.
'ingress_name': 'nextcloud-ingress',
'ingress_namespace': 'default',
'health_path': '/status.php',
'volumes': ['nextcloud-setup_nextcloud-data', 'nextcloud-setup_nextcloud-db-data'],
'pvcs': ['nextcloud-app-data', 'nextcloud-postgres-data'],
'roles': {
'nextcloud': 'App',
'nextcloud-postgres': 'Database',
},
},
{
'id': 'mautic',
@@ -99,20 +110,17 @@ SITES = [
'category': 'Marketing',
'icon': 'mautic',
'brand_color': '#4E5E9E',
'compose_dir': 'mautic-setup',
'main_container': 'mautic-app',
'containers': [
{'name': 'mautic-app', 'role': 'App', 'image': 'mautic/mautic:latest'},
{'name': 'mautic-mariadb', 'role': 'Database', 'image': 'mariadb:10.11'},
],
'port': 8081,
'internal_port': 80,
'domain': None,
'domain_protocol': 'http',
'ssl_configured': False,
'namespace': 'mautic',
'main_deployment': 'mautic',
'service_port': 80,
'ingress_name': 'mautic-ingress',
'ingress_namespace': 'mautic',
'health_path': '/',
'volumes': ['mautic-setup_mautic-data', 'mautic-setup_mautic-db-data'],
'networks': ['mautic-network'],
'pvcs': ['mautic-app-data', 'mautic-db-data'],
'roles': {
'mautic': 'App',
'mautic-mariadb': 'Database',
},
},
{
'id': 'n8n',
@@ -121,144 +129,107 @@ SITES = [
'category': 'Automation',
'icon': 'n8n',
'brand_color': '#EA4B71',
'compose_dir': 'n8n-setup',
'main_container': 'n8n-app',
'containers': [
{'name': 'n8n-app', 'role': 'App', 'image': 'n8nio/n8n:latest'},
{'name': 'n8n-postgres', 'role': 'Database', 'image': 'postgres:15'},
],
'port': 5678,
'internal_port': 5678,
'domain': None,
'domain_protocol': 'http',
'ssl_configured': False,
'namespace': 'n8n',
'main_deployment': 'n8n',
'service_port': 5678,
'ingress_name': 'n8n-ingress',
'ingress_namespace': 'n8n',
'health_path': '/healthz',
'volumes': ['n8n-setup_n8n-data', 'n8n-setup_n8n-db-data'],
'networks': ['n8n-network', 'integration-network'],
'pvcs': ['n8n-data', 'n8n-postgres-data'],
'roles': {
'n8n': 'App',
'n8n-postgres': 'Database',
},
},
]
SITE_BY_ID = {s['id']: s for s in SITES}
def _build_urls(site):
ip = MAIN_SERVER_IP
port = site['port']
proto = site.get('domain_protocol', 'http')
def _build_urls(site, ingress_info):
"""Domain/TLS come from the live Ingress object — never hardcoded.
ingress_info is None if the Ingress couldn't be read (RBAC/API issue)
or {'host': str|None, 'tls': bool} otherwise."""
path = site.get('health_path', '/') or '/'
ssl_configured = site.get('ssl_configured', proto == 'https')
domain = site.get('domain')
has_domain = bool(domain)
host = ingress_info.get('host') if ingress_info else None
has_domain = bool(host)
ssl_configured = bool(ingress_info and ingress_info.get('tls'))
if has_domain:
if ssl_configured:
access_url = f"{proto}://{domain}"
proto = 'https' if ssl_configured else 'http'
access_url = f"{proto}://{host}"
health_url = f"{access_url}{path}"
else:
access_url = f"http://{domain}:{port}"
health_url = f"{access_url}{path}"
else:
access_url = f"http://{ip}:{port}"
health_url = f"{access_url}{path}"
access_url = None
health_url = None
return {
'has_domain': has_domain,
'domain': domain,
'domain_protocol': proto,
'domain': host,
'domain_protocol': 'https' if ssl_configured else 'http',
'ssl_configured': ssl_configured,
'ip_url': f"http://{ip}:{port}",
'access_url': access_url.rstrip('/'),
'access_url': (access_url or '').rstrip('/'),
'health_url': health_url,
}
def _parse_port_mapping(ports_str, host_port, internal_port=None):
"""Extract host:container mapping from docker ports string."""
if not ports_str:
if internal_port and internal_port != host_port:
return f"{host_port}→{internal_port}"
return str(host_port) if host_port else None
m = re.search(rf'0\.0\.0\.0:{host_port}->(\d+)/', ports_str)
if m:
return f"{host_port}→{m.group(1)}"
m = re.search(r'0\.0\.0\.0:(\d+)->(\d+)/', ports_str)
if m:
return f"{m.group(1)}→{m.group(2)}"
return ports_str[:60] if ports_str else None
def _probe_http(url, timeout=5):
"""HTTP health probe — local urllib or remote curl via SSH."""
if RUNNING_ON_MAIN_SERVER:
"""Direct HTTP probe. The management-platform pod has normal internet
egress and resolves public DNS just like any other pod — no need to
hop back out to the host via SSH the way the old Docker-era code did."""
try:
req = Request(url, method='GET', headers={'User-Agent': 'CloudOps-HealthCheck/1.0'})
with urlopen(req, timeout=timeout) as resp:
req = urllib.request.Request(url, method='GET', headers={'User-Agent': 'CloudOps-HealthCheck/1.0'})
t0 = time.time()
with urllib.request.urlopen(req, timeout=timeout) as resp:
code = resp.getcode()
latency_ms = int((time.time() - t0) * 1000)
return {
'reachable': 200 <= code < 400,
'http_code': code,
'latency_ms': None,
'latency_ms': latency_ms,
'error': None,
}
except URLError as e:
return {'reachable': False, 'http_code': None, 'latency_ms': None, 'error': str(e.reason)[:120]}
except Exception as e:
return {'reachable': False, 'http_code': None, 'latency_ms': None, 'error': str(e)[:120]}
else:
safe_url = url.replace("'", "'\\''")
t0 = time.time()
out, _ = _ssh_main(
f"curl -sf -o /dev/null -w '%{{http_code}}' --connect-timeout {timeout} "
f"-m {timeout} '{safe_url}' 2>/dev/null || echo '000'",
timeout=timeout + 5,
)
latency = int((time.time() - t0) * 1000)
code_str = (out or '000').strip()
try:
code = int(code_str)
except ValueError:
code = 0
return {
'reachable': 200 <= code < 400,
'http_code': code if code else None,
'latency_ms': latency,
'error': None if 200 <= code < 400 else f'HTTP {code_str}',
}
def _container_map():
ctrs = get_all_root_containers()
return {c['name']: c for c in ctrs}
def get_sites_list(include_health=False):
"""Return all sites with live container + URL metadata."""
ctr_map = _container_map()
results = []
for site in SITES:
urls = _build_urls(site)
main_name = site['main_container']
main_ctr = ctr_map.get(main_name, {})
main_status = get_container_status(main_name) if main_name else {'status': 'unknown'}
containers_live = []
def _deployments_for_site(site):
"""Live Deployment status for every stack member (app/db/cache/etc),
keyed by k8s Deployment name."""
live = {d['name']: d for d in list_deployments_for_namespace(site['namespace'])}
roles = site.get('roles', {})
containers = []
running_count = 0
for ctr_def in site.get('containers', []):
name = ctr_def['name']
live = ctr_map.get(name, {})
status_raw = live.get('status', '')
is_up = 'Up' in status_raw
for dep_name, role in roles.items():
d = live.get(dep_name)
if d:
is_up = d['ready'] >= 1 and d['desired'] >= 1
if is_up:
running_count += 1
containers_live.append({
**ctr_def,
'status': 'running' if is_up else ('stopped' if live else 'not_found'),
'status_raw': status_raw or 'not found',
'image_live': live.get('image', ctr_def.get('image', '—')),
'ports': live.get('ports', '—'),
containers.append({
'name': dep_name,
'role': role,
'status': 'running' if is_up else 'stopped',
'image_live': d['image'],
'ports': f"{d['ready']}/{d['desired']} ready",
})
else:
containers.append({
'name': dep_name,
'role': role,
'status': 'not_found',
'image_live': '—',
'ports': '—',
})
main = live.get(site['main_deployment'])
main_running = bool(main and main['ready'] >= 1 and main['desired'] >= 1)
return containers, running_count, main_running
def _site_entry(site, include_health):
ingress_info = get_ingress_info(site['ingress_name'], site['ingress_namespace'])
urls = _build_urls(site, ingress_info)
containers, running_count, main_running = _deployments_for_site(site)
entry = {
'id': site['id'],
@@ -267,64 +238,67 @@ def get_sites_list(include_health=False):
'category': site['category'],
'icon': site['icon'],
'brand_color': site['brand_color'],
'compose_dir': site['compose_dir'],
'main_container': main_name,
'port': site['port'],
'internal_port': site.get('internal_port', site['port']),
'port_mapping': _parse_port_mapping(
main_ctr.get('ports', ''), site['port'], site.get('internal_port')
),
'ports_raw': main_ctr.get('ports', '—'),
'volumes': site.get('volumes', []),
'networks': site.get('networks', ['default']),
'container_status': main_status['status'],
'container_running': main_status['status'] == 'running',
'containers': containers_live,
'compose_dir': site['namespace'], # k8s namespace, shown as "COMPOSE PROJECT" in the UI
'main_container': site['main_deployment'],
'port': site['service_port'],
'internal_port': site['service_port'],
'port_mapping': f"svc/{site['service_port']}",
'ports_raw': f"ClusterIP:{site['service_port']}",
'volumes': site.get('pvcs', []),
'ip_url': f"{site['main_deployment']}.{site['namespace']}.svc.cluster.local:{site['service_port']} (cluster-internal only)",
'container_status': 'running' if main_running else 'stopped',
'container_running': main_running,
'containers': containers,
'containers_running': running_count,
'containers_total': len(site.get('containers', [])),
'containers_total': len(site.get('roles', {})),
**urls,
}
if include_health:
if entry['container_running']:
probe = _probe_http(urls['health_url'])
if not urls['has_domain']:
entry['health'] = {
'status': 'up' if probe['reachable'] else 'down',
**probe,
'status': 'down', 'reachable': False, 'http_code': None,
'latency_ms': None, 'error': 'No Ingress host found for this app',
}
elif not main_running:
entry['health'] = {
'status': 'offline', 'reachable': False, 'http_code': None,
'latency_ms': None, 'error': 'Deployment not running',
}
else:
entry['health'] = {
'status': 'offline',
'reachable': False,
'http_code': None,
'latency_ms': None,
'error': 'Container not running',
}
probe = _probe_http(urls['health_url'])
entry['health'] = {'status': 'up' if probe['reachable'] else 'down', **probe}
results.append(entry)
return entry
return results
def get_sites_list(include_health=False):
"""Return all sites with live k8s Deployment + Ingress state."""
return [_site_entry(site, include_health) for site in SITES]
def get_site_health(site_id):
site = SITE_BY_ID.get(site_id)
if not site:
return None
urls = _build_urls(site)
status = get_container_status(site['main_container'])
if status['status'] != 'running':
ingress_info = get_ingress_info(site['ingress_name'], site['ingress_namespace'])
urls = _build_urls(site, ingress_info)
_, _, main_running = _deployments_for_site(site)
if not main_running:
return {
'site_id': site_id,
'status': 'offline',
'reachable': False,
'http_code': None,
'latency_ms': None,
'error': 'Container not running',
'site_id': site_id, 'status': 'offline', 'reachable': False,
'http_code': None, 'latency_ms': None, 'error': 'Deployment not running',
'checked_at': time.time(),
}
t0 = time.time()
if not urls['has_domain']:
return {
'site_id': site_id, 'status': 'down', 'reachable': False,
'http_code': None, 'latency_ms': None, 'error': 'No Ingress host found',
'checked_at': time.time(),
}
probe = _probe_http(urls['health_url'])
probe['latency_ms'] = probe.get('latency_ms') or int((time.time() - t0) * 1000)
probe['site_id'] = site_id
probe['status'] = 'up' if probe['reachable'] else 'down'
probe['checked_at'] = time.time()