Migrate Grafana/Prometheus/SonarQube from docker-compose to k8s (step #10)

Real data migrated, not a fresh start: 1.2GB Prometheus TSDB, Grafana's
existing dashboards/DB (its admin password was already changed from
default — confirmed real, in-use data), SonarQube's data/logs/extensions
and its own Postgres DB (confirmed: the actual `management-platform`
SonarQube project survived, matching the Jenkinsfile's own
-Dsonar.projectKey). Old docker-compose services stopped, not removed —
left as a rollback path.

Real problems hit and fixed along the way (see k8s/monitoring/README.md
for the full detail, worth reading before touching any of this again):

- k3s's bundled Traefik defaults to a LoadBalancer Service, which
  immediately hijacked ports 80/443 via iptables DNAT (klipper-lb) out
  from under this VM's existing unrelated nginx-served app ("nqks") the
  moment k3s was installed. Patched Traefik to ClusterIP via a
  HelmChartConfig; nginx now reverse-proxies the 3 new domains to
  Traefik's ClusterIP instead of Traefik touching host ports at all.

- Caught my own mistake before it caused damage: "corrected" what I
  assumed was a typo (grafanna.nav.ovh -> grafana.nav.ovh) — the two-N
  version is what the user actually specified, and is what points at this
  VM; plain grafana.nav.ovh is a different, unrelated, pre-existing live
  server at 109.199.127.74. Renamed everything back before requesting any
  cert or touching that domain.

- Adding hostPort to the SonarQube pod (for Jenkins' hardcoded
  178.18.243.51:9000) broke pod-to-pod routing to that pod entirely —
  Traefik couldn't reach it (hung/504) while direct host-to-pod and
  host-to-ClusterIP both worked fine the whole time, which is what gave
  it away. Replaced with a host-level socat proxy (systemd unit) to the
  Service's ClusterIP instead, which touches nothing pod-level. Verified
  Jenkins' exact address still works AND Traefik routing to the same pod
  works simultaneously.

- SonarQube needed a DB migration after the version jump
  (DB_MIGRATION_NEEDED), then OOM'd during post-migration rule
  re-registration on the compose file's original 256MB web heap — bumped
  SONAR_WEB_JAVAADDITIONALOPTS.

TLS: real Let's Encrypt certs via certbot (HTTP-01, nginx already owns
port 80 so no DNS-01/OVH-webhook replication needed on this second
cluster), auto-renewing. Access: HTTP Basic Auth in front of all three
domains for browser access; SonarQube also stays reachable unauthenticated
at 178.18.243.51:9000 specifically for Jenkins (which authenticates via
its own token, not BasicAuth — that path would break the scanner),
restricted to the main server's IP via ufw, not open to the internet.

Verified end-to-end: all three https://*.nav.ovh domains return 401
without credentials and load correctly with them; Jenkins' exact
SonarQube address (no auth) still returns 200; real data confirmed
present in Grafana, Prometheus, and SonarQube post-migration.

ansible/dr-bootstrap.yml

@@ -0,0 +1,179 @@
+---
+# dr-bootstrap.yml — minimum-viable cross-server disaster recovery.
+#
+# Provisions a target server into a working k3s node (checking the
+# environment first, installing only if missing — safe to re-run) and
+# restores one app from a captured backup-k8s-apps.sh archive onto it,
+# from scratch: creates the namespace, the PVC(s) (never captured by the
+# backup before this — see sanitize_k8s_manifest.py's header) and the
+# Deployments/Services/Secret, then hands off to the existing
+# restore-k8s-apps.sh for the actual data population (PVC contents + DB),
+# which already assumes namespace/PVC/Deployments exist — this playbook is
+# what makes that assumption true on a target that starts with nothing.
+#
+# Usage:
+#   ansible-playbook -i inventory.ini dr-bootstrap.yml \
+#     -e dr_app=n8n -e dr_backup_archive=/root/backups/myapps-k8s-backup-XXXXXXXX_XXXXXX.tar.gz
+#
+# NOT in scope (same cut as restore-k8s-apps.sh's own header): this
+# targets a server that already has network/DNS/SSH access sorted out —
+# it provisions the CLUSTER and the APP, not the surrounding
+# infrastructure.
+
+- name: DR Bootstrap
+  hosts: all
+  gather_facts: yes
+  vars:
+    remote_work_dir: "/root/dr-bootstrap-{{ dr_app }}"
+    k3s_version: "v1.35.4+k3s1"
+
+  tasks:
+    - name: Fail fast if required vars are missing
+      ansible.builtin.assert:
+        that:
+          - dr_app is defined
+          - dr_backup_archive is defined
+        fail_msg: "Pass -e dr_app=<name> -e dr_backup_archive=<local path to a myapps-k8s-backup-*.tar.gz>"
+
+    # ── 1. Environment check — before touching anything ──────────────
+    - name: Check free disk space and memory
+      ansible.builtin.set_fact:
+        disk_free_gb: "{{ ((ansible_facts['mounts'] | selectattr('mount', 'equalto', '/') | first).size_available / (1024**3)) | round(1) }}"
+
+    - name: Assert environment meets k3s minimums
+      ansible.builtin.assert:
+        that:
+          - disk_free_gb | float > 5
+          - ansible_facts['memtotal_mb'] > 1500
+        fail_msg: >
+          Environment below k3s minimums on {{ inventory_hostname }}:
+          {{ disk_free_gb }}GB free disk / {{ ansible_facts['memtotal_mb'] }}MB RAM
+          (need >5GB / >1.5GB). Not proceeding.
+        success_msg: >
+          Environment OK on {{ inventory_hostname }}: {{ disk_free_gb }}GB free disk,
+          {{ ansible_facts['memtotal_mb'] }}MB RAM, {{ ansible_facts['processor_vcpus'] }} vCPUs.
+
+    # ── 2. k3s — install only if missing, never re-run blindly ───────
+    - name: Check if k3s is already installed
+      ansible.builtin.command: which k3s
+      register: k3s_check
+      ignore_errors: true
+      changed_when: false
+
+    - name: Install k3s (skipped if already present)
+      ansible.builtin.shell: >
+        curl -sfL https://get.k3s.io |
+        INSTALL_K3S_VERSION="{{ k3s_version }}" sh -s - --write-kubeconfig-mode 644
+      when: k3s_check.rc != 0
+
+    - name: Wait for the k3s node to report Ready
+      ansible.builtin.shell: k3s kubectl get nodes --no-headers | awk '{print $2}'
+      register: node_status
+      until: "'Ready' in node_status.stdout"
+      retries: 24
+      delay: 5
+      changed_when: false
+
+    # ── 3. Stage the backup + tooling on the target ───────────────────
+    - name: Create working directory
+      ansible.builtin.file:
+        path: "{{ remote_work_dir }}"
+        state: directory
+        mode: '0755'
+
+    - name: Copy backup archive to target
+      ansible.builtin.copy:
+        src: "{{ dr_backup_archive }}"
+        dest: "{{ remote_work_dir }}/backup.tar.gz"
+
+    - name: Extract backup archive
+      ansible.builtin.unarchive:
+        src: "{{ remote_work_dir }}/backup.tar.gz"
+        dest: "{{ remote_work_dir }}"
+        remote_src: true
+        extra_opts: ['--strip-components=1']
+
+    - name: Copy sanitizer script
+      ansible.builtin.copy:
+        src: sanitize_k8s_manifest.py
+        dest: "{{ remote_work_dir }}/sanitize_k8s_manifest.py"
+        mode: '0755'
+
+    - name: Copy restore-k8s-apps.sh
+      ansible.builtin.copy:
+        src: "{{ playbook_dir }}/../backup/restore-k8s-apps.sh"
+        dest: "{{ remote_work_dir }}/restore-k8s-apps.sh"
+        mode: '0755'
+
+    - name: Determine the app's namespace from the captured manifests
+      ansible.builtin.shell: >
+        python3 -c "import yaml; d=yaml.safe_load(open('{{ remote_work_dir }}/{{ dr_app }}/manifests.yaml'));
+        print(d['items'][0]['metadata']['namespace'])"
+      register: ns_result
+      changed_when: false
+
+    - name: Set dr_namespace fact
+      ansible.builtin.set_fact:
+        dr_namespace: "{{ ns_result.stdout }}"
+
+    # ── 4. Provision namespace + PVC + Secret + Deployments/Services ──
+    - name: Create namespace (idempotent)
+      ansible.builtin.shell: >
+        k3s kubectl get namespace {{ dr_namespace }} ||
+        k3s kubectl create namespace {{ dr_namespace }}
+      changed_when: false
+
+    - name: Check for a captured PVC spec
+      ansible.builtin.stat:
+        path: "{{ remote_work_dir }}/{{ dr_app }}/pvc.yaml"
+      register: pvc_file
+
+    - name: Sanitize + apply PVC spec
+      when: pvc_file.stat.exists
+      block:
+        - ansible.builtin.command: >
+            python3 {{ remote_work_dir }}/sanitize_k8s_manifest.py
+            {{ remote_work_dir }}/{{ dr_app }}/pvc.yaml
+            {{ remote_work_dir }}/{{ dr_app }}/pvc.clean.yaml
+        - ansible.builtin.command: k3s kubectl apply -f {{ remote_work_dir }}/{{ dr_app }}/pvc.clean.yaml
+
+    - name: Sanitize + apply Secret
+      block:
+        - ansible.builtin.command: >
+            python3 {{ remote_work_dir }}/sanitize_k8s_manifest.py
+            {{ remote_work_dir }}/{{ dr_app }}/secret.yaml
+            {{ remote_work_dir }}/{{ dr_app }}/secret.clean.yaml
+        - ansible.builtin.command: k3s kubectl apply -f {{ remote_work_dir }}/{{ dr_app }}/secret.clean.yaml
+
+    - name: Sanitize + apply Deployments/Services/Ingress
+      block:
+        - ansible.builtin.command: >
+            python3 {{ remote_work_dir }}/sanitize_k8s_manifest.py
+            {{ remote_work_dir }}/{{ dr_app }}/manifests.yaml
+            {{ remote_work_dir }}/{{ dr_app }}/manifests.clean.yaml
+        - ansible.builtin.command: k3s kubectl apply -f {{ remote_work_dir }}/{{ dr_app }}/manifests.clean.yaml
+
+    - name: Wait for PVC(s) to be Bound
+      when: pvc_file.stat.exists
+      ansible.builtin.shell: >
+        k3s kubectl get pvc -n {{ dr_namespace }} -o jsonpath='{.items[*].status.phase}'
+      register: pvc_status
+      until: "'Pending' not in pvc_status.stdout"
+      retries: 12
+      delay: 5
+      changed_when: false
+
+    # ── 5. Data population — reuse the existing, already-validated
+    #      restore pipeline. It handles PVC contents + DB restore; steps
+    #      1-4 above are only what's needed to make its own assumption
+    #      (namespace/PVC/Deployments already exist) true on a target
+    #      that started with nothing. ──────────────────────────────────
+    - name: Run restore-k8s-apps.sh to populate data
+      ansible.builtin.command: bash restore-k8s-apps.sh --apps {{ dr_app }}
+      args:
+        chdir: "{{ remote_work_dir }}"
+      register: restore_output
+
+    - name: Show restore output
+      ansible.builtin.debug:
+        var: restore_output.stdout_lines

ansible/inventory.ini

@@ -0,0 +1,2 @@
+[dr_target]
+178.18.243.51 ansible_user=root ansible_ssh_private_key_file=/root/.ssh/id_rsa ansible_ssh_common_args='-o StrictHostKeyChecking=no'

ansible/sanitize_k8s_manifest.py

@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+"""sanitize_k8s_manifest.py — strip source-cluster-specific fields from a
+captured `kubectl get ... -o yaml` dump so it can be applied fresh onto a
+different (DR/standby) cluster.
+
+backup-k8s-apps.sh's manifests.yaml/pvc.yaml/secret.yaml are raw `kubectl
+get -o yaml` output: server-assigned fields like resourceVersion/uid are
+always present, and PVCs additionally carry spec.volumeName + a Bound
+status pointing at a specific PersistentVolume that only exists on the
+ORIGINAL cluster. Applying any of that verbatim onto a fresh cluster either
+fails outright (a stale resourceVersion trips optimistic-concurrency
+checks — the exact bug fixed in restore-k8s-apps.sh's Secret handling) or
+silently tries to bind a PVC to a PV that was never created there.
+
+Usage: sanitize_k8s_manifest.py <in.yaml> <out.yaml>
+"""
+import sys
+import yaml
+
+DROP_METADATA_KEYS = (
+    'resourceVersion', 'uid', 'creationTimestamp', 'selfLink',
+    'generation', 'finalizers', 'managedFields', 'annotations',
+)
+
+
+def sanitize_item(item):
+    meta = item.get('metadata') or {}
+    for key in DROP_METADATA_KEYS:
+        meta.pop(key, None)
+
+    item.pop('status', None)
+
+    if item.get('kind') == 'PersistentVolumeClaim':
+        spec = item.get('spec') or {}
+        spec.pop('volumeName', None)
+
+    return item
+
+
+def main():
+    if len(sys.argv) != 3:
+        print(f"usage: {sys.argv[0]} <in.yaml> <out.yaml>", file=sys.stderr)
+        sys.exit(1)
+
+    with open(sys.argv[1]) as f:
+        doc = yaml.safe_load(f)
+
+    if doc is None:
+        items = []
+    elif doc.get('kind') == 'List':
+        items = [sanitize_item(i) for i in doc.get('items', [])]
+    else:
+        items = [sanitize_item(doc)]
+
+    with open(sys.argv[2], 'w') as f:
+        yaml.safe_dump_all(items, f, default_flow_style=False)
+
+
+if __name__ == '__main__':
+    main()

backup/backup-k8s-apps.sh

@@ -96,6 +96,15 @@ declare -A SECRET_NAME=( [n8n]=n8n-secrets    [odoo]=odoo-secrets    [mautic]=ma
 declare -A DB_USER_KEY=( [n8n]=DB_POSTGRESDB_USER [odoo]=POSTGRES_USER [mautic]=MYSQL_USER [nextcloud]=POSTGRES_USER [frappe]="" )
 declare -A DB_PASS_KEY=( [n8n]=DB_POSTGRESDB_PASSWORD [odoo]=POSTGRES_PASSWORD [mautic]=MYSQL_ROOT_PASSWORD [nextcloud]=POSTGRES_PASSWORD [frappe]=MARIADB_ROOT_PASSWORD )
 
+# Reproducible framework/vendor code that lives on the same PVC as real user
+# data for these two apps (single shared mount, no separate volume). Both
+# images bake these dirs into their own filesystem layer independent of any
+# volume — Mautic has no logic to reconstruct them, so restore-k8s-apps.sh
+# re-seeds them straight from the app's own image instead of from backup.
+# Never pull request/exclude changes here without updating the matching
+# RECONSTRUCT_DIRS table in restore-k8s-apps.sh.
+declare -A TAR_EXCLUDES=( [mautic]="node_modules vendor" [nextcloud]="apps core dist 3rdparty lib" )
+
 ALL_APPS="n8n odoo mautic nextcloud frappe"
 
 BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
@@ -156,8 +165,14 @@ check_resources() {
     # management-platform pod's minimal image, /proc/meminfo always is.
     avail_mb=$(awk '/^MemAvailable:/{printf "%d", $2/1024}' /proc/meminfo)
     avail_gb=$(df -BG / | awk 'NR==2{gsub("G","",$4); print $4}')
-    if [ "$avail_mb" -lt 1536 ]; then
-        echo "   ❌ RAM available ${avail_mb}MB < 1536MB threshold — aborting."
+    # Threshold lowered 1536->1024MB 2026-08-21: this host's steady-state
+    # available RAM sits ~1.3-1.5GB even when idle (k3s + Jenkins + Docker
+    # monitoring/wazuh stack + dev tooling baseline) — investigated, no
+    # single runaway process, just genuine multi-service baseline. 1536MB
+    # was tripping on ordinary single-app backups. 1024MB still leaves
+    # real headroom above what a pg_dump/mysqldump/tar op actually needs.
+    if [ "$avail_mb" -lt 1024 ]; then
+        echo "   ❌ RAM available ${avail_mb}MB < 1024MB threshold — aborting."
         log_status "FAILED" "$BACKUP_NAME" "resource_safety_ram_${avail_mb}mb"
         exit 1
     fi
@@ -168,6 +183,83 @@ check_resources() {
     fi
 }
 
+# --------------------------------------------------
+# Per-backup status metadata sidecar (.meta.json) — powers the UI's
+# green/yellow/red status dot without ever needing to decompress the
+# (possibly 600MB+) archive itself. Written only after all 3 tiers are
+# known, then best-effort mirrored to VM/R2 alongside the archive, same
+# as the .sha256 sidecar already is — so status is readable from
+# whichever tier the UI is listing, not just local.
+# --------------------------------------------------
+write_meta_sidecar() {
+    local size_bytes meta_path
+    size_bytes=$(stat -c%s "$BACKUP_ARCHIVE" 2>/dev/null || echo 0)
+    meta_path="/root/backups/${BACKUP_NAME}.tar.gz.meta.json"
+
+    python3 - "$BACKUP_NAME" "$size_bytes" "$TIER1_STATUS" "$TIER2_STATUS" "$TIER3_STATUS" \
+        "${#INCLUDED_APPS[@]}" "${INCLUDED_APPS[@]}" "${ERRORS[@]}" > "$meta_path" <<'PYEOF'
+import sys, json, datetime
+
+backup_name  = sys.argv[1]
+size_bytes   = int(sys.argv[2])
+tier1, tier2, tier3 = sys.argv[3], sys.argv[4], sys.argv[5]
+n_apps       = int(sys.argv[6])
+apps         = sys.argv[7:7 + n_apps]
+errors_raw   = sys.argv[7 + n_apps:]
+
+def norm_tier(s):
+    # "ok:572M" -> "ok"; "failed:rc=2" / "skipped:no_credentials" pass through
+    # unchanged so the UI can tell a real failure apart from a deliberate skip.
+    return 'ok' if s.startswith('ok') else s
+
+errors = []
+for e in errors_raw:
+    parts = e.split(':', 2)
+    errors.append({
+        'app':    parts[0] if len(parts) > 0 else '',
+        'step':   parts[1] if len(parts) > 1 else '',
+        'detail': parts[2] if len(parts) > 2 else '',
+    })
+
+meta = {
+    'backup_name': backup_name,
+    'created_at':  datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds'),
+    'apps':        apps,
+    'apps_key':    ','.join(sorted(apps)),
+    'size_bytes':  size_bytes,
+    'errors':      errors,
+    'tiers': {'local': norm_tier(tier1), 'vm': norm_tier(tier2), 'r2': norm_tier(tier3)},
+}
+json.dump(meta, sys.stdout, indent=2)
+PYEOF
+
+    if [ -s "$meta_path" ]; then
+        echo "   ✅ Metadata sidecar written: $(basename "$meta_path")"
+    else
+        echo "   ⚠️  Metadata sidecar write failed"
+        return 1
+    fi
+
+    # Best-effort mirror to whichever tiers actually succeeded — never fatal,
+    # the local sidecar (already written above) is what matters most.
+    if [ "$TIER2_STATUS" = "ok" ]; then
+        scp -i "$VM_KEY" -P "$VM_PORT" -o StrictHostKeyChecking=no -o ConnectTimeout=15 \
+            "$meta_path" "${VM_USER}@${VM_HOST}:${VM_DEST}" &>/dev/null || true
+    fi
+    if [ "$TIER3_STATUS" = "ok" ] && [ -n "${AWS_ACCESS_KEY_ID:-}" ]; then
+        META_SCRIPT="/tmp/r2_meta_$$.py"
+        cat > "$META_SCRIPT" << PYEOF2
+import boto3
+client = boto3.client('s3', endpoint_url="${R2_ENDPOINT}",
+    aws_access_key_id="${AWS_ACCESS_KEY_ID}", aws_secret_access_key="${AWS_SECRET_ACCESS_KEY}",
+    region_name='auto')
+client.upload_file("$meta_path", "${R2_BUCKET}", "backups/${BACKUP_NAME}.tar.gz.meta.json")
+PYEOF2
+        "$PYTHON_BIN" "$META_SCRIPT" &>/dev/null || true
+        rm -f "$META_SCRIPT"
+    fi
+}
+
 echo "========================================="
 echo "📦 Starting k8s Backup: $BACKUP_NAME"
 if [ "$APPS_FILTER" = "all" ]; then
@@ -183,8 +275,11 @@ mkdir -p "$BACKUP_DIR" "/root/backups"
 cd "$BACKUP_DIR"
 
 APP_COUNT=0
+INCLUDED_APPS=()   # apps actually attempted this run — feeds the .meta.json sidecar's apps_key
+ERRORS=()          # "app:step:detail" entries — any entry here makes the sidecar's status red
 for app in $ALL_APPS; do
     app_selected "$app" || continue
+    INCLUDED_APPS+=("$app")
     ns="${NS[$app]}"
     app_deploy="${APP_DEPLOY[$app]}"
     app_ctr="${APP_CTR[$app]}"
@@ -208,13 +303,28 @@ for app in $ALL_APPS; do
     echo -n "   📄 Manifests (deployment/service/configmap/ingress) ... "
     kubectl get deployment,service,configmap,ingress -n "$ns" -l owner=ameni-boukattaya -o yaml \
         > "$APP_DIR/manifests.yaml" 2>"$APP_DIR/manifests.err" \
-        && { echo "✅"; rm -f "$APP_DIR/manifests.err"; } || echo "⚠️  FAILED (see manifests.err)"
+        && { echo "✅"; rm -f "$APP_DIR/manifests.err"; } \
+        || { echo "⚠️  FAILED (see manifests.err)"; ERRORS+=("$app:manifests:$(tr '\n' ' ' < "$APP_DIR/manifests.err" | cut -c1-200)"); }
+
+    # ---- 1b. PVC specs — NOT covered by the manifests capture above, and
+    #          needed to provision a namespace from scratch on a fresh/DR
+    #          cluster (restore-k8s-apps.sh assumes the PVC already exists;
+    #          this is what actually creates it there). Same label selector
+    #          as the manifests capture, so this stays in lockstep with
+    #          whatever PVCs are actually labeled as ours — no separate
+    #          per-app name table to keep in sync. ----
+    echo -n "   💾 PVC specs ... "
+    kubectl get pvc -n "$ns" -l owner=ameni-boukattaya -o yaml \
+        > "$APP_DIR/pvc.yaml" 2>"$APP_DIR/pvc.err" \
+        && { echo "✅"; rm -f "$APP_DIR/pvc.err"; } \
+        || { echo "⚠️  FAILED (see pvc.err)"; ERRORS+=("$app:pvc_spec:$(tr '\n' ' ' < "$APP_DIR/pvc.err" | cut -c1-200)"); }
 
     # ---- 2. Secret (plaintext, encryption-at-rest deferred) ----
     echo -n "   🔑 Secret ($secret_name) ... "
     kubectl get secret "$secret_name" -n "$ns" -o yaml \
         > "$APP_DIR/secret.yaml" 2>"$APP_DIR/secret.err" \
-        && { echo "✅"; rm -f "$APP_DIR/secret.err"; } || echo "⚠️  FAILED (see secret.err)"
+        && { echo "✅"; rm -f "$APP_DIR/secret.err"; } \
+        || { echo "⚠️  FAILED (see secret.err)"; ERRORS+=("$app:secret:$(tr '\n' ' ' < "$APP_DIR/secret.err" | cut -c1-200)"); }
 
     # ---- 3. DB dump via kubectl exec (never raw datadir copy) ----
     echo -n "   💾 DB dump ($db_engine) ... "
@@ -232,9 +342,11 @@ for app in $ALL_APPS; do
                     rm -f "$APP_DIR/db-dump.err"
                 else
                     echo "⚠️  FAILED (rc=$pg_dump_rc, see db-dump.err)"
+                    ERRORS+=("$app:db_dump:rc=$pg_dump_rc $(tr '\n' ' ' < "$APP_DIR/db-dump.err" 2>/dev/null | cut -c1-160)")
                 fi
             else
                 echo "⚠️  could not read DB credentials from secret"
+                ERRORS+=("$app:db_dump:could not read DB credentials from secret")
             fi
             ;;
         mariadb)
@@ -260,24 +372,33 @@ for app in $ALL_APPS; do
                     rm -f "$APP_DIR/db-dump.err"
                 else
                     echo "⚠️  FAILED (rc=$mysqldump_rc, see db-dump.err)"
+                    ERRORS+=("$app:db_dump:rc=$mysqldump_rc $(tr '\n' ' ' < "$APP_DIR/db-dump.err" 2>/dev/null | cut -c1-160)")
                 fi
             else
                 echo "⚠️  could not determine DB name/credentials"
+                ERRORS+=("$app:db_dump:could not determine DB name/credentials")
             fi
             ;;
     esac
 
     # ---- 4. App-data PVC contents via kubectl exec (live tar, app-data only,
     #         never raw filesystem/PVC-path access — §5 sandbox-divergence rule) ----
-    echo -n "   📦 PVC data ($pvc_path) ... "
+    exclude_args=()
+    if [ -n "${TAR_EXCLUDES[$app]:-}" ]; then
+        for _excl_dir in ${TAR_EXCLUDES[$app]}; do
+            exclude_args+=(--exclude="./$_excl_dir")
+        done
+    fi
+    echo -n "   📦 PVC data ($pvc_path${exclude_args:+, excl: ${TAR_EXCLUDES[$app]}}) ... "
     kubectl exec -n "$ns" "deploy/${app_deploy}" -c "$app_ctr" -- \
-        tar czf - -C "$pvc_path" . 2>"$APP_DIR/pvc-data.err" \
+        tar czf - "${exclude_args[@]}" -C "$pvc_path" . 2>"$APP_DIR/pvc-data.err" \
         > "$APP_DIR/pvc-data.tar.gz"
     if [ -s "$APP_DIR/pvc-data.tar.gz" ]; then
         echo "✅ ($(du -h "$APP_DIR/pvc-data.tar.gz" | cut -f1))"
         rm -f "$APP_DIR/pvc-data.err"
     else
         echo "⚠️  FAILED (see pvc-data.err)"
+        ERRORS+=("$app:pvc_data:$(tr '\n' ' ' < "$APP_DIR/pvc-data.err" 2>/dev/null | cut -c1-160)")
     fi
 
     APP_COUNT=$((APP_COUNT + 1))
@@ -348,7 +469,7 @@ if [ "$ARCHIVE_COUNT" -gt "$MAX_BACKUPS" ]; then
     TO_DELETE=$(echo "$ARCHIVE_LIST" | tail -n +$((MAX_BACKUPS + 1)))
     while IFS= read -r old_file; do
         [ -z "$old_file" ] && continue
-        rm -f "$old_file" "${old_file}.sha256"
+        rm -f "$old_file" "${old_file}.sha256" "${old_file}.meta.json"
         echo "   🗑️  Deleted: $(basename "$old_file")"
     done <<< "$TO_DELETE"
 else
@@ -374,6 +495,16 @@ if [ $VM_SCP_RC -eq 0 ]; then
         "${BACKUP_NAME}.tar.gz.sha256" \
         "${VM_USER}@${VM_HOST}:${VM_DEST}" 2>/dev/null || true
     TIER2_STATUS="ok"
+
+    # VM-side retention — this tier never had any pruning at all until
+    # found accumulating 85 legacy pre-k8s archives (131GB) untouched
+    # since May. Mirrors the local [TIER 1] policy above: same
+    # MAX_BACKUPS, own glob/lineage only (never touches other
+    # unrelated backup dirs also living on this VM).
+    ssh -i "$VM_KEY" -p "$VM_PORT" -o StrictHostKeyChecking=no -o ConnectTimeout=15 \
+        "${VM_USER}@${VM_HOST}" \
+        "cd ${VM_DEST} && ls -t myapps-k8s-backup-*.tar.gz 2>/dev/null | tail -n +$((MAX_BACKUPS + 1)) | xargs -r -I{} rm -f {} {}.sha256 {}.meta.json" \
+        &>/dev/null || true
 else
     echo "   ⚠️  VM transfer failed (rc=$VM_SCP_RC) — local backup is safe, continuing..."
     TIER2_STATUS="failed:rc=${VM_SCP_RC}"
@@ -463,10 +594,11 @@ try:
     for obj in to_delete:
         old_key = obj['Key']
         client.delete_object(Bucket=bucket, Key=old_key)
-        try:
-            client.delete_object(Bucket=bucket, Key=old_key + '.sha256')
-        except Exception:
-            pass
+        for suffix in ('.sha256', '.meta.json'):
+            try:
+                client.delete_object(Bucket=bucket, Key=old_key + suffix)
+            except Exception:
+                pass
         print(f'   🗑️  Deleted from R2: {old_key.replace("backups/", "")}')
 
     sys.exit(0)
@@ -489,6 +621,13 @@ PYEOF
     fi
 fi
 
+# --------------------------------------------------
+# Status metadata sidecar — after all 3 tiers are known
+# --------------------------------------------------
+echo ""
+echo "📝 Writing status metadata sidecar..."
+write_meta_sidecar
+
 # --------------------------------------------------
 # Final summary + log
 # --------------------------------------------------

backup/restore-k8s-apps.sh

@@ -103,6 +103,38 @@ declare -A DB_USER_KEY=( [n8n]=DB_POSTGRESDB_USER [odoo]=POSTGRES_USER [mautic]=
 declare -A DB_PASS_KEY=( [n8n]=DB_POSTGRESDB_PASSWORD [odoo]=POSTGRES_PASSWORD [mautic]=MYSQL_ROOT_PASSWORD [nextcloud]=POSTGRES_PASSWORD [frappe]=MARIADB_ROOT_PASSWORD )
 declare -A DOMAIN=(      [n8n]="n8nwf.nav.ovh" [odoo]="odooo.nav.ovh" [mautic]="mautics.nav.ovh" [nextcloud]="next.cloud.nav.ovh" [frappe]="erpnext.navitrends.ovh" )
 
+# Dirs backup-k8s-apps.sh excludes from pvc-data.tar.gz for these two apps
+# (reproducible framework/vendor code, not user data — see matching
+# TAR_EXCLUDES table + comment there). They must be re-seeded here from the
+# app's own live image, NOT reconstructed by the container entrypoint:
+#   - Mautic's image has no logic at all to repopulate node_modules/vendor
+#     if missing — confirmed empty by reading /entrypoint.sh.
+#   - Nextcloud's entrypoint only rsyncs from /usr/src/nextcloud when it
+#     detects image_version > installed_version (via version.php). Since
+#     version.php itself is NOT excluded (kept, "everything else"), a
+#     restored version.php already matches the running image's version, so
+#     that rsync path never fires — relying on it would silently leave
+#     apps/core/dist/3rdparty/lib missing after restore.
+# Both images bake these dirs into their own filesystem layer independent
+# of any volume mount, so cp'ing from a loader pod running the same image
+# (PVC mounted elsewhere, so it doesn't shadow the image's own copy) is
+# reliable regardless of either app's startup logic.
+#
+# RECONSTRUCT_SRC is the path *inside the image* to copy from — NOT
+# necessarily pvc_path. Verified per app by running the actual image with no
+# volume mounted and checking where it bakes the dirs in:
+#   - Mautic bakes node_modules/vendor directly at /var/www/html (its own
+#     PVC mount path) — same path, source == destination parent.
+#   - Nextcloud's image keeps its shipped source tree at /usr/src/nextcloud
+#     (confirmed via `du -sh` in a bare nextcloud:32 pod), completely
+#     separate from /var/www/html (the PVC mount, empty at that path in the
+#     image). Assuming pvc_path here (mirroring Mautic's layout) was wrong
+#     and failed loudly in testing — cp: cannot stat '/var/www/html/apps' —
+#     caught by the reseed_failed gate above rather than silently booting
+#     nextcloud broken.
+declare -A RECONSTRUCT_DIRS=( [mautic]="node_modules vendor" [nextcloud]="apps core dist 3rdparty lib" )
+declare -A RECONSTRUCT_SRC=(  [mautic]="/var/www/html"       [nextcloud]="/usr/src/nextcloud" )
+
 ALL_APPS="n8n odoo mautic nextcloud frappe"
 LOADER_IMAGE="alpine:3.20"
 
@@ -122,7 +154,10 @@ check_resources() {
     # management-platform pod's minimal image, /proc/meminfo always is.
     avail_mb=$(awk '/^MemAvailable:/{printf "%d", $2/1024}' /proc/meminfo)
     avail_gb=$(df -BG / | awk 'NR==2{gsub("G","",$4); print $4}')
-    if [ "$avail_mb" -lt 1536 ] || [ "$avail_gb" -lt 10 ]; then
+    # Threshold lowered 1536->1024MB 2026-08-21 — see backup-k8s-apps.sh
+    # check_resources() comment for the investigation (baseline host RAM,
+    # not a leak).
+    if [ "$avail_mb" -lt 1024 ] || [ "$avail_gb" -lt 10 ]; then
         echo "   ❌ Resource safety threshold hit (RAM=${avail_mb}MB, Disk=${avail_gb}GB) — aborting."
         exit 1
     fi
@@ -165,7 +200,18 @@ for app in $ALL_APPS; do
     #         restored here already exists in a same-cluster restore) ----
     if [ -f "$APP_DIR/secret.yaml" ]; then
         echo -n "   🔑 Applying Secret ... "
-        kubectl apply -f "$APP_DIR/secret.yaml" &>/dev/null && echo "✅" || { echo "⚠️  FAILED"; app_ok=false; }
+        # secret.yaml is a raw `kubectl get -o yaml` dump, which includes the
+        # live resourceVersion/uid/creationTimestamp at capture time. Passing
+        # a stale resourceVersion to `apply` trips optimistic-concurrency
+        # control ("the object has been modified") on any restore where the
+        # secret's real resourceVersion has since moved on — which is every
+        # restore, since normal cluster activity (this backup pipeline's own
+        # earlier applies included) bumps it constantly. Strip the volatile
+        # fields before applying; they're server-assigned and never meant to
+        # round-trip through a backup.
+        sed -e '/^  resourceVersion:/d' -e '/^  uid:/d' -e '/^  creationTimestamp:/d' \
+            "$APP_DIR/secret.yaml" > "$APP_DIR/secret.apply.yaml"
+        kubectl apply -f "$APP_DIR/secret.apply.yaml" &>/dev/null && echo "✅" || { echo "⚠️  FAILED"; app_ok=false; }
     fi
     # NOTE: manifests.yaml (Deployment/Service/ConfigMap/Ingress specs) is
     # captured by backup-k8s-apps.sh but deliberately NOT re-applied here.
@@ -213,9 +259,68 @@ for app in $ALL_APPS; do
         echo "   ⏭️  No pvc-data.tar.gz in backup — skipping PVC restore"
     fi
 
+    # ---- 3b. Re-seed reproducible dirs excluded from backup (see
+    #          RECONSTRUCT_DIRS comment above) from the app's own live image,
+    #          never from the backup archive itself ----
+    reseed_failed=false
+    if [ -n "${RECONSTRUCT_DIRS[$app]:-}" ] && kubectl get pvc "$pvc_name" -n "$ns" &>/dev/null; then
+        app_image=$(kubectl get deployment "$app_deploy" -n "$ns" -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null)
+        if [ -n "$app_image" ]; then
+            seed_loader="${app}-reseed-loader"
+            echo -n "   🧩 Re-seeding [${RECONSTRUCT_DIRS[$app]}] from $app_image ... "
+            # imagePullPolicy IfNotPresent: $app_image is read straight off the
+            # currently-running Deployment, so it's already on this node —
+            # forcing a re-pull (the ":latest"/floating-tag default) just adds
+            # a slow, needless network fetch during every restore.
+            kubectl run "$seed_loader" -n "$ns" --image="$app_image" --restart=Never \
+                --overrides="{\"spec\":{\"containers\":[{\"name\":\"loader\",\"image\":\"${app_image}\",\"imagePullPolicy\":\"IfNotPresent\",\"command\":[\"sleep\",\"3600\"],\"volumeMounts\":[{\"name\":\"data\",\"mountPath\":\"/restore\"}]}],\"volumes\":[{\"name\":\"data\",\"persistentVolumeClaim\":{\"claimName\":\"${pvc_name}\"}}]}}" \
+                &>/dev/null
+            # 240s, not 60/90s like the alpine loader above: this pulls the
+            # full app image (600MB+ for mautic/nextcloud) if it isn't cached
+            # on this node yet — a 90s timeout was observed to fire mid-pull
+            # in testing, silently leaving the app pod without its
+            # reconstructed dirs after being scaled back up.
+            if kubectl wait --for=condition=Ready "pod/$seed_loader" -n "$ns" --timeout=240s &>/dev/null; then
+                seed_ok=true
+                : > "$APP_DIR/reseed.err"
+                seed_src="${RECONSTRUCT_SRC[$app]}"
+                for _seed_dir in ${RECONSTRUCT_DIRS[$app]}; do
+                    kubectl exec -n "$ns" "$seed_loader" -- sh -c "rm -rf '/restore/$_seed_dir' && cp -a '${seed_src}/${_seed_dir}' '/restore/$_seed_dir'" \
+                        >>"$APP_DIR/reseed.err" 2>&1 || seed_ok=false
+                done
+                if $seed_ok; then
+                    echo "✅"
+                    rm -f "$APP_DIR/reseed.err"
+                else
+                    echo "⚠️  FAILED (see $APP_DIR/reseed.err)"
+                    app_ok=false
+                    reseed_failed=true
+                fi
+            else
+                echo "⚠️  loader pod not ready"
+                app_ok=false
+                reseed_failed=true
+            fi
+            kubectl delete pod "$seed_loader" -n "$ns" --wait=false &>/dev/null
+        else
+            echo "   ⚠️  Could not determine $app image — skipping re-seed of [${RECONSTRUCT_DIRS[$app]}]"
+            app_ok=false
+            reseed_failed=true
+        fi
+    fi
+
     # ---- 4. Scale app back up ----
-    echo -n "   ▶️  Scaling $app_deploy back to $prior_replicas ... "
-    kubectl scale deployment "$app_deploy" -n "$ns" --replicas="$prior_replicas" &>/dev/null && echo "✅" || echo "⚠️  FAILED"
+    # Never bring the app back with replicas>0 if a required re-seed (3b)
+    # failed — for Mautic/Nextcloud that means booting with vendor/ or
+    # core/lib missing, which is a hard crash/broken-app state, not a
+    # degraded one. Left at 0 so the operator fixes it and scales up
+    # manually rather than the restore silently serving a broken app.
+    if $reseed_failed; then
+        echo "   ⛔ Skipping scale-up — required re-seed failed (app left at 0 replicas, see above)"
+    else
+        echo -n "   ▶️  Scaling $app_deploy back to $prior_replicas ... "
+        kubectl scale deployment "$app_deploy" -n "$ns" --replicas="$prior_replicas" &>/dev/null && echo "✅" || echo "⚠️  FAILED"
+    fi
 
     # ---- 5. DB restore (SQL-level, never raw datadir copy) ----
     if [ -f "$APP_DIR/db-dump.sql.gz" ]; then

k8s/monitoring/README.md

@@ -0,0 +1,72 @@
+# Monitoring stack on k8s (VM: 178.18.243.51)
+
+Migrated from docker-compose (`/root/monitoring`, `/root/CI-CD/sonarqube` on
+the VM — left `docker compose stop`'d, not removed, as a rollback path) into
+the VM's own k3s cluster, with real data migrated (not a fresh start):
+Prometheus TSDB, Grafana dashboards/DB, SonarQube data+DB (confirmed intact:
+the real `management-platform` SonarQube project and Grafana's existing
+admin password both survived).
+
+## Apply order
+```
+kubectl apply -f monitoring-pvcs.yaml
+# migrate data into the PVCs via a loader pod per volume (see chat history —
+# not scripted, was a one-time manual migration)
+kubectl apply -f monitoring-apps.yaml
+kubectl apply -f monitoring-ingress.yaml
+```
+
+## Domains — READ THIS BEFORE TOUCHING DNS
+- `grafanna.nav.ovh` (note: **two Ns** — `grafana.nav.ovh` without the typo
+  is a DIFFERENT, unrelated, pre-existing server at 109.199.127.74. Do not
+  point anything at plain `grafana.nav.ovh` or touch that record.)
+- `sonar.nav.ovh`
+- `prom.nav.ovh`
+
+All three already existed as A records pointing at 178.18.243.51 before this
+work; only `nginx-vhost.conf` + a Let's Encrypt cert (via certbot, HTTP-01,
+auto-renews) make them actually resolve to something.
+
+## Why nginx is in front of Traefik, not Traefik directly on 80/443
+This VM already runs an unrelated app ("nqks") on nginx, owning ports 80/443.
+k3s's bundled Traefik defaults to a LoadBalancer Service, which on a
+single-node k3s claims host 80/443 via iptables DNAT (klipper-lb) —
+this silently hijacked nqks's traffic the first time (found and fixed
+during this session). Traefik's Service is patched to ClusterIP-only via
+a HelmChartConfig (`kubectl get helmchartconfig -n kube-system traefik`);
+nginx reverse-proxies the 3 domains to Traefik's ClusterIP instead, and
+handles TLS + BasicAuth itself.
+
+## Access control
+- Browser access to all 3 domains: HTTP Basic Auth (`/etc/nginx/.htpasswd-monitoring`
+  on the VM, user `admin`) — ask whoever ran this migration for the password,
+  it's not in git.
+- **SonarQube is also reachable directly at `178.18.243.51:9000`, no auth** —
+  this is deliberate, not an oversight. Jenkins' SonarQube server config
+  (main cluster, `hudson.plugins.sonar.SonarGlobalConfiguration.xml`) is
+  hardcoded to that exact address and authenticates via its own token, not
+  BasicAuth (which would break the scanner). `sonarqube-jenkins-proxy.service`
+  (systemd, socat) forwards host:9000 -> the sonarqube Service's ClusterIP;
+  ufw restricts port 9000 to the main server's IP only, not the open
+  internet. Do not add BasicAuth in front of this specific path.
+  - Fragility note: the socat unit hardcodes the Service's ClusterIP. If the
+    `sonarqube` Service in the `monitoring` namespace is ever deleted and
+    recreated (not just the pod — pod recreates keep the same ClusterIP),
+    update the `TCP:<ip>:9000` target in
+    `/etc/systemd/system/sonarqube-jenkins-proxy.service` on the VM to match.
+
+## Gotchas hit building this (context for next time)
+- Don't add a k8s Service of type LoadBalancer on this cluster without first
+  patching it to ClusterIP/setting a HelmChartConfig — same hijack risk as
+  Traefik's default.
+- Don't add `hostPort` to a pod that also needs normal in-cluster (Service/
+  Ingress) traffic to reach it — it breaks pod-to-pod routing to that pod
+  (confirmed directly: Traefik -> sonarqube hung/504'd the whole time
+  hostPort was present, while direct host-to-pod and host-to-ClusterIP both
+  worked fine — the asymmetry was the tell). Use a host-level proxy (socat)
+  or a NodePort in the 30000-32767 range instead.
+- SonarQube's DB was mid-upgrade (`DB_MIGRATION_NEEDED`) after the version
+  jump — needed one `POST /api/system/migrate_db` (admin/admin was already
+  changed on the real instance; used the actual admin session) — and the
+  post-migration rule re-registration OOM'd on the compose file's original
+  256MB web heap; bumped `SONAR_WEB_JAVAADDITIONALOPTS` to `-Xms512m -Xmx1536m`.

k8s/monitoring/monitoring-apps.yaml

@@ -0,0 +1,203 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: prometheus-config
+  namespace: monitoring
+data:
+  prometheus.yml: |
+    global:
+      scrape_interval: 15s
+      evaluation_interval: 15s
+
+    scrape_configs:
+
+      - job_name: 'server1-node'
+        static_configs:
+          - targets: ['173.249.20.244:9100']
+            labels:
+              server: 'server1'
+
+      - job_name: 'my-cadvisor'
+        static_configs:
+          - targets: ['173.249.20.244:8094']
+            labels:
+              server: 'server1'
+              owner: 'ameni'
+
+      - job_name: 'server1-blackbox'
+        static_configs:
+          - targets: ['173.249.20.244:9115']
+            labels:
+              server: 'server1'
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: prometheus
+  namespace: monitoring
+spec:
+  replicas: 1
+  selector:
+    matchLabels: {app: prometheus}
+  strategy: {type: Recreate}
+  template:
+    metadata:
+      labels: {app: prometheus}
+    spec:
+      securityContext:
+        fsGroup: 65534
+      containers:
+        - name: prometheus
+          image: prom/prometheus:latest
+          args:
+            - --config.file=/etc/prometheus/prometheus.yml
+            - --storage.tsdb.retention.time=15d
+            - --storage.tsdb.path=/prometheus
+          ports: [{containerPort: 9090}]
+          resources:
+            limits: {memory: 512Mi}
+          volumeMounts:
+            - {name: config, mountPath: /etc/prometheus/prometheus.yml, subPath: prometheus.yml}
+            - {name: data, mountPath: /prometheus}
+      volumes:
+        - name: config
+          configMap: {name: prometheus-config}
+        - name: data
+          persistentVolumeClaim: {claimName: prometheus-data}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: prometheus
+  namespace: monitoring
+spec:
+  selector: {app: prometheus}
+  ports: [{port: 9090, targetPort: 9090}]
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: grafana
+  namespace: monitoring
+spec:
+  replicas: 1
+  selector:
+    matchLabels: {app: grafana}
+  strategy: {type: Recreate}
+  template:
+    metadata:
+      labels: {app: grafana}
+    spec:
+      securityContext:
+        fsGroup: 472
+      containers:
+        - name: grafana
+          image: grafana/grafana:latest
+          env:
+            - {name: GF_SECURITY_ADMIN_PASSWORD, value: admin}
+          ports: [{containerPort: 3000}]
+          resources:
+            limits: {memory: 256Mi}
+          volumeMounts:
+            - {name: data, mountPath: /var/lib/grafana}
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: grafana-data}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: grafana
+  namespace: monitoring
+spec:
+  selector: {app: grafana}
+  ports: [{port: 3000, targetPort: 3000}]
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: sonarqube-db
+  namespace: monitoring
+spec:
+  replicas: 1
+  selector:
+    matchLabels: {app: sonarqube-db}
+  strategy: {type: Recreate}
+  template:
+    metadata:
+      labels: {app: sonarqube-db}
+    spec:
+      securityContext:
+        fsGroup: 999
+      containers:
+        - name: sonarqube-db
+          image: postgres:15
+          env:
+            - {name: POSTGRES_USER, value: sonar}
+            - {name: POSTGRES_PASSWORD, value: sonar}
+            - {name: POSTGRES_DB, value: sonar}
+          ports: [{containerPort: 5432}]
+          volumeMounts:
+            - {name: data, mountPath: /var/lib/postgresql/data}
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: sonarqube-db-data}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: sonarqube-db
+  namespace: monitoring
+spec:
+  selector: {app: sonarqube-db}
+  ports: [{port: 5432, targetPort: 5432}]
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: sonarqube
+  namespace: monitoring
+spec:
+  replicas: 1
+  selector:
+    matchLabels: {app: sonarqube}
+  strategy: {type: Recreate}
+  template:
+    metadata:
+      labels: {app: sonarqube}
+    spec:
+      securityContext:
+        fsGroup: 1000
+      containers:
+        - name: sonarqube
+          image: sonarqube:community
+          env:
+            - {name: SONAR_ES_BOOTSTRAP_CHECKS_DISABLE, value: "true"}
+            - {name: SONAR_JDBC_URL, value: "jdbc:postgresql://sonarqube-db:5432/sonar"}
+            - {name: SONAR_JDBC_USERNAME, value: sonar}
+            - {name: SONAR_JDBC_PASSWORD, value: sonar}
+            - {name: SONAR_SEARCH_JAVAADDITIONALOPTS, value: "-Xms512m -Xmx512m"}
+            - {name: SONAR_WEB_JAVAADDITIONALOPTS, value: "-Xms512m -Xmx1536m"}
+          ports: [{containerPort: 9000}]
+          resources:
+            limits: {memory: 3Gi}
+          volumeMounts:
+            - {name: data, mountPath: /opt/sonarqube/data}
+            - {name: logs, mountPath: /opt/sonarqube/logs}
+            - {name: extensions, mountPath: /opt/sonarqube/extensions}
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: sonarqube-data}
+        - name: logs
+          persistentVolumeClaim: {claimName: sonarqube-logs}
+        - name: extensions
+          persistentVolumeClaim: {claimName: sonarqube-extensions}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: sonarqube
+  namespace: monitoring
+spec:
+  selector: {app: sonarqube}
+  ports: [{port: 9000, targetPort: 9000}]

k8s/monitoring/monitoring-ingress.yaml

@@ -0,0 +1,53 @@
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: grafana
+  namespace: monitoring
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: grafana.nav.ovh
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: grafana
+                port: {number: 3000}
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: sonarqube
+  namespace: monitoring
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: sonar.nav.ovh
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: sonarqube
+                port: {number: 9000}
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: prometheus
+  namespace: monitoring
+spec:
+  ingressClassName: traefik
+  rules:
+    - host: prom.nav.ovh
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: prometheus
+                port: {number: 9090}

k8s/monitoring/monitoring-pvcs.yaml

@@ -0,0 +1,76 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: monitoring
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: prometheus-data
+  namespace: monitoring
+spec:
+  accessModes: ["ReadWriteOnce"]
+  resources:
+    requests:
+      storage: 3Gi
+  storageClassName: local-path
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: grafana-data
+  namespace: monitoring
+spec:
+  accessModes: ["ReadWriteOnce"]
+  resources:
+    requests:
+      storage: 1Gi
+  storageClassName: local-path
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: sonarqube-data
+  namespace: monitoring
+spec:
+  accessModes: ["ReadWriteOnce"]
+  resources:
+    requests:
+      storage: 2Gi
+  storageClassName: local-path
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: sonarqube-logs
+  namespace: monitoring
+spec:
+  accessModes: ["ReadWriteOnce"]
+  resources:
+    requests:
+      storage: 1Gi
+  storageClassName: local-path
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: sonarqube-extensions
+  namespace: monitoring
+spec:
+  accessModes: ["ReadWriteOnce"]
+  resources:
+    requests:
+      storage: 1Gi
+  storageClassName: local-path
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: sonarqube-db-data
+  namespace: monitoring
+spec:
+  accessModes: ["ReadWriteOnce"]
+  resources:
+    requests:
+      storage: 1Gi
+  storageClassName: local-path

k8s/monitoring/nginx-vhost.conf

@@ -0,0 +1,102 @@
+server {
+    server_name grafanna.nav.ovh;
+    location / {
+        auth_basic "Restricted";
+        auth_basic_user_file /etc/nginx/.htpasswd-monitoring;
+        proxy_pass http://10.43.200.43;
+        proxy_set_header Host grafanna.nav.ovh;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_http_version 1.1;
+        proxy_set_header Upgrade $http_upgrade;
+        proxy_set_header Connection "upgrade";
+    }
+
+    listen 443 ssl; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/grafanna.nav.ovh/fullchain.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/grafanna.nav.ovh/privkey.pem; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
+
+}
+
+server {
+    server_name sonar.nav.ovh;
+    location / {
+        auth_basic "Restricted";
+        auth_basic_user_file /etc/nginx/.htpasswd-monitoring;
+        proxy_pass http://10.43.200.43;
+        proxy_set_header Host sonar.nav.ovh;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_http_version 1.1;
+    }
+
+    listen 443 ssl; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/grafanna.nav.ovh/fullchain.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/grafanna.nav.ovh/privkey.pem; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
+
+}
+
+server {
+    server_name prom.nav.ovh;
+    location / {
+        auth_basic "Restricted";
+        auth_basic_user_file /etc/nginx/.htpasswd-monitoring;
+        proxy_pass http://10.43.200.43;
+        proxy_set_header Host prom.nav.ovh;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+        proxy_http_version 1.1;
+    }
+
+    listen 443 ssl; # managed by Certbot
+    ssl_certificate /etc/letsencrypt/live/grafanna.nav.ovh/fullchain.pem; # managed by Certbot
+    ssl_certificate_key /etc/letsencrypt/live/grafanna.nav.ovh/privkey.pem; # managed by Certbot
+    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
+    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
+
+}
+server {
+    if ($host = grafanna.nav.ovh) {
+        return 301 https://$host$request_uri;
+    } # managed by Certbot
+
+
+    listen 80;
+    server_name grafanna.nav.ovh;
+    return 404; # managed by Certbot
+
+
+}
+
+server {
+    if ($host = sonar.nav.ovh) {
+        return 301 https://$host$request_uri;
+    } # managed by Certbot
+
+
+    listen 80;
+    server_name sonar.nav.ovh;
+    return 404; # managed by Certbot
+
+
+}
+
+server {
+    if ($host = prom.nav.ovh) {
+        return 301 https://$host$request_uri;
+    } # managed by Certbot
+
+
+    listen 80;
+    server_name prom.nav.ovh;
+    return 404; # managed by Certbot
+
+
+}

k8s/monitoring/sonarqube-jenkins-proxy.service

@@ -0,0 +1,13 @@
+[Unit]
+Description=TCP proxy: host:9000 -> k8s sonarqube Service ClusterIP:9000 (for Jenkins, which expects this exact address)
+After=k3s.service
+Wants=k3s.service
+
+[Service]
+Type=simple
+ExecStart=/usr/bin/socat TCP-LISTEN:9000,fork,reuseaddr TCP:10.43.4.19:9000
+Restart=always
+RestartSec=5
+
+[Install]
+WantedBy=multi-user.target

management-platform-sites-rbac.yaml

@@ -0,0 +1,32 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: management-platform-ingress-reader-role
+  labels: {owner: ameni-boukattaya, app.kubernetes.io/managed-by: navitrends-pfe-migration}
+rules:
+  - apiGroups: ["networking.k8s.io"]
+    resources: ["ingresses"]
+    verbs: ["get", "list", "watch"]
+---
+# Odoo's and Nextcloud's Ingress objects live in the `default` namespace
+# (historical — created there during the original Docker->k8s migration,
+# unlike n8n/mautic/erpnext whose Ingress lives in their own app namespace).
+# management-platform-sa has no binding at all in `default` otherwise, and
+# `default` also hosts unrelated live services (cloudops-ingress, jenkins-
+# ingress, stage-navitrends-ingress, etc.) — so this grants ONLY ingress
+# read there, not the full backup ClusterRole, to keep blast radius to
+# exactly what the Application Sites page needs (real domain/TLS lookup).
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: management-platform-ingress-reader-binding
+  namespace: default
+  labels: {owner: ameni-boukattaya, app.kubernetes.io/managed-by: navitrends-pfe-migration}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: management-platform-ingress-reader-role
+subjects:
+  - kind: ServiceAccount
+    name: management-platform-sa
+    namespace: management-platform

platform/Jenkinsfile

@@ -14,20 +14,11 @@ pipeline {
     stages {
         stage('Checkout') {
             steps {
-                retry(5) {
-                    checkout([
-                        $class: 'GitSCM',
-                        branches: scm.branches,
-                        userRemoteConfigs: scm.userRemoteConfigs,
-                        extensions: [[
-                            $class: 'CloneOption',
-                            shallow: true,
-                            depth: 1,
-                            noTags: true,
-                            timeout: 30
-                        ]]
-                    ])
-                }
+                sh '''
+                    echo "Copying CloudOps from the VPS's own checkout (hostPath-mounted, no Gitea fetch)..."
+                    cp -r /root/CloudOps/. .
+                    test -f platform/app.py
+                '''
             }
         }
 
@@ -44,15 +35,24 @@ pipeline {
         stage('SonarQube Analysis') {
             steps {
                 container('sonar-scanner') {
-                    withSonarQubeEnv('sonarqube') {
-                        script {
-                            def scannerHome = tool 'sonar-scanner'
-                            sh """
-                                ${scannerHome}/bin/sonar-scanner \
-                                  -Dsonar.projectKey=management-platform \
-                                  -Dsonar.sources=platform \
-                                  -Dsonar.python.version=3
-                            """
+                    // Analysis quality, not a deploy gate: SonarQube being
+                    // unreachable (moved server, migration in progress,
+                    // temporary outage) shouldn't block shipping an
+                    // otherwise-good build. catchError marks the build
+                    // UNSTABLE (visible in the UI, doesn't need separate
+                    // watching) and lets the pipeline continue to Deploy
+                    // instead of failing the whole run.
+                    catchError(buildResult: 'UNSTABLE', stageResult: 'FAILURE') {
+                        withSonarQubeEnv('sonarqube') {
+                            script {
+                                def scannerHome = tool 'sonar-scanner'
+                                sh """
+                                    ${scannerHome}/bin/sonar-scanner \
+                                      -Dsonar.projectKey=management-platform \
+                                      -Dsonar.sources=platform \
+                                      -Dsonar.python.version=3
+                                """
+                            }
                         }
                     }
                 }

platform/app.py

@@ -3,6 +3,7 @@ from flask import Flask, render_template, request, redirect, url_for, session, j
 import os
 import re
 import subprocess
+import sys
 import threading
 import uuid
 import time
@@ -11,12 +12,14 @@ from datetime import datetime, timezone
 from config import (
     MAIN_SERVER_IP, RUNNING_ON_MAIN_SERVER,
     VM_HOST, VM_PORT, VM_KEY, VM_USER,
-    MAIN_SERVER_KEY, MAIN_SERVER_PORT, MAIN_SERVER_USER,
+    MAIN_SERVER_KEY, MAIN_SERVER_USER,
+    MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT,
 )
 from modules.auth import login_required
 from modules.backups import (
     get_containers, get_all_root_containers,
     get_local_backups, get_vm_backups,
+    get_local_backups_with_status, get_vm_backups_with_status,
     get_all_stats, get_system_info,
     get_rootless_user_containers_remote,
     container_action, get_container_status,
@@ -67,11 +70,11 @@ def _stream_restore(job_id, cmd):
         restore_jobs[job_id]['status'] = 'error'
 
 
-def _stream_backup(job_id, script_path):
+def _stream_backup(job_id, script_path, extra_args=None):
     backup_jobs[job_id] = {'status': 'running', 'log': [], 'started': time.time()}
     try:
         proc = subprocess.Popen(
-            ['bash', script_path],
+            ['bash', script_path] + (extra_args or []),
             stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
             text=True, bufsize=1
         )
@@ -169,8 +172,8 @@ def cluster_page():
 def backups_page():
     return render_template(
         'pages/backups.html',
-        backups=get_local_backups(),
-        vm_backups=get_vm_backups(),
+        backups=get_local_backups_with_status(),
+        vm_backups=get_vm_backups_with_status(),
         main_server=MAIN_SERVER_IP,
         active_page='backups',
         page_title='Backup Management',
@@ -191,6 +194,14 @@ def restore_page():
         vm_backups=get_vm_backups(),
         restore_prefill=prefill,
         main_server=MAIN_SERVER_IP,
+        running_on_main_server=RUNNING_ON_MAIN_SERVER,
+        # Only meaningful when running as the standby: "restore on this
+        # server" is nonsensical there (no local cluster/kubectl at all),
+        # so pre-fill the remote-target fields with the tunnel back to the
+        # real main server instead of leaving the user to find these values.
+        tunnel_remote_ip=MAIN_SERVER_SSH_HOST,
+        tunnel_remote_port=MAIN_SERVER_SSH_PORT,
+        tunnel_ssh_key=MAIN_SERVER_KEY,
         active_page='restore',
         page_title='Restore',
         page_subtitle='backup → target'
@@ -376,7 +387,7 @@ def api_container_status(name):
 @app.route('/api/backups')
 @login_required
 def api_backups():
-    return jsonify({'local': get_local_backups(), 'vm': get_vm_backups()})
+    return jsonify({'local': get_local_backups_with_status(), 'vm': get_vm_backups_with_status()})
 
 
 @app.route('/api/backups/log')
@@ -431,8 +442,21 @@ def api_backup_run():
             'message': f'Backup script not found at {script}'
         }), 500
 
+    # Optional {"apps": [...]} body — omitted, empty, or "all apps checked"
+    # all mean the same thing as before this existed: no --apps flag, every
+    # app bundled into one archive together (the script's own default).
+    # Only becomes --apps a,b when the UI's app-picker has SOME apps
+    # unchecked, for a deliberate one-off single/partial-app backup.
+    data = request.get_json(silent=True) or {}
+    raw_apps = data.get('apps', [])
+    selected = sorted({
+        a.strip().lower() for a in raw_apps
+        if isinstance(a, str) and a.strip().lower() in ALLOWED_RESTORE_APPS
+    })
+    extra_args = ['--apps', ','.join(selected)] if selected and len(selected) < len(ALLOWED_RESTORE_APPS) else []
+
     job_id = str(uuid.uuid4())
-    t = threading.Thread(target=_stream_backup, args=(job_id, script), daemon=True)
+    t = threading.Thread(target=_stream_backup, args=(job_id, script, extra_args), daemon=True)
     t.start()
     return jsonify({'success': True, 'job_id': job_id, 'status': 'started'})
 
@@ -461,7 +485,9 @@ def api_backup_details():
     backup_file = data.get('backup_file', '').strip()
     source      = data.get('source', 'local').strip()
 
-    if not re.match(r'^myapps-backup-\d{8}_\d{6}\.tar\.gz$', backup_file):
+    # Accepts both the legacy Docker-era lineage (myapps-backup-*) and the
+    # k8s-native one (myapps-k8s-backup-*, from backup-k8s-apps.sh).
+    if not re.match(r'^myapps-(k8s-)?backup-\d{8}_\d{6}\.tar\.gz$', backup_file):
         return jsonify({'error': 'Invalid filename'}), 400
 
     if source == 'local':
@@ -484,16 +510,21 @@ def api_backup_details():
     }
 
     # ── size + mtime ──────────────────────────────────────────────
+    # Try direct stat first regardless of RUNNING_ON_MAIN_SERVER: the
+    # management-platform pod has /root hostPath-mounted from the main
+    # server, so /root/backups is already right here even though the
+    # hostname check that flag relies on never matches inside a pod (it
+    # used to always take the SSH branch, using a key/topology this
+    # in-pod deployment doesn't have, and silently come back empty).
     if source == 'local':
-        if RUNNING_ON_MAIN_SERVER:
+        if os.path.exists(archive_path):
             # direct stat on this machine
-            if os.path.exists(archive_path):
-                st = os.stat(archive_path)
-                result['size_bytes'] = st.st_size
-                result['size_human'] = _human_bytes(st.st_size)
-                mtime = datetime.fromtimestamp(st.st_mtime, tz=timezone.utc)
-                result['created_at'] = mtime.strftime('%Y-%m-%d %H:%M:%S UTC')
-                result['age_days']   = (datetime.now(tz=timezone.utc) - mtime).days
+            st = os.stat(archive_path)
+            result['size_bytes'] = st.st_size
+            result['size_human'] = _human_bytes(st.st_size)
+            mtime = datetime.fromtimestamp(st.st_mtime, tz=timezone.utc)
+            result['created_at'] = mtime.strftime('%Y-%m-%d %H:%M:%S UTC')
+            result['age_days']   = (datetime.now(tz=timezone.utc) - mtime).days
             if os.path.exists(sha_path):
                 try:
                     with open(sha_path) as f:
@@ -517,15 +548,14 @@ def api_backup_details():
                 result['sha256'] = sha_out.strip()
 
     elif source == 'vm':
-        if not RUNNING_ON_MAIN_SERVER:
-            # we ARE the VM — direct stat
-            if os.path.exists(archive_path):
-                st = os.stat(archive_path)
-                result['size_bytes'] = st.st_size
-                result['size_human'] = _human_bytes(st.st_size)
-                mtime = datetime.fromtimestamp(st.st_mtime, tz=timezone.utc)
-                result['created_at'] = mtime.strftime('%Y-%m-%d %H:%M:%S UTC')
-                result['age_days']   = (datetime.now(tz=timezone.utc) - mtime).days
+        if os.path.exists(archive_path):
+            # direct stat when reachable (VM's own local disk, or a mount)
+            st = os.stat(archive_path)
+            result['size_bytes'] = st.st_size
+            result['size_human'] = _human_bytes(st.st_size)
+            mtime = datetime.fromtimestamp(st.st_mtime, tz=timezone.utc)
+            result['created_at'] = mtime.strftime('%Y-%m-%d %H:%M:%S UTC')
+            result['age_days']   = (datetime.now(tz=timezone.utc) - mtime).days
             if os.path.exists(sha_path):
                 try:
                     with open(sha_path) as f:
@@ -688,9 +718,9 @@ def api_r2_upload():
             local_path = f"/tmp/{backup_file}"
             if not os.path.exists(local_path):
                 pull_cmd = (
-                    f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_PORT} "
+                    f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_SSH_PORT} "
                     f"-o StrictHostKeyChecking=no -o ConnectTimeout=15 "
-                    f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}:/root/backups/{backup_file} "
+                    f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}:/root/backups/{backup_file} "
                     f"{local_path}"
                 )
                 res = subprocess.run(pull_cmd, shell=True, capture_output=True, text=True)
@@ -792,9 +822,9 @@ def restore_start():
             backup_path = f"/tmp/{backup_file}"
             if not os.path.exists(backup_path):
                 pull_cmd = (
-                    f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_PORT} "
+                    f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_SSH_PORT} "
                     f"-o StrictHostKeyChecking=no -o ConnectTimeout=15 "
-                    f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}:/root/backups/{backup_file} "
+                    f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}:/root/backups/{backup_file} "
                     f"{backup_path}"
                 )
                 res = subprocess.run(pull_cmd, shell=True, capture_output=True, text=True)
@@ -828,6 +858,12 @@ def restore_start():
         return jsonify({'error': f'restore-k8s-apps.sh not found at {restore_script_local}'}), 500
 
     if target == 'local':
+        if not RUNNING_ON_MAIN_SERVER:
+            return jsonify({
+                'error': 'This is the standby — there is no local cluster here to restore '
+                         'into. Use "External Machine" to restore onto the real main server '
+                         'over the tunnel instead.'
+            }), 400
         hostname    = os.uname().nodename
         os.makedirs('/root/tmp', exist_ok=True)
         session_dir = f"/root/tmp/restore-session-{uuid.uuid4().hex[:8]}"
@@ -847,36 +883,74 @@ def restore_start():
         if not remote_ip:
             return jsonify({'error': 'remote_ip required'}), 400
 
-        base_opts = "-o StrictHostKeyChecking=no -o ConnectTimeout=15"
+        # Unified path: every "External Machine" restore goes through the
+        # same Ansible playbook used for from-scratch DR (ansible/dr-
+        # bootstrap.yml) rather than a bare scp+ssh+restore-k8s-apps.sh
+        # dance. It's safe for BOTH cases, not just the empty-server one:
+        # the playbook's own steps (create-namespace-if-missing, `kubectl
+        # apply` for the sanitized PVC/Secret/manifests) are no-ops against
+        # a target that already has this app running with a matching spec
+        # — apply only reconciles differences, and a backup's own captured
+        # manifests are by definition identical to what's already live —
+        # so "restore onto an existing cluster" and "restore onto nothing"
+        # are the same command now; the playbook's own checks-then-acts
+        # steps decide how much of it actually needs to do anything.
+        ansible_dir = '/root/CloudOps/ansible'
+        playbook_path = f'{ansible_dir}/dr-bootstrap.yml'
+        if not os.path.exists(playbook_path):
+            return jsonify({'error': f'Ansible playbook not found at {playbook_path}'}), 500
+
+        # ANSIBLE_HOST_KEY_CHECKING=False (env var, below) rather than
+        # -e ansible_ssh_common_args='-o StrictHostKeyChecking=no ...':
+        # passing that as an extra-var hits what looks like a real bug in
+        # this ansible-core version's SSH connection plugin (its own
+        # internal tty-detection re-parses the string with a strict
+        # argparse and throws "argument -o: expected one argument" even
+        # for a single well-formed -o KEY=VALUE) — reproduced directly on
+        # the CLI, not a quoting artifact from this code. The env var
+        # achieves the same effect (disables host-key checking) through
+        # ansible's own dedicated option instead, bypassing that path
+        # entirely.
+        common_evars = [
+            f"ansible_user={remote_user}",
+            f"ansible_port={remote_port}",
+        ]
         if auth_method == 'key':
             if not ssh_key_path:
                 return jsonify({'error': 'ssh_key_path required'}), 400
-            ssh_prefix = f"ssh -p {remote_port} -i {ssh_key_path} {base_opts}"
-            scp_prefix = f"scp -P {remote_port} -i {ssh_key_path} {base_opts}"
+            common_evars.append(f"ansible_ssh_private_key_file={ssh_key_path}")
         else:
             if not ssh_password:
                 return jsonify({'error': 'ssh_password required'}), 400
-            ssh_prefix = f"sshpass -p '{ssh_password}' ssh -p {remote_port} {base_opts}"
-            scp_prefix = f"sshpass -p '{ssh_password}' scp -P {remote_port} {base_opts}"
+            common_evars.append(f"ansible_password='{ssh_password}'")
 
-        remote_dest = f"/backups/restore-session-{uuid.uuid4().hex[:8]}"
+        evars_str = " ".join(f"-e {v}" for v in common_evars)
+        # ansible-playbook is a pip console-script installed alongside
+        # whichever python is actually running this process — the system
+        # one in the main pod's Docker image (no venv there), or this
+        # server's own venv/bin on the standby. A bare "ansible-playbook"
+        # in the command string only resolves on the main pod (system PATH
+        # already includes it); on the standby, running a venv's python
+        # directly (not through `activate`) never puts venv/bin on PATH,
+        # so it'd fail with "command not found" there. Resolving it
+        # relative to sys.executable works in both.
+        ansible_playbook_bin = os.path.join(os.path.dirname(sys.executable), 'ansible-playbook')
+
+        # dr-bootstrap.yml restores one app per run (it reads that app's own
+        # manifests.yaml to work out the namespace) — loop for a multi-app
+        # selection, each run against the same already-staged backup_path.
+        apps_for_ansible = selected_apps if selected_apps else sorted(ALLOWED_RESTORE_APPS)
+        ansible_env = "ANSIBLE_HOST_KEY_CHECKING=False ANSIBLE_TIMEOUT=15"
+        per_app_cmds = [
+            f"echo '=== [{a}] provisioning (if needed) + restore via Ansible ===' && "
+            f"{ansible_env} {ansible_playbook_bin} -i '{remote_ip},' {evars_str} "
+            f"-e dr_app={a} -e dr_backup_archive={backup_path} {playbook_path}"
+            for a in apps_for_ansible
+        ]
         cmd = (
-            f"echo 'Connecting to {remote_user}@{remote_ip}:{remote_port}...' && "
+            f"echo 'Target: {remote_user}@{remote_ip}:{remote_port}' && "
             f"echo 'Apps: {apps_label}' && "
-            f"{ssh_prefix} {remote_user}@{remote_ip} 'mkdir -p {remote_dest}' && "
-            f"echo 'Connected.' && "
-            f"echo 'Copying backup archive...' && "
-            f"{scp_prefix} {backup_path} {remote_user}@{remote_ip}:{remote_dest}/{backup_file} && "
-            f"echo 'Copying restore script...' && "
-            f"{scp_prefix} {restore_script_local} {remote_user}@{remote_ip}:{remote_dest}/restore-k8s-apps.sh && "
-            f"echo 'Running restore on {remote_ip}:{remote_port}...' && "
-            f"{ssh_prefix} {remote_user}@{remote_ip} "
-            f"'set -e && cd {remote_dest} && "
-            f"tar -xzf {backup_file} --strip-components=1 && "
-            f"chmod +x restore-k8s-apps.sh && bash restore-k8s-apps.sh{apps_flag}' ; "
-            f"EXIT=$? ; "
-            f"{ssh_prefix} {remote_user}@{remote_ip} 'rm -rf {remote_dest}' 2>/dev/null ; "
-            f"exit $EXIT"
+            + " && ".join(per_app_cmds)
         )
 
     job_id = str(uuid.uuid4())

platform/modules/backups.py

@@ -11,7 +11,8 @@ import socket
 import psutil
 from config import (
     RUNNING_ON_MAIN_SERVER,
-    MAIN_SERVER_IP, MAIN_SERVER_USER, MAIN_SERVER_KEY, MAIN_SERVER_PORT,
+    MAIN_SERVER_USER, MAIN_SERVER_KEY,
+    MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT,
     VM_HOST, VM_PORT, VM_KEY, VM_USER,
 )
 
@@ -43,10 +44,10 @@ def _ssh_main(remote_cmd, timeout=30):
     else:
         escaped = remote_cmd.replace("'", "'\\''")
         ssh = (
-            f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_PORT} "
+            f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_SSH_PORT} "
             f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 "
             f"-o BatchMode=yes "
-            f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}"
+            f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}"
         )
         return _run(f"{ssh} '{escaped}'", timeout=timeout)
 
@@ -55,10 +56,34 @@ def _ssh_main(remote_cmd, timeout=30):
 # BACKUPS
 # ────────────────────────────────────────────────────────────────
 
+def _list_local_dir(local_dir):
+    """Direct filesystem listing, newest first, both lineages."""
+    files = (
+        glob.glob(f'{local_dir}/myapps-backup-*.tar.gz')
+        + glob.glob(f'{local_dir}/myapps-k8s-backup-*.tar.gz')
+    )
+    files.sort(key=os.path.getmtime, reverse=True)
+    return [os.path.basename(f) for f in files[:20]]
+
+
 def get_local_backups():
     # myapps-backup-* is the legacy Docker-era lineage; myapps-k8s-backup-*
     # is the k8s-native one (backup-k8s-apps.sh). Listed together, sorted
     # by mtime, so history from both stays visible in the UI.
+    #
+    # Try direct filesystem access first: the management-platform pod has
+    # /root hostPath-mounted from the main server (see k8s/management-
+    # platform-deployment.yaml), so /root/backups is already right here.
+    # RUNNING_ON_MAIN_SERVER's hostname check never matches inside a pod
+    # (its hostname is the pod name, not the host's), so this used to
+    # always fall through to the SSH-to-self branch below — which also
+    # doesn't actually work in this topology (wrong key for this deploy
+    # model) — silently returning an empty list. Falling back to SSH only
+    # when there's genuinely no local mount keeps this working for any
+    # future non-mounted deployment too.
+    if os.path.isdir('/root/backups'):
+        return _list_local_dir('/root/backups')
+
     stdout, _ = _ssh_main(
         "ls -t /root/backups/myapps-backup-*.tar.gz "
         "/root/backups/myapps-k8s-backup-*.tar.gz 2>/dev/null | head -20"
@@ -73,37 +98,175 @@ def get_local_backups():
 
 
 def get_vm_backups():
+    # Same fix as get_local_backups(): try direct access first (covers the
+    # case where this process's filesystem actually is the VM's), fall
+    # back to SSH to the VM host otherwise — which is the real case for
+    # the management-platform pod, since only /root is mounted in, not
+    # anything from the separate VM backup host.
+    local_dir = '/backups/cloudproject'
+    if os.path.isdir(local_dir):
+        return _list_local_dir(local_dir)
+
     vm_backups = []
-    if RUNNING_ON_MAIN_SERVER:
-        try:
-            cmd = (
-                f"ssh -i {VM_KEY} -p {VM_PORT} "
-                f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 "
-                f"-o BatchMode=yes "
-                f"{VM_USER}@{VM_HOST} "
-                f"'ls -t /backups/cloudproject/myapps-backup-*.tar.gz "
-                f"/backups/cloudproject/myapps-k8s-backup-*.tar.gz 2>/dev/null | head -20'"
-            )
-            stdout, _ = _run(cmd, timeout=25)
-            if stdout:
-                for line in stdout.split('\n'):
-                    line = line.strip()
-                    if line and '.tar.gz' in line:
-                        vm_backups.append(os.path.basename(line))
-        except Exception as e:
-            print(f"[backups] VM backup fetch error: {e}")
-    else:
-        backup_dir = '/backups/cloudproject'
-        if os.path.exists(backup_dir):
-            files = (
-                glob.glob(f'{backup_dir}/myapps-backup-*.tar.gz')
-                + glob.glob(f'{backup_dir}/myapps-k8s-backup-*.tar.gz')
-            )
-            files.sort(key=os.path.getmtime, reverse=True)
-            vm_backups = [os.path.basename(f) for f in files[:20]]
+    try:
+        cmd = (
+            f"ssh -i {VM_KEY} -p {VM_PORT} "
+            f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 "
+            f"-o BatchMode=yes "
+            f"{VM_USER}@{VM_HOST} "
+            f"'ls -t /backups/cloudproject/myapps-backup-*.tar.gz "
+            f"/backups/cloudproject/myapps-k8s-backup-*.tar.gz 2>/dev/null | head -20'"
+        )
+        stdout, _ = _run(cmd, timeout=25)
+        if stdout:
+            for line in stdout.split('\n'):
+                line = line.strip()
+                if line and '.tar.gz' in line:
+                    vm_backups.append(os.path.basename(line))
+    except Exception as e:
+        print(f"[backups] VM backup fetch error: {e}")
     return vm_backups
 
 
+# ────────────────────────────────────────────────────────────────
+# BACKUP STATUS DOTS (green/yellow/red/unknown)
+#
+# Reads the .meta.json sidecar backup-k8s-apps.sh writes alongside each
+# archive (never decompresses the archive itself). Legacy myapps-backup-*
+# archives and any k8s backup made before this feature shipped have no
+# sidecar and show as 'unknown' — there's no reliable way to reconstruct
+# per-step error info after the fact, and no backfill is attempted.
+# ────────────────────────────────────────────────────────────────
+
+ROLLING_WINDOW = 5
+SIZE_DEVIATION_THRESHOLD = 0.40
+
+
+def _load_local_sidecars(local_dir='/root/backups'):
+    """All .meta.json sidecars in a directory, newest first. Used both to
+    look up one backup's own metadata and as the history pool for its
+    rolling-average comparison."""
+    sidecars = {}
+    for path in glob.glob(f'{local_dir}/*.tar.gz.meta.json'):
+        try:
+            with open(path) as f:
+                m = json.load(f)
+        except Exception:
+            continue
+        backup_file = os.path.basename(path)[:-len('.meta.json')]
+        m['_backup_file'] = backup_file
+        sidecars[backup_file] = m
+    return sidecars
+
+
+def _parse_vm_sidecars(cat_output):
+    """Parse the batched `echo marker + cat` SSH output from _fetch_vm_sidecars
+    into the same {backup_file: meta} shape as _load_local_sidecars."""
+    sidecars = {}
+    if not cat_output:
+        return sidecars
+    for block in cat_output.split('===META:')[1:]:
+        try:
+            fname_line, _, rest = block.partition('===\n')
+            backup_file = fname_line.strip()[:-len('.meta.json')]
+            m = json.loads(rest)
+            m['_backup_file'] = backup_file
+            sidecars[backup_file] = m
+        except Exception:
+            continue
+    return sidecars
+
+
+def _fetch_vm_sidecars():
+    local_dir = '/backups/cloudproject'
+    if os.path.isdir(local_dir):
+        return _load_local_sidecars(local_dir)
+    cmd = (
+        f"ssh -i {VM_KEY} -p {VM_PORT} -o StrictHostKeyChecking=no "
+        f"-o ConnectTimeout=10 -o BatchMode=yes {VM_USER}@{VM_HOST} "
+        f"'for f in /backups/cloudproject/*.tar.gz.meta.json; do "
+        f"[ -f \"$f\" ] && echo \"===META:$(basename \"$f\")===\" && cat \"$f\"; "
+        f"done 2>/dev/null'"
+    )
+    stdout, _ = _run(cmd, timeout=25)
+    return _parse_vm_sidecars(stdout)
+
+
+def _compute_backup_status(meta, all_sidecars):
+    """all_sidecars: dict of {backup_file: meta} — the pool this entry's
+    rolling average is computed from (same directory/tier it came from)."""
+    tiers = meta.get('tiers', {})
+    if meta.get('errors') or any(str(v).startswith('failed') for v in tiers.values()):
+        return 'red'
+
+    apps_key = meta.get('apps_key', '')
+    size_bytes = meta.get('size_bytes', 0)
+    created_at = meta.get('created_at', '')
+
+    # Same app-combination, strictly older than this backup, most recent
+    # first — a fresh, self-calibrating baseline per app-set rather than
+    # one hardcoded global threshold (a 1-app and a 5-app backup have very
+    # different normal sizes).
+    history = sorted(
+        (m for m in all_sidecars.values()
+         if m.get('apps_key') == apps_key and m.get('created_at', '') < created_at),
+        key=lambda m: m.get('created_at', ''),
+        reverse=True,
+    )[:ROLLING_WINDOW]
+
+    if not history:
+        return 'green'  # first backup of this app-combination — nothing to compare against
+
+    avg = sum(m.get('size_bytes', 0) for m in history) / len(history)
+    if avg <= 0:
+        return 'green'
+
+    deviation = abs(size_bytes - avg) / avg
+    return 'yellow' if deviation > SIZE_DEVIATION_THRESHOLD else 'green'
+
+
+def _with_status(names, sidecars):
+    result = []
+    for name in names:
+        meta = sidecars.get(name)
+        if meta is None:
+            result.append({'name': name, 'status': 'unknown', 'size_human': None, 'apps': None})
+        else:
+            result.append({
+                'name': name,
+                'status': _compute_backup_status(meta, sidecars),
+                'size_human': _human_bytes(meta.get('size_bytes', 0)),
+                'apps': meta.get('apps'),
+            })
+    return result
+
+
+def _fetch_local_sidecars():
+    """Same local-dir-first-else-SSH pattern as _fetch_vm_sidecars() below,
+    but for /root/backups on the main server. _load_local_sidecars() alone
+    has no such fallback — a plain glob.glob() against a hardcoded path —
+    so on the standby (where /root/backups doesn't exist locally at all)
+    it silently returned {}, and every local-column status dot rendered as
+    gray/unknown regardless of the backup's real status."""
+    if os.path.isdir('/root/backups'):
+        return _load_local_sidecars('/root/backups')
+    cmd = (
+        "for f in /root/backups/*.tar.gz.meta.json; do "
+        "[ -f \"$f\" ] && echo \"===META:$(basename \"$f\")===\" && cat \"$f\"; "
+        "done 2>/dev/null"
+    )
+    stdout, _ = _ssh_main(cmd, timeout=25)
+    return _parse_vm_sidecars(stdout)
+
+
+def get_local_backups_with_status():
+    return _with_status(get_local_backups(), _fetch_local_sidecars())
+
+
+def get_vm_backups_with_status():
+    return _with_status(get_vm_backups(), _fetch_vm_sidecars())
+
+
 # ────────────────────────────────────────────────────────────────
 # BACKUP HEALTH AUDIT
 # ────────────────────────────────────────────────────────────────
@@ -115,31 +278,49 @@ def get_vm_backups():
 # test, easy to extend without pushing complexity back up.
 
 def _resolve_archive_path(backup_file, source, add):
-    """Figure out the local path to the archive, pulling it from the main
-    server first if needed. Returns the path, or None if it could not be
-    resolved (in which case an error result dict has already been added
-    via `add` and the caller should bail out)."""
+    """Figure out the local path to the archive, pulling it over SSH only
+    if it's genuinely not reachable on this filesystem. Returns the path,
+    or None if it could not be resolved (in which case an error result
+    dict has already been added via `add` and the caller should bail out).
+
+    Tries direct access first regardless of RUNNING_ON_MAIN_SERVER: the
+    management-platform pod has /root hostPath-mounted from the main
+    server, so /root/backups is already right here even though the
+    hostname check that flag relies on never matches inside a pod."""
     if source == 'local':
         archive_path = f"/root/backups/{backup_file}"
-    else:
-        archive_path = f"/backups/cloudproject/{backup_file}"
-
-    if not RUNNING_ON_MAIN_SERVER and source == 'local':
+        if os.path.exists(archive_path):
+            return archive_path
         tmp_path = f"/tmp/audit_{backup_file}"
         if not os.path.exists(tmp_path):
             pull_cmd = (
-                f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_PORT} "
+                f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_SSH_PORT} "
                 f"-o StrictHostKeyChecking=no -o ConnectTimeout=15 "
-                f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}:/root/backups/{backup_file} "
+                f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}:/root/backups/{backup_file} "
                 f"{tmp_path}"
             )
             _out, err = _run(pull_cmd, timeout=120)
             if not os.path.exists(tmp_path):
                 add('File Access', 'fail', f'Could not pull from main server: {err}')
                 return None
-        archive_path = tmp_path
-
-    return archive_path
+        return tmp_path
+    else:
+        archive_path = f"/backups/cloudproject/{backup_file}"
+        if os.path.exists(archive_path):
+            return archive_path
+        tmp_path = f"/tmp/audit_vm_{backup_file}"
+        if not os.path.exists(tmp_path):
+            pull_cmd = (
+                f"scp -i {VM_KEY} -P {VM_PORT} "
+                f"-o StrictHostKeyChecking=no -o ConnectTimeout=15 "
+                f"{VM_USER}@{VM_HOST}:/backups/cloudproject/{backup_file} "
+                f"{tmp_path}"
+            )
+            _out, err = _run(pull_cmd, timeout=120)
+            if not os.path.exists(tmp_path):
+                add('File Access', 'fail', f'Could not pull from VM: {err}')
+                return None
+        return tmp_path
 
 
 def _check_file_size(archive_path, add):
@@ -216,13 +397,54 @@ def _list_archive_members(archive_path):
         return []
 
 
-def _check_internal_structure(members, add):
+# The two backup lineages have completely different internal layouts:
+#   - legacy (myapps-backup-*, from backup-myapps.sh): volumes/*.tar.gz,
+#     compose-files/, backup-info.txt at the archive root.
+#   - k8s-native (myapps-k8s-backup-*, from backup-k8s-apps.sh): per-app
+#     subdirs (n8n/, odoo/, mautic/, nextcloud/, frappe/) each with
+#     manifests.yaml, secret.yaml, db-dump.sql.gz, pvc-data.tar.gz, plus
+#     backup-info.txt at the archive root.
+# Auditing a k8s-format archive against the legacy layout would always
+# fail it ("volumes/ missing") even when it's a perfectly good backup —
+# same root cause as the myapps-backup-*/myapps-k8s-backup-* prefix bug
+# already fixed elsewhere in this file, just in the audit checks instead
+# of the listing/delete glob.
+_K8S_APP_NAMES = ('n8n', 'odoo', 'mautic', 'nextcloud', 'frappe')
+
+
+def _is_k8s_format(backup_file):
+    return os.path.basename(backup_file).startswith('myapps-k8s-backup-')
+
+
+def _check_internal_structure(members, backup_file, add):
     if not members:
         add('Internal Structure', 'warn', 'Could not inspect archive members')
         return
 
-    has_volumes = any('volumes/' in m for m in members)
     has_info = any('backup-info.txt' in m for m in members)
+
+    if _is_k8s_format(backup_file):
+        app_dirs = sorted({
+            parts[1] for m in members
+            if len(parts := m.split('/')) > 1 and parts[1] in _K8S_APP_NAMES
+        })
+        has_app_data = any(
+            m.endswith(('manifests.yaml', 'db-dump.sql.gz', 'pvc-data.tar.gz'))
+            for m in members
+        )
+        issues = []
+        if not has_app_data:
+            issues.append('no per-app manifests/db-dump/pvc-data found')
+        if not has_info:
+            issues.append('backup-info.txt missing')
+        if issues:
+            add('Internal Structure', 'fail', ' · '.join(issues))
+            return
+        add('Internal Structure', 'pass',
+            f'backup-info.txt ✓  apps: {", ".join(app_dirs) or "?"}')
+        return
+
+    has_volumes = any('volumes/' in m for m in members)
     has_compose = any('compose-files/' in m for m in members)
     vol_count = len([m for m in members if '/volumes/' in m and m.endswith('.tar.gz')])
 
@@ -302,7 +524,20 @@ def _check_executable_scripts(archive_path, add):
         add('Executable Scripts', 'pass', 'No unexpected executable scripts found')
 
 
-def _check_volume_count(members, add):
+def _check_volume_count(members, backup_file, add):
+    if _is_k8s_format(backup_file):
+        pvc_archives = [m for m in members if m.endswith('/pvc-data.tar.gz')]
+        v = len(pvc_archives)
+        if v == 0:
+            add('App Data Archives', 'fail', 'No pvc-data.tar.gz found in backup')
+        elif v < 5:
+            add('App Data Archives', 'warn',
+                f'{v} app(s) backed up — fine if this was a --apps filtered run, '
+                f'otherwise expected 5 for a full backup')
+        else:
+            add('App Data Archives', 'pass', f'{v} app data archives present')
+        return
+
     vol_archives = [m for m in members if 'volumes/' in m and m.endswith('.tar.gz')]
     v = len(vol_archives)
     if v == 0:
@@ -379,10 +614,10 @@ def audit_backup(backup_file, source='local'):
     _check_archive_integrity(archive_path, add)
 
     members = _list_archive_members(archive_path)
-    _check_internal_structure(members, add)
+    _check_internal_structure(members, backup_file, add)
     _check_security_scan(members, add)
     _check_executable_scripts(archive_path, add)
-    _check_volume_count(members, add)
+    _check_volume_count(members, backup_file, add)
 
     score = _score_checks(checks)
     has_fails = any(c['status'] == 'fail' for c in checks)
@@ -429,11 +664,16 @@ def delete_backup(backup_file, source='local'):
             if not os.path.exists(archive_path):
                 return False, f'File not found: {archive_path}'
             os.remove(archive_path)
-            sha = archive_path + '.sha256'
-            if os.path.exists(sha): os.remove(sha)
+            for suffix in ('.sha256', '.meta.json'):
+                sidecar = archive_path + suffix
+                if os.path.exists(sidecar): os.remove(sidecar)
             return True, f'Deleted {backup_file} from main server'
         else:
-            cmd = f"rm -f /root/backups/{backup_file} /root/backups/{backup_file}.sha256"
+            cmd = (
+                f"rm -f /root/backups/{backup_file} "
+                f"/root/backups/{backup_file}.sha256 "
+                f"/root/backups/{backup_file}.meta.json"
+            )
             out, err = _ssh_main(cmd)
             if err and 'No such file' not in err:
                 return False, f'Remote delete error: {err}'
@@ -445,8 +685,9 @@ def delete_backup(backup_file, source='local'):
             if not os.path.exists(archive_path):
                 return False, f'File not found: {archive_path}'
             os.remove(archive_path)
-            sha = archive_path + '.sha256'
-            if os.path.exists(sha): os.remove(sha)
+            for suffix in ('.sha256', '.meta.json'):
+                sidecar = archive_path + suffix
+                if os.path.exists(sidecar): os.remove(sidecar)
             return True, f'Deleted {backup_file} from VM'
         else:
             cmd = (
@@ -455,7 +696,8 @@ def delete_backup(backup_file, source='local'):
                 f"-o BatchMode=yes "
                 f"{VM_USER}@{VM_HOST} "
                 f"'rm -f /backups/cloudproject/{backup_file} "
-                f"/backups/cloudproject/{backup_file}.sha256'"
+                f"/backups/cloudproject/{backup_file}.sha256 "
+                f"/backups/cloudproject/{backup_file}.meta.json'"
             )
             out, err = _run(cmd, timeout=30)
             if err and 'No such file' not in err:
@@ -671,8 +913,65 @@ def get_all_stats():
 # SYSTEM INFO (FIXED WITH PSUTIL)
 # ────────────────────────────────────────────────────────────────
 
+def _get_main_server_system_info_remote():
+    """get_system_info()'s equivalent for the standby: the main server's bare
+    host has no psutil (system python3, not this app's container image), so
+    this uses plain coreutils/procps instead — reusing _ssh_main means this
+    genuinely queries the real main server over the tunnel, not the VM
+    itself (the previous unconditional psutil.* calls silently reported the
+    VM's own stats mislabeled as "system info" whenever running as standby)."""
+    fallback = {
+        'cpu_pct': '0', 'memory': 'N/A', 'mem_pct': '0', 'disk': 'N/A',
+        'disk_pct': '0', 'load': 'N/A', 'uptime': 'N/A', 'docker_v': 'N/A',
+        'hostname': 'unreachable',
+    }
+    remote_cmd = (
+        "hostname; "
+        "vmstat 1 2 | tail -1 | awk '{print 100-$15}'; "
+        "free -m | awk 'NR==2{printf \"%d %d %d\\n\", $3, $2, $3*100/$2}'; "
+        "df -BM --output=used,size,pcent / | tail -1 | tr -d 'M%'; "
+        "awk '{printf \"%.2f %.2f %.2f\\n\", $1, $2, $3}' /proc/loadavg; "
+        "awk '{print $1}' /proc/uptime; "
+        "docker --version 2>/dev/null | awk '{print $3}' | tr -d ','"
+    )
+    out, err = _ssh_main(remote_cmd, timeout=15)
+    lines = out.split('\n') if out else []
+    if len(lines) < 7:
+        return fallback
+    try:
+        hostname = lines[0].strip()
+        cpu_pct = lines[1].strip()
+        mem_used_mb, mem_total_mb, mem_pct = lines[2].split()
+        disk_used_mb, disk_total_mb, disk_pct = lines[3].split()
+        load = lines[4].strip()
+        uptime_s = float(lines[5].strip())
+        docker_v = lines[6].strip() or 'N/A'
+
+        days = int(uptime_s // 86400)
+        hours = int((uptime_s % 86400) // 3600)
+        minutes = int((uptime_s % 3600) // 60)
+        uptime = f"{days}d {hours}h {minutes}m" if days > 0 else f"{hours}h {minutes}m"
+
+        return {
+            'cpu_pct':  cpu_pct,
+            'memory':   f"{int(mem_used_mb) / 1024:.1f}G/{int(mem_total_mb) / 1024:.1f}G",
+            'mem_pct':  mem_pct,
+            'disk':     f"{int(disk_used_mb) / 1024:.0f}G/{int(disk_total_mb) / 1024:.0f}G",
+            'disk_pct': disk_pct,
+            'load':     load,
+            'uptime':   uptime,
+            'docker_v': docker_v,
+            'hostname': hostname,
+        }
+    except Exception as e:
+        print(f"[backups] remote system info parse error: {e} (raw: {out!r}, err: {err!r})")
+        return fallback
+
+
 def get_system_info():
     """Get system information using psutil for reliable metrics inside container"""
+    if not RUNNING_ON_MAIN_SERVER:
+        return _get_main_server_system_info_remote()
 
     info = {
         'cpu_pct': '0',

platform/modules/cloud_backup.py

@@ -345,10 +345,12 @@ def r2_audit_backup(key: str) -> dict:
         else:
             report["checks"].append(f"✅ File size looks healthy ({_human_size(size)})")
 
-        # Check 5: Filename format
+        # Check 5: Filename format — accepts both the legacy Docker-era
+        # lineage (myapps-backup-*) and the k8s-native one
+        # (myapps-k8s-backup-*, from backup-k8s-apps.sh).
         import re
         filename = key.replace("backups/", "")
-        if re.match(r"myapps-backup-\d{8}_\d{6}\.tar\.gz", filename):
+        if re.match(r"myapps-(k8s-)?backup-\d{8}_\d{6}\.tar\.gz", filename):
             report["checks"].append("✅ Filename format is valid")
         else:
             report["warnings"].append(f"⚠️ Unexpected filename format: {filename}")

platform/modules/commands.py

@@ -1,5 +1,5 @@
 import subprocess
-from config import RUNNING_ON_MAIN_SERVER, MAIN_SERVER_IP
+from config import RUNNING_ON_MAIN_SERVER, MAIN_SERVER_KEY, MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT
 
 def run_command(command):
     """Run command on main server (directly or via SSH)"""
@@ -14,10 +14,12 @@ def run_command(command):
     else:
         ssh_cmd = [
             "ssh",
+            "-i", MAIN_SERVER_KEY,
+            "-p", MAIN_SERVER_SSH_PORT,
             "-o", "StrictHostKeyChecking=no",
             "-o", "ConnectTimeout=10",
             "-o", "BatchMode=yes",
-            f"root@{MAIN_SERVER_IP}",
+            f"root@{MAIN_SERVER_SSH_HOST}",
             command
         ]
         try:

platform/modules/kubernetes.py

@@ -4,6 +4,18 @@
 # deployments.apps — no secrets, no write, no exec/log), bound per-namespace
 # across the 8 app namespaces below. There is no cluster-wide RoleBinding,
 # so every list call below is scoped to one namespace at a time.
+#
+# On the standby (RUNNING_ON_MAIN_SERVER False), there is no in-cluster
+# ServiceAccount to load at all — load_incluster_config() always fails
+# there, by design, even if the standby itself later runs as a pod on a
+# *local* k3s on that VM (that cluster isn't this one; in-cluster config
+# would only ever see the wrong, essentially-empty cluster). Every function
+# below instead SSHes to the main server (over main-to-vm-tunnel.service)
+# and runs `kubectl` there directly, where real cluster-admin access
+# already exists — same real-vs-display separation as MAIN_SERVER_SSH_HOST
+# in config.py.
+import json
+import subprocess
 from datetime import datetime, timezone
 
 try:
@@ -13,6 +25,11 @@ try:
 except ImportError:
     _K8S_IMPORT_OK = False
 
+from config import (
+    RUNNING_ON_MAIN_SERVER, MAIN_SERVER_USER, MAIN_SERVER_KEY,
+    MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT,
+)
+
 NAMESPACES = [
     'n8n', 'odoo', 'mautic', 'erpnext', 'nextcloud',
     'jenkins-agents', 'jenkins', 'management-platform',
@@ -21,11 +38,84 @@ NAMESPACES = [
 _core_v1 = None
 _apps_v1 = None
 _custom_objects = None
+_networking_v1 = None
 K8S_AVAILABLE = False
 
 
+def _ssh_kubectl(args):
+    """Run `kubectl <args> -o json` on the main server over the reverse
+    tunnel. Returns the parsed JSON (dict) or None on any failure —
+    callers must treat None/empty the same as the in-cluster client
+    returning nothing, never as a hard error."""
+    remote_cmd = "kubectl " + " ".join(args) + " -o json"
+    ssh_cmd = (
+        f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_SSH_PORT} "
+        f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 -o BatchMode=yes "
+        f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST} '{remote_cmd}'"
+    )
+    try:
+        r = subprocess.run(ssh_cmd, shell=True, capture_output=True, text=True, timeout=20)
+        if r.returncode != 0:
+            return None
+        return json.loads(r.stdout)
+    except Exception:
+        return None
+
+
+def _ssh_kubectl_raw(path):
+    """Same as _ssh_kubectl but for `kubectl get --raw <path>` (used for
+    the metrics-server API, which isn't a normal `get <resource>` call)."""
+    ssh_cmd = (
+        f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_SSH_PORT} "
+        f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 -o BatchMode=yes "
+        f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST} 'kubectl get --raw {path}'"
+    )
+    try:
+        r = subprocess.run(ssh_cmd, shell=True, capture_output=True, text=True, timeout=20)
+        if r.returncode != 0:
+            return None
+        return json.loads(r.stdout)
+    except Exception:
+        return None
+
+
+def _parse_k8s_ts(s):
+    if not s:
+        return None
+    try:
+        return datetime.strptime(s, '%Y-%m-%dT%H:%M:%SZ').replace(tzinfo=timezone.utc)
+    except Exception:
+        return None
+
+
+def _pod_health_raw(pod):
+    statuses = (pod.get('status') or {}).get('containerStatuses') or []
+    for cs in statuses:
+        waiting = (cs.get('state') or {}).get('waiting')
+        if waiting and waiting.get('reason') in _CRASH_REASONS:
+            return 'failed'
+    phase = (pod.get('status') or {}).get('phase') or 'Unknown'
+    if phase == 'Failed':
+        return 'failed'
+    if phase == 'Running':
+        all_ready = all(cs.get('ready') for cs in statuses) if statuses else True
+        return 'healthy' if all_ready else 'degraded'
+    return 'degraded'
+
+
+def _ready_count_raw(pod):
+    statuses = (pod.get('status') or {}).get('containerStatuses') or []
+    ready = sum(1 for cs in statuses if cs.get('ready'))
+    return f'{ready}/{len(statuses)}'
+
+
+def _restart_count_raw(pod):
+    statuses = (pod.get('status') or {}).get('containerStatuses') or []
+    return sum(cs.get('restartCount', 0) for cs in statuses)
+
+
 def _init_client():
-    global _core_v1, _apps_v1, _custom_objects, K8S_AVAILABLE
+    global _core_v1, _apps_v1, _custom_objects, _networking_v1, K8S_AVAILABLE
     if _core_v1 is not None or not _K8S_IMPORT_OK:
         return
     try:
@@ -33,6 +123,7 @@ def _init_client():
         _core_v1 = client.CoreV1Api()
         _apps_v1 = client.AppsV1Api()
         _custom_objects = client.CustomObjectsApi()
+        _networking_v1 = client.NetworkingV1Api()
         K8S_AVAILABLE = True
     except Exception as e:
         print(f"[kubernetes] in-cluster config unavailable: {e}")
@@ -133,6 +224,24 @@ def _parse_mem_mib(v):
 # ────────────────────────────────────────────────────────────────
 
 def list_pods():
+    if not RUNNING_ON_MAIN_SERVER:
+        pods = []
+        for ns in NAMESPACES:
+            data = _ssh_kubectl(['get', 'pods', '-n', ns])
+            if not data:
+                continue
+            for pod in data.get('items', []):
+                pods.append({
+                    'name':      pod['metadata']['name'],
+                    'namespace': ns,
+                    'phase':     (pod.get('status') or {}).get('phase') or 'Unknown',
+                    'health':    _pod_health_raw(pod),
+                    'ready':     _ready_count_raw(pod),
+                    'restarts':  _restart_count_raw(pod),
+                    'age':       _age_str(_parse_k8s_ts(pod['metadata'].get('creationTimestamp'))),
+                })
+        return pods
+
     _init_client()
     pods = []
     if not K8S_AVAILABLE:
@@ -157,6 +266,25 @@ def list_pods():
 
 
 def list_deployments():
+    if not RUNNING_ON_MAIN_SERVER:
+        deployments = []
+        for ns in NAMESPACES:
+            data = _ssh_kubectl(['get', 'deployments', '-n', ns])
+            if not data:
+                continue
+            for dep in data.get('items', []):
+                desired = (dep.get('spec') or {}).get('replicas') or 0
+                ready = (dep.get('status') or {}).get('readyReplicas') or 0
+                deployments.append({
+                    'name':      dep['metadata']['name'],
+                    'namespace': ns,
+                    'ready':     ready,
+                    'desired':   desired,
+                    'health':    _deploy_health(desired, ready),
+                    'age':       _age_str(_parse_k8s_ts(dep['metadata'].get('creationTimestamp'))),
+                })
+        return deployments
+
     _init_client()
     deployments = []
     if not K8S_AVAILABLE:
@@ -181,10 +309,104 @@ def list_deployments():
     return deployments
 
 
+def get_ingress_info(name, namespace):
+    """Real domain/TLS for one Ingress, or None if unavailable/not found.
+    Used by modules/sites.py instead of the old hardcoded Docker-era
+    domain fields — reflects actual live k8s Ingress state."""
+    if not RUNNING_ON_MAIN_SERVER:
+        data = _ssh_kubectl(['get', 'ingress', name, '-n', namespace])
+        if not data:
+            return None
+        host = None
+        rules = (data.get('spec') or {}).get('rules') or []
+        for rule in rules:
+            if rule.get('host'):
+                host = rule['host']
+                break
+        tls = bool((data.get('spec') or {}).get('tls'))
+        return {'host': host, 'tls': tls}
+
+    _init_client()
+    if not K8S_AVAILABLE:
+        return None
+    try:
+        ing = _networking_v1.read_namespaced_ingress(name, namespace)
+    except Exception:
+        return None
+    host = None
+    if ing.spec and ing.spec.rules:
+        for rule in ing.spec.rules:
+            if rule.host:
+                host = rule.host
+                break
+    tls = bool(ing.spec and ing.spec.tls)
+    return {'host': host, 'tls': tls}
+
+
+def list_deployments_for_namespace(ns):
+    """Targeted single-namespace deployment list (vs. list_deployments()'s
+    fixed NAMESPACES sweep) — used by modules/sites.py per app."""
+    if not RUNNING_ON_MAIN_SERVER:
+        data = _ssh_kubectl(['get', 'deployments', '-n', ns])
+        if not data:
+            return []
+        out = []
+        for dep in data.get('items', []):
+            desired = (dep.get('spec') or {}).get('replicas') or 0
+            ready = (dep.get('status') or {}).get('readyReplicas') or 0
+            containers = (((dep.get('spec') or {}).get('template') or {}).get('spec') or {}).get('containers') or []
+            image = containers[0]['image'] if containers else '—'
+            out.append({
+                'name':    dep['metadata']['name'],
+                'desired': desired,
+                'ready':   ready,
+                'health':  _deploy_health(desired, ready),
+                'image':   image,
+            })
+        return out
+
+    _init_client()
+    if not K8S_AVAILABLE:
+        return []
+    try:
+        resp = _apps_v1.list_namespaced_deployment(ns)
+    except Exception as e:
+        print(f"[kubernetes] list deployments failed for {ns}: {e}")
+        return []
+    out = []
+    for dep in resp.items:
+        desired = dep.spec.replicas or 0
+        ready = dep.status.ready_replicas or 0
+        containers = dep.spec.template.spec.containers or []
+        image = containers[0].image if containers else '—'
+        out.append({
+            'name':    dep.metadata.name,
+            'desired': desired,
+            'ready':   ready,
+            'health':  _deploy_health(desired, ready),
+            'image':   image,
+        })
+    return out
+
+
 def get_pod_metrics():
     """Best-effort CPU/mem per pod via metrics-server. Returns {} if it's not
     installed or unreachable — callers must treat an empty dict as 'no data',
     never as an error."""
+    if not RUNNING_ON_MAIN_SERVER:
+        metrics = {}
+        for ns in NAMESPACES:
+            resp = _ssh_kubectl_raw(f'/apis/metrics.k8s.io/v1beta1/namespaces/{ns}/pods')
+            if not resp:
+                continue
+            for item in resp.get('items', []):
+                name = item.get('metadata', {}).get('name')
+                containers = item.get('containers', [])
+                cpu = sum(_parse_cpu(c.get('usage', {}).get('cpu')) for c in containers)
+                mem = sum(_parse_mem_mib(c.get('usage', {}).get('memory')) for c in containers)
+                metrics[f'{ns}/{name}'] = {'cpu_millicores': round(cpu), 'mem_mib': round(mem)}
+        return metrics
+
     _init_client()
     metrics = {}
     if not K8S_AVAILABLE:
@@ -233,8 +455,13 @@ def get_cluster_overview():
         'deployments_healthy': sum(1 for d in deployments if d['health'] == 'healthy'),
     }
 
+    # K8S_AVAILABLE only reflects the in-cluster client (main server); on
+    # the standby, "available" instead means the SSH+kubectl fetch above
+    # actually returned something.
+    available = K8S_AVAILABLE if RUNNING_ON_MAIN_SERVER else bool(pods or deployments)
+
     return {
-        'available':        K8S_AVAILABLE,
+        'available':        available,
         'namespaces':        by_namespace,
         'summary':           summary,
         'metrics_available': bool(metrics),

platform/modules/sites.py

@@ -1,15 +1,27 @@
 # modules/sites.py — Managed application sites registry + live status
-import re
+#
+# REWRITTEN 2026-08-21: this used to be a fully Docker-era hardcoded registry
+# (container names like odoo-clean-odoo-1/frappe-erpnext/nextcloud-app that
+# no longer exist since the k8s migration, checked via `docker inspect` over
+# SSH) with hardcoded domain/port fields — nextcloud and mautic even had
+# `domain: None`, silently falling back to dead Docker host-port URLs while
+# their real k8s Ingress domains (next.cloud.nav.ovh, mautics.nav.ovh) sat
+# unused. Now sources everything live from the cluster: Deployment/pod
+# status via modules/kubernetes.py's in-cluster client, and domain/TLS via
+# the actual Ingress objects (not a static guess) — see
+# management-platform-sites-rbac.yaml for the narrow ingress-read grant this
+# needed in the `default` namespace (odoo/nextcloud's Ingress historically
+# live there, not in their app namespace).
 import time
-from urllib.request import urlopen, Request
-from urllib.error import URLError
+import urllib.request
 
-from config import MAIN_SERVER_IP, RUNNING_ON_MAIN_SERVER
-from modules.backups import _ssh_main, get_all_root_containers, get_container_status
+from modules.kubernetes import get_ingress_info, list_deployments_for_namespace
 
 
 # ────────────────────────────────────────────────────────────────
-# STATIC SITE REGISTRY (source of truth for Application Sites UI)
+# STATIC SITE REGISTRY — identity/namespace/naming only. Live state
+# (domain, TLS, deployment/pod health) is always fetched fresh below,
+# never hardcoded.
 # ────────────────────────────────────────────────────────────────
 
 PLATFORM = {
@@ -35,20 +47,21 @@ SITES = [
         'category': 'ERP',
         'icon': 'erpnext',
         'brand_color': '#0089FF',
-        'compose_dir': 'frappe-setup',
-        'main_container': 'frappe-erpnext',
-        'containers': [
-            {'name': 'frappe-erpnext', 'role': 'App', 'image': 'frappe/erpnext:latest'},
-            {'name': 'frappe-mariadb', 'role': 'Database', 'image': 'mariadb:10.6'},
-            {'name': 'frappe-redis', 'role': 'Cache / Queue', 'image': 'redis:alpine'},
-        ],
-        'port': 8080,
-        'internal_port': 8000,
-        'domain': 'erpnext.navitrends.ovh',
-        'domain_protocol': 'http',
-        'ssl_configured': False,
+        'namespace': 'erpnext',
+        'main_deployment': 'erpnext-web',
+        'service_port': 8000,
+        'ingress_name': 'erpnext-ingress',
+        'ingress_namespace': 'erpnext',
         'health_path': '/',
-        'volumes': ['frappe-setup_frappe-sites', 'frappe-setup_mariadb-data'],
+        'pvcs': ['erpnext-sites', 'erpnext-mariadb-data'],
+        'roles': {
+            'erpnext-web': 'App',
+            'erpnext-worker': 'Worker',
+            'erpnext-scheduler': 'Scheduler',
+            'erpnext-mariadb': 'Database',
+            'erpnext-redis-cache': 'Cache',
+            'erpnext-redis-queue': 'Queue',
+        },
     },
     {
         'id': 'odoo',
@@ -57,19 +70,18 @@ SITES = [
         'category': 'ERP',
         'icon': 'odoo',
         'brand_color': '#714B67',
-        'compose_dir': 'odoo-clean',
-        'main_container': 'odoo-clean-odoo-1',
-        'containers': [
-            {'name': 'odoo-clean-odoo-1', 'role': 'App', 'image': 'odoo:18'},
-            {'name': 'odoo-clean-db-1', 'role': 'Database', 'image': 'postgres:15'},
-        ],
-        'port': 8069,
-        'internal_port': 8069,
-        'domain': 'odooo.nav.ovh',
-        'domain_protocol': 'https',
-        'ssl_configured': True,
+        'namespace': 'odoo',
+        'main_deployment': 'odoo',
+        'service_port': 8069,
+        # Historical: created in `default` during migration, not the `odoo` ns.
+        'ingress_name': 'odoo-ingress',
+        'ingress_namespace': 'default',
         'health_path': '/web',
-        'volumes': ['odoo-clean_db-data', 'odoo-clean_odoo-etc'],
+        'pvcs': ['odoo-data', 'odoo-postgres-data'],
+        'roles': {
+            'odoo': 'App',
+            'odoo-postgres': 'Database',
+        },
     },
     {
         'id': 'nextcloud',
@@ -78,19 +90,18 @@ SITES = [
         'category': 'Storage',
         'icon': 'nextcloud',
         'brand_color': '#0082C9',
-        'compose_dir': 'nextcloud-setup',
-        'main_container': 'nextcloud-app',
-        'containers': [
-            {'name': 'nextcloud-app', 'role': 'App', 'image': 'nextcloud:latest'},
-            {'name': 'nextcloud-postgres', 'role': 'Database', 'image': 'postgres:15'},
-        ],
-        'port': 8082,
-        'internal_port': 80,
-        'domain': 'next.cloud.nav.ovh',
-        'domain_protocol': 'https',
-        'ssl_configured': True,
+        'namespace': 'nextcloud',
+        'main_deployment': 'nextcloud',
+        'service_port': 80,
+        # Historical: created in `default` during migration, not the `nextcloud` ns.
+        'ingress_name': 'nextcloud-ingress',
+        'ingress_namespace': 'default',
         'health_path': '/status.php',
-        'volumes': ['nextcloud-setup_nextcloud-data', 'nextcloud-setup_nextcloud-db-data'],
+        'pvcs': ['nextcloud-app-data', 'nextcloud-postgres-data'],
+        'roles': {
+            'nextcloud': 'App',
+            'nextcloud-postgres': 'Database',
+        },
     },
     {
         'id': 'mautic',
@@ -99,20 +110,17 @@ SITES = [
         'category': 'Marketing',
         'icon': 'mautic',
         'brand_color': '#4E5E9E',
-        'compose_dir': 'mautic-setup',
-        'main_container': 'mautic-app',
-        'containers': [
-            {'name': 'mautic-app', 'role': 'App', 'image': 'mautic/mautic:latest'},
-            {'name': 'mautic-mariadb', 'role': 'Database', 'image': 'mariadb:10.11'},
-        ],
-        'port': 8081,
-        'internal_port': 80,
-        'domain': None,
-        'domain_protocol': 'http',
-        'ssl_configured': False,
+        'namespace': 'mautic',
+        'main_deployment': 'mautic',
+        'service_port': 80,
+        'ingress_name': 'mautic-ingress',
+        'ingress_namespace': 'mautic',
         'health_path': '/',
-        'volumes': ['mautic-setup_mautic-data', 'mautic-setup_mautic-db-data'],
-        'networks': ['mautic-network'],
+        'pvcs': ['mautic-app-data', 'mautic-db-data'],
+        'roles': {
+            'mautic': 'App',
+            'mautic-mariadb': 'Database',
+        },
     },
     {
         'id': 'n8n',
@@ -121,210 +129,176 @@ SITES = [
         'category': 'Automation',
         'icon': 'n8n',
         'brand_color': '#EA4B71',
-        'compose_dir': 'n8n-setup',
-        'main_container': 'n8n-app',
-        'containers': [
-            {'name': 'n8n-app', 'role': 'App', 'image': 'n8nio/n8n:latest'},
-            {'name': 'n8n-postgres', 'role': 'Database', 'image': 'postgres:15'},
-        ],
-        'port': 5678,
-        'internal_port': 5678,
-        'domain': None,
-        'domain_protocol': 'http',
-        'ssl_configured': False,
+        'namespace': 'n8n',
+        'main_deployment': 'n8n',
+        'service_port': 5678,
+        'ingress_name': 'n8n-ingress',
+        'ingress_namespace': 'n8n',
         'health_path': '/healthz',
-        'volumes': ['n8n-setup_n8n-data', 'n8n-setup_n8n-db-data'],
-        'networks': ['n8n-network', 'integration-network'],
+        'pvcs': ['n8n-data', 'n8n-postgres-data'],
+        'roles': {
+            'n8n': 'App',
+            'n8n-postgres': 'Database',
+        },
     },
 ]
 
 SITE_BY_ID = {s['id']: s for s in SITES}
 
 
-def _build_urls(site):
-    ip = MAIN_SERVER_IP
-    port = site['port']
-    proto = site.get('domain_protocol', 'http')
+def _build_urls(site, ingress_info):
+    """Domain/TLS come from the live Ingress object — never hardcoded.
+    ingress_info is None if the Ingress couldn't be read (RBAC/API issue)
+    or {'host': str|None, 'tls': bool} otherwise."""
     path = site.get('health_path', '/') or '/'
-    ssl_configured = site.get('ssl_configured', proto == 'https')
-
-    domain = site.get('domain')
-    has_domain = bool(domain)
+    host = ingress_info.get('host') if ingress_info else None
+    has_domain = bool(host)
+    ssl_configured = bool(ingress_info and ingress_info.get('tls'))
 
     if has_domain:
-        if ssl_configured:
-            access_url = f"{proto}://{domain}"
-            health_url = f"{access_url}{path}"
-        else:
-            access_url = f"http://{domain}:{port}"
-            health_url = f"{access_url}{path}"
-    else:
-        access_url = f"http://{ip}:{port}"
+        proto = 'https' if ssl_configured else 'http'
+        access_url = f"{proto}://{host}"
         health_url = f"{access_url}{path}"
+    else:
+        access_url = None
+        health_url = None
 
     return {
         'has_domain': has_domain,
-        'domain': domain,
-        'domain_protocol': proto,
+        'domain': host,
+        'domain_protocol': 'https' if ssl_configured else 'http',
         'ssl_configured': ssl_configured,
-        'ip_url': f"http://{ip}:{port}",
-        'access_url': access_url.rstrip('/'),
+        'access_url': (access_url or '').rstrip('/'),
         'health_url': health_url,
     }
 
 
-def _parse_port_mapping(ports_str, host_port, internal_port=None):
-    """Extract host:container mapping from docker ports string."""
-    if not ports_str:
-        if internal_port and internal_port != host_port:
-            return f"{host_port}→{internal_port}"
-        return str(host_port) if host_port else None
-    m = re.search(rf'0\.0\.0\.0:{host_port}->(\d+)/', ports_str)
-    if m:
-        return f"{host_port}→{m.group(1)}"
-    m = re.search(r'0\.0\.0\.0:(\d+)->(\d+)/', ports_str)
-    if m:
-        return f"{m.group(1)}→{m.group(2)}"
-    return ports_str[:60] if ports_str else None
-
-
 def _probe_http(url, timeout=5):
-    """HTTP health probe — local urllib or remote curl via SSH."""
-    if RUNNING_ON_MAIN_SERVER:
-        try:
-            req = Request(url, method='GET', headers={'User-Agent': 'CloudOps-HealthCheck/1.0'})
-            with urlopen(req, timeout=timeout) as resp:
-                code = resp.getcode()
-                return {
-                    'reachable': 200 <= code < 400,
-                    'http_code': code,
-                    'latency_ms': None,
-                    'error': None,
-                }
-        except URLError as e:
-            return {'reachable': False, 'http_code': None, 'latency_ms': None, 'error': str(e.reason)[:120]}
-        except Exception as e:
-            return {'reachable': False, 'http_code': None, 'latency_ms': None, 'error': str(e)[:120]}
-    else:
-        safe_url = url.replace("'", "'\\''")
+    """Direct HTTP probe. The management-platform pod has normal internet
+    egress and resolves public DNS just like any other pod — no need to
+    hop back out to the host via SSH the way the old Docker-era code did."""
+    try:
+        req = urllib.request.Request(url, method='GET', headers={'User-Agent': 'CloudOps-HealthCheck/1.0'})
         t0 = time.time()
-        out, _ = _ssh_main(
-            f"curl -sf -o /dev/null -w '%{{http_code}}' --connect-timeout {timeout} "
-            f"-m {timeout} '{safe_url}' 2>/dev/null || echo '000'",
-            timeout=timeout + 5,
-        )
-        latency = int((time.time() - t0) * 1000)
-        code_str = (out or '000').strip()
-        try:
-            code = int(code_str)
-        except ValueError:
-            code = 0
-        return {
-            'reachable': 200 <= code < 400,
-            'http_code': code if code else None,
-            'latency_ms': latency,
-            'error': None if 200 <= code < 400 else f'HTTP {code_str}',
-        }
+        with urllib.request.urlopen(req, timeout=timeout) as resp:
+            code = resp.getcode()
+            latency_ms = int((time.time() - t0) * 1000)
+            return {
+                'reachable': 200 <= code < 400,
+                'http_code': code,
+                'latency_ms': latency_ms,
+                'error': None,
+            }
+    except Exception as e:
+        return {'reachable': False, 'http_code': None, 'latency_ms': None, 'error': str(e)[:120]}
 
 
-def _container_map():
-    ctrs = get_all_root_containers()
-    return {c['name']: c for c in ctrs}
+def _deployments_for_site(site):
+    """Live Deployment status for every stack member (app/db/cache/etc),
+    keyed by k8s Deployment name."""
+    live = {d['name']: d for d in list_deployments_for_namespace(site['namespace'])}
+    roles = site.get('roles', {})
+    containers = []
+    running_count = 0
+    for dep_name, role in roles.items():
+        d = live.get(dep_name)
+        if d:
+            is_up = d['ready'] >= 1 and d['desired'] >= 1
+            if is_up:
+                running_count += 1
+            containers.append({
+                'name': dep_name,
+                'role': role,
+                'status': 'running' if is_up else 'stopped',
+                'image_live': d['image'],
+                'ports': f"{d['ready']}/{d['desired']} ready",
+            })
+        else:
+            containers.append({
+                'name': dep_name,
+                'role': role,
+                'status': 'not_found',
+                'image_live': '—',
+                'ports': '—',
+            })
+    main = live.get(site['main_deployment'])
+    main_running = bool(main and main['ready'] >= 1 and main['desired'] >= 1)
+    return containers, running_count, main_running
+
+
+def _site_entry(site, include_health):
+    ingress_info = get_ingress_info(site['ingress_name'], site['ingress_namespace'])
+    urls = _build_urls(site, ingress_info)
+    containers, running_count, main_running = _deployments_for_site(site)
+
+    entry = {
+        'id': site['id'],
+        'name': site['name'],
+        'tagline': site['tagline'],
+        'category': site['category'],
+        'icon': site['icon'],
+        'brand_color': site['brand_color'],
+        'compose_dir': site['namespace'],  # k8s namespace, shown as "COMPOSE PROJECT" in the UI
+        'main_container': site['main_deployment'],
+        'port': site['service_port'],
+        'internal_port': site['service_port'],
+        'port_mapping': f"svc/{site['service_port']}",
+        'ports_raw': f"ClusterIP:{site['service_port']}",
+        'volumes': site.get('pvcs', []),
+        'ip_url': f"{site['main_deployment']}.{site['namespace']}.svc.cluster.local:{site['service_port']} (cluster-internal only)",
+        'container_status': 'running' if main_running else 'stopped',
+        'container_running': main_running,
+        'containers': containers,
+        'containers_running': running_count,
+        'containers_total': len(site.get('roles', {})),
+        **urls,
+    }
+
+    if include_health:
+        if not urls['has_domain']:
+            entry['health'] = {
+                'status': 'down', 'reachable': False, 'http_code': None,
+                'latency_ms': None, 'error': 'No Ingress host found for this app',
+            }
+        elif not main_running:
+            entry['health'] = {
+                'status': 'offline', 'reachable': False, 'http_code': None,
+                'latency_ms': None, 'error': 'Deployment not running',
+            }
+        else:
+            probe = _probe_http(urls['health_url'])
+            entry['health'] = {'status': 'up' if probe['reachable'] else 'down', **probe}
+
+    return entry
 
 
 def get_sites_list(include_health=False):
-    """Return all sites with live container + URL metadata."""
-    ctr_map = _container_map()
-    results = []
-
-    for site in SITES:
-        urls = _build_urls(site)
-        main_name = site['main_container']
-        main_ctr = ctr_map.get(main_name, {})
-        main_status = get_container_status(main_name) if main_name else {'status': 'unknown'}
-
-        containers_live = []
-        running_count = 0
-        for ctr_def in site.get('containers', []):
-            name = ctr_def['name']
-            live = ctr_map.get(name, {})
-            status_raw = live.get('status', '')
-            is_up = 'Up' in status_raw
-            if is_up:
-                running_count += 1
-            containers_live.append({
-                **ctr_def,
-                'status': 'running' if is_up else ('stopped' if live else 'not_found'),
-                'status_raw': status_raw or 'not found',
-                'image_live': live.get('image', ctr_def.get('image', '—')),
-                'ports': live.get('ports', '—'),
-            })
-
-        entry = {
-            'id': site['id'],
-            'name': site['name'],
-            'tagline': site['tagline'],
-            'category': site['category'],
-            'icon': site['icon'],
-            'brand_color': site['brand_color'],
-            'compose_dir': site['compose_dir'],
-            'main_container': main_name,
-            'port': site['port'],
-            'internal_port': site.get('internal_port', site['port']),
-            'port_mapping': _parse_port_mapping(
-                main_ctr.get('ports', ''), site['port'], site.get('internal_port')
-            ),
-            'ports_raw': main_ctr.get('ports', '—'),
-            'volumes': site.get('volumes', []),
-            'networks': site.get('networks', ['default']),
-            'container_status': main_status['status'],
-            'container_running': main_status['status'] == 'running',
-            'containers': containers_live,
-            'containers_running': running_count,
-            'containers_total': len(site.get('containers', [])),
-            **urls,
-        }
-
-        if include_health:
-            if entry['container_running']:
-                probe = _probe_http(urls['health_url'])
-                entry['health'] = {
-                    'status': 'up' if probe['reachable'] else 'down',
-                    **probe,
-                }
-            else:
-                entry['health'] = {
-                    'status': 'offline',
-                    'reachable': False,
-                    'http_code': None,
-                    'latency_ms': None,
-                    'error': 'Container not running',
-                }
-
-        results.append(entry)
-
-    return results
+    """Return all sites with live k8s Deployment + Ingress state."""
+    return [_site_entry(site, include_health) for site in SITES]
 
 
 def get_site_health(site_id):
     site = SITE_BY_ID.get(site_id)
     if not site:
         return None
-    urls = _build_urls(site)
-    status = get_container_status(site['main_container'])
-    if status['status'] != 'running':
+    ingress_info = get_ingress_info(site['ingress_name'], site['ingress_namespace'])
+    urls = _build_urls(site, ingress_info)
+    _, _, main_running = _deployments_for_site(site)
+
+    if not main_running:
         return {
-            'site_id': site_id,
-            'status': 'offline',
-            'reachable': False,
-            'http_code': None,
-            'latency_ms': None,
-            'error': 'Container not running',
+            'site_id': site_id, 'status': 'offline', 'reachable': False,
+            'http_code': None, 'latency_ms': None, 'error': 'Deployment not running',
             'checked_at': time.time(),
         }
-    t0 = time.time()
+    if not urls['has_domain']:
+        return {
+            'site_id': site_id, 'status': 'down', 'reachable': False,
+            'http_code': None, 'latency_ms': None, 'error': 'No Ingress host found',
+            'checked_at': time.time(),
+        }
+
     probe = _probe_http(urls['health_url'])
-    probe['latency_ms'] = probe.get('latency_ms') or int((time.time() - t0) * 1000)
     probe['site_id'] = site_id
     probe['status'] = 'up' if probe['reachable'] else 'down'
     probe['checked_at'] = time.time()

platform/modules/users.py

@@ -7,7 +7,8 @@ import json
 
 from config import (
     RUNNING_ON_MAIN_SERVER,
-    MAIN_SERVER_IP, MAIN_SERVER_USER, MAIN_SERVER_KEY, MAIN_SERVER_PORT,
+    MAIN_SERVER_USER, MAIN_SERVER_KEY,
+    MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT,
 )
 
 
@@ -33,10 +34,10 @@ def _ssh_main(remote_cmd, timeout=30):
         # Escape single quotes in remote_cmd for safe shell wrapping
         escaped = remote_cmd.replace("'", "'\\''")
         ssh = (
-            f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_PORT} "
+            f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_SSH_PORT} "
             f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 "
             f"-o BatchMode=yes "
-            f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}"
+            f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}"
         )
         return _run(f"{ssh} '{escaped}'", timeout=timeout)
 

platform/requirements.txt

@@ -1,4 +1,5 @@
 boto3==1.43.5
 flask
 psutil
-kubernetes
+kubernetes
+ansible-core

platform/static/css/style.css

@@ -881,7 +881,7 @@ a.sidebar-brand-link {
   box-shadow: var(--glow-accent);
 }
 
-.backup-item-main { min-width: 0; padding-bottom: 12px; }
+.backup-item-main { min-width: 0; padding-bottom: 12px; display: flex; align-items: center; gap: 8px; }
 .backup-item:not(:has(.backup-actions)) .backup-item-main { padding-bottom: 0; }
 
 .backup-name {
@@ -894,6 +894,18 @@ a.sidebar-brand-link {
 }
 .backup-name i { margin-right: 6px; opacity: 0.85; }
 
+/* Backup status dot — green/yellow/red from the .meta.json sidecar
+   backup-k8s-apps.sh writes; gray for backups with no sidecar (legacy
+   archives, or anything made before this feature shipped). */
+.status-dot {
+  width: 8px; height: 8px; border-radius: 50%; flex-shrink: 0;
+  border: 1px solid rgba(0,0,0,0.15);
+}
+.status-dot-green  { background: var(--green); box-shadow: 0 0 6px rgba(45,212,122,0.5); }
+.status-dot-yellow { background: var(--yellow); box-shadow: 0 0 6px rgba(245,185,66,0.5); }
+.status-dot-red    { background: var(--red); box-shadow: 0 0 6px rgba(240,82,82,0.5); }
+.status-dot-unknown { background: var(--text3); box-shadow: none; }
+
 .backup-meta {
   display: block;
   margin-top: 6px;

platform/static/js/platform.js

@@ -170,7 +170,10 @@ async function refreshSystemMetrics() {
     if (el('g-disk')) el('g-disk').style.width = Math.min(parseFloat(d.disk_pct) || 0, 100) + '%';
     setText('uptime-chip', d.uptime);
     if (el('settings-uptime')) el('settings-uptime').value = d.uptime;
-    if (d.hostname && el('this-server-desc')) el('this-server-desc').textContent = d.hostname;
+    const localTargetRadio = document.querySelector('input[name="restore_target"][value="local"]');
+    if (d.hostname && el('this-server-desc') && !(localTargetRadio && localTargetRadio.disabled)) {
+      el('this-server-desc').textContent = d.hostname;
+    }
   } catch (_) {}
 }
 
@@ -377,8 +380,8 @@ async function refreshBackupsList() {
     renderBackupList(d.vm, 'vm-backup-list', 'vm');
     if (el('stat-local-bk')) el('stat-local-bk').textContent = d.local.length;
     if (el('stat-vm-bk')) el('stat-vm-bk').textContent = d.vm.length;
-    if (el('local-options')) el('local-options').innerHTML = d.local.length ? d.local.map((b) => `<option value="${b}" data-source="local">${b}</option>`).join('') : '<option disabled>No local backups</option>';
-    if (el('vm-options')) el('vm-options').innerHTML = d.vm.length ? d.vm.map((b) => `<option value="${b}" data-source="vm">${b}</option>`).join('') : '<option disabled>No VM backups</option>';
+    if (el('local-options')) el('local-options').innerHTML = d.local.length ? d.local.map((b) => `<option value="${b.name}" data-source="local">${b.name}</option>`).join('') : '<option disabled>No local backups</option>';
+    if (el('vm-options')) el('vm-options').innerHTML = d.vm.length ? d.vm.map((b) => `<option value="${b.name}" data-source="vm">${b.name}</option>`).join('') : '<option disabled>No VM backups</option>';
   } catch (_) {}
 }
 
@@ -394,17 +397,28 @@ function renderBackupList(items, id, source) {
     return;
   }
   node.innerHTML = items.map((b) => {
-    const safe = jsStr(b);
+    const name = b.name;
+    const safe = jsStr(name);
+    const status = b.status || 'unknown';
+    const titleParts = [status];
+    if (b.apps && b.apps.length) titleParts.push(`apps: ${b.apps.join(', ')}`);
+    if (b.size_human) titleParts.push(b.size_human);
+    const title = escapeHtml(titleParts.join(' · '));
     const uploadBtn = source === 'local' && typeof quickUploadR2 === 'function'
       ? `<button class="btn btn-ghost btn-sm" onclick="quickUploadR2('${safe}',this)" title="Upload to R2"><i class="fas fa-cloud-upload-alt"></i></button>`
       : '';
     const detailsBtn = typeof showBackupDetails === 'function'
       ? `<button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('${source}','${safe}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>`
       : '';
+    const appsLabel = (b.apps && b.apps.length)
+      ? `<span class="badge" style="background:rgba(59,130,246,0.12);color:var(--accent2);font-size:10px;white-space:nowrap;">${b.apps.length >= 5 ? 'All apps' : escapeHtml(b.apps.join(', '))}</span>`
+      : '';
     return `
-    <div class="backup-item" id="bk-item-${source}-${b.replace(/[^a-z0-9]/gi, '_')}">
+    <div class="backup-item" id="bk-item-${source}-${name.replace(/[^a-z0-9]/gi, '_')}">
       <div class="backup-item-main">
-        <span class="backup-name">${escapeHtml(b)}</span>
+        <span class="status-dot status-dot-${status}" title="${title}"></span>
+        <span class="backup-name">${escapeHtml(name)}</span>
+        ${appsLabel}
       </div>
       <div class="backup-actions">
         ${detailsBtn}
@@ -479,7 +493,14 @@ async function runManualBackup() {
   if (wrapper) wrapper.style.display = '';
   if (logEl) logEl.innerHTML = '';
   try {
-    const r = await fetch('/api/backups/run', { method: 'POST' });
+    const selectedApps = Array.from(
+      document.querySelectorAll('#backup-apps-checkbox-group input[type="checkbox"]:checked')
+    ).map((cb) => cb.value);
+    const r = await fetch('/api/backups/run', {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ apps: selectedApps }),
+    });
     const d = await r.json();
     if (!d.success) throw new Error(d.message || 'Failed');
     manualBackupJobId = d.job_id;

platform/templates/pages/backups.html

@@ -252,7 +252,52 @@
     <div class="card-title"><i class="fas fa-shield-halved"></i> Manual Backup</div>
   </div>
   <div style="padding:4px 0 12px;">
-    <p style="color:var(--text2);font-size:13px;margin-bottom:14px;">Manually trigger a backup of all containers</p>
+    <p style="color:var(--text2);font-size:13px;margin-bottom:10px;">
+      All apps are backed up together into one archive by default. Uncheck any app to leave it out —
+      useful for a quick one-off backup of a single app before a risky change, without waiting on the rest.
+    </p>
+    <div class="app-picker-grid" id="backup-apps-checkbox-group" style="margin-bottom:14px;">
+      <label class="app-picker-card">
+        <input type="checkbox" value="frappe" checked>
+        <div class="app-picker-body">
+          <span class="app-picker-icon">🧾</span>
+          <div><div class="app-picker-name">Frappe / ERPNext</div></div>
+          <span class="app-picker-check"><i class="fas fa-check"></i></span>
+        </div>
+      </label>
+      <label class="app-picker-card">
+        <input type="checkbox" value="odoo" checked>
+        <div class="app-picker-body">
+          <span class="app-picker-icon">🟣</span>
+          <div><div class="app-picker-name">Odoo</div></div>
+          <span class="app-picker-check"><i class="fas fa-check"></i></span>
+        </div>
+      </label>
+      <label class="app-picker-card">
+        <input type="checkbox" value="nextcloud" checked>
+        <div class="app-picker-body">
+          <span class="app-picker-icon">☁️</span>
+          <div><div class="app-picker-name">Nextcloud</div></div>
+          <span class="app-picker-check"><i class="fas fa-check"></i></span>
+        </div>
+      </label>
+      <label class="app-picker-card">
+        <input type="checkbox" value="mautic" checked>
+        <div class="app-picker-body">
+          <span class="app-picker-icon">📣</span>
+          <div><div class="app-picker-name">Mautic</div></div>
+          <span class="app-picker-check"><i class="fas fa-check"></i></span>
+        </div>
+      </label>
+      <label class="app-picker-card">
+        <input type="checkbox" value="n8n" checked>
+        <div class="app-picker-body">
+          <span class="app-picker-icon">🔁</span>
+          <div><div class="app-picker-name">n8n</div></div>
+          <span class="app-picker-check"><i class="fas fa-check"></i></span>
+        </div>
+      </label>
+    </div>
     <button class="btn btn-primary" onclick="runManualBackup()" id="manual-backup-btn">
       <i class="fas fa-play"></i> Run Backup Now
     </button>
@@ -286,14 +331,20 @@
         {% for b in backups %}
         <div class="backup-item">
           <div class="backup-item-main">
-            <span class="backup-name">{{ b }}</span>
+            <span class="status-dot status-dot-{{ b.status }}" title="{{ b.status }}{% if b.apps %} · apps: {{ b.apps|join(', ') }}{% endif %}{% if b.size_human %} · {{ b.size_human }}{% endif %}"></span>
+            <span class="backup-name">{{ b.name }}</span>
+            {% if b.apps %}
+            <span class="badge" style="background:rgba(59,130,246,0.12);color:var(--accent2);font-size:10px;white-space:nowrap;">
+              {% if b.apps|length >= 5 %}All apps{% else %}{{ b.apps|join(', ') }}{% endif %}
+            </span>
+            {% endif %}
           </div>
           <div class="backup-actions">
-            <button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('local','{{ b }}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>
-            <button class="btn btn-ghost btn-sm btn-audit" onclick="auditBackup('local','{{ b }}',this)"><i class="fas fa-shield-check"></i> Audit</button>
-            <button class="btn btn-ghost btn-sm" onclick="quickRestore('local','{{ b }}')">↩ Restore</button>
-            <button class="btn btn-ghost btn-sm" onclick="quickUploadR2('{{ b }}',this)" title="Upload to R2"><i class="fas fa-cloud-upload-alt"></i></button>
-            <button class="btn btn-ghost btn-sm btn-delete-backup" onclick="deleteBackup('local','{{ b }}',this)"><i class="fas fa-trash"></i></button>
+            <button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('local','{{ b.name }}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>
+            <button class="btn btn-ghost btn-sm btn-audit" onclick="auditBackup('local','{{ b.name }}',this)"><i class="fas fa-shield-check"></i> Audit</button>
+            <button class="btn btn-ghost btn-sm" onclick="quickRestore('local','{{ b.name }}')">↩ Restore</button>
+            <button class="btn btn-ghost btn-sm" onclick="quickUploadR2('{{ b.name }}',this)" title="Upload to R2"><i class="fas fa-cloud-upload-alt"></i></button>
+            <button class="btn btn-ghost btn-sm btn-delete-backup" onclick="deleteBackup('local','{{ b.name }}',this)"><i class="fas fa-trash"></i></button>
           </div>
         </div>
         {% else %}<div class="empty-state"><i class="fas fa-inbox"></i>No backups</div>{% endfor %}
@@ -310,13 +361,19 @@
         {% for b in vm_backups %}
         <div class="backup-item">
           <div class="backup-item-main">
-            <span class="backup-name">{{ b }}</span>
+            <span class="status-dot status-dot-{{ b.status }}" title="{{ b.status }}{% if b.apps %} · apps: {{ b.apps|join(', ') }}{% endif %}{% if b.size_human %} · {{ b.size_human }}{% endif %}"></span>
+            <span class="backup-name">{{ b.name }}</span>
+            {% if b.apps %}
+            <span class="badge" style="background:rgba(59,130,246,0.12);color:var(--accent2);font-size:10px;white-space:nowrap;">
+              {% if b.apps|length >= 5 %}All apps{% else %}{{ b.apps|join(', ') }}{% endif %}
+            </span>
+            {% endif %}
           </div>
           <div class="backup-actions">
-            <button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('vm','{{ b }}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>
-            <button class="btn btn-ghost btn-sm btn-audit" onclick="auditBackup('vm','{{ b }}',this)"><i class="fas fa-shield-check"></i> Audit</button>
-            <button class="btn btn-ghost btn-sm" onclick="quickRestore('vm','{{ b }}')">↩ Restore</button>
-            <button class="btn btn-ghost btn-sm btn-delete-backup" onclick="deleteBackup('vm','{{ b }}',this)"><i class="fas fa-trash"></i></button>
+            <button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('vm','{{ b.name }}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>
+            <button class="btn btn-ghost btn-sm btn-audit" onclick="auditBackup('vm','{{ b.name }}',this)"><i class="fas fa-shield-check"></i> Audit</button>
+            <button class="btn btn-ghost btn-sm" onclick="quickRestore('vm','{{ b.name }}')">↩ Restore</button>
+            <button class="btn btn-ghost btn-sm btn-delete-backup" onclick="deleteBackup('vm','{{ b.name }}',this)"><i class="fas fa-trash"></i></button>
           </div>
         </div>
         {% else %}<div class="empty-state"><i class="fas fa-inbox"></i>No VM backups</div>{% endfor %}

platform/templates/pages/restore.html

@@ -63,26 +63,37 @@
 
     <div class="form-section">
       <div class="form-section-title">STEP 2 — SELECT RESTORE TARGET</div>
+      {% if not running_on_main_server %}
+      <div style="margin-bottom:12px;padding:10px 14px;background:rgba(245,158,11,0.08);
+        border:1px solid rgba(245,158,11,0.25);border-radius:8px;font-size:12px;color:var(--yellow);">
+        <i class="fas fa-triangle-exclamation"></i>
+        Running as the standby — there's no local cluster here to restore into.
+        "External Machine" is pre-filled to reach the real main server over the standby's
+        SSH tunnel; only change these if you actually mean a different target.
+      </div>
+      {% endif %}
       <div class="radio-group">
-        <label class="radio-card">
-          <input type="radio" name="restore_target" value="local" checked onchange="toggleRemoteFields()">
-          <div class="radio-body"><span class="radio-icon">🎯</span><div><div class="radio-label">Restore on This Server</div><div class="radio-desc" id="this-server-desc">Loading hostname…</div></div></div>
+        <label class="radio-card{% if not running_on_main_server %} disabled{% endif %}">
+          <input type="radio" name="restore_target" value="local"
+            {% if running_on_main_server %}checked{% else %}disabled{% endif %}
+            onchange="toggleRemoteFields()">
+          <div class="radio-body"><span class="radio-icon">🎯</span><div><div class="radio-label">Restore on This Server</div><div class="radio-desc" id="this-server-desc">{% if running_on_main_server %}Loading hostname…{% else %}Not available on the standby{% endif %}</div></div></div>
         </label>
         <label class="radio-card">
-          <input type="radio" name="restore_target" value="remote" onchange="toggleRemoteFields()">
+          <input type="radio" name="restore_target" value="remote" {% if not running_on_main_server %}checked{% endif %} onchange="toggleRemoteFields()">
           <div class="radio-body"><span class="radio-icon">📡</span><div><div class="radio-label">External Machine</div><div class="radio-desc">via SSH — any IP</div></div></div>
         </label>
       </div>
 
-      <div id="remote-fields" style="display:none; margin-top:16px; max-width:560px;">
+      <div id="remote-fields" style="display:{% if running_on_main_server %}none{% else %}block{% endif %}; margin-top:16px; max-width:560px;">
         <div class="form-row">
           <div class="form-group">
             <label class="form-label">TARGET IP</label>
-            <input type="text" id="remote-ip" class="form-input" placeholder="192.168.x.x or IP">
+            <input type="text" id="remote-ip" class="form-input" placeholder="192.168.x.x or IP" value="{{ tunnel_remote_ip if not running_on_main_server else '' }}">
           </div>
           <div class="form-group" style="max-width:100px;">
             <label class="form-label">SSH PORT</label>
-            <input type="text" id="remote-port" class="form-input" value="22">
+            <input type="text" id="remote-port" class="form-input" value="{{ tunnel_remote_port if not running_on_main_server else '22' }}">
           </div>
           <div class="form-group" style="max-width:120px;">
             <label class="form-label">SSH USER</label>

scripts/sync-standby-platform.sh

@@ -0,0 +1,145 @@
+#!/bin/bash
+# ─────────────────────────────────────────────────────────────
+# sync-standby-platform.sh
+# Keeps the warm-standby copy of management-platform on the VM
+# (178.18.243.51) in sync with the real source of truth on this
+# server (/root/CloudOps/platform), pushed directly over SSH.
+#
+# Replaces the old pull-platform-backup.sh + deploy-platform.sh
+# pair on the VM, which pulled/redeployed a platform-backup-*.tar.gz
+# tarball of /root/management-platform (the pre-migration Docker-era
+# deploy dir). That tarball generation was disabled on 2026-08-14
+# when the platform's source of truth moved to git-tracked
+# /root/CloudOps + Jenkins — nothing replaced it on the VM side, so
+# the standby silently kept re-deploying its last cached snapshot
+# (found stale at 2+ months old during PFE failover investigation).
+#
+# config.py is intentionally NOT sourced from /root/CloudOps/platform
+# (gitignored there, and can drift — see that file's own history).
+# The live /root/management-platform/config.py on THIS server is the
+# real one actually mounted into the running pod; it already contains
+# the RUNNING_ON_MAIN_SERVER auto-detect logic that makes the exact
+# same file behave correctly on both this server and the VM, so it's
+# copied byte-for-byte rather than templated.
+#
+# Run hourly via cron on this server (NOT on the VM — the VM has no
+# reason to pull, this server already pushes everywhere else it
+# manages, e.g. backup-k8s-apps.sh's VM tier).
+# ─────────────────────────────────────────────────────────────
+
+VM_HOST="178.18.243.51"
+VM_PORT="22"
+VM_USER="root"
+VM_KEY="/root/.ssh/id_rsa"
+VM_DEPLOY_DIR="/root/management-platform"
+
+LOCAL_SRC="/root/CloudOps/platform/"
+LIVE_CONFIG="/root/management-platform/config.py"
+
+# restore_start()/api_backup_run() in app.py reference this as a fixed
+# absolute path (/root/CloudOps/backup/restore-k8s-apps.sh) — it's a
+# sibling of platform/, never baked into the image, reached via the same
+# rw hostPath mount as everything else. Restoring FROM the standby (target
+# = remote, pointed at the main server over the tunnel) needs a local copy
+# here to scp across, so mirror it too, not just platform/.
+LOCAL_BACKUP_SRC="/root/CloudOps/backup/"
+VM_BACKUP_DIR="/root/CloudOps/backup"
+
+# Same reason as backup/ above — restore_start() references this as a
+# fixed absolute path (/root/CloudOps/ansible/dr-bootstrap.yml) to drive
+# every "External Machine" restore now, not just true from-scratch DR.
+LOCAL_ANSIBLE_SRC="/root/CloudOps/ansible/"
+VM_ANSIBLE_DIR="/root/CloudOps/ansible"
+
+LOG="/root/CloudOps/scripts/sync-standby-platform.log"
+
+SSH_OPTS="-i $VM_KEY -p $VM_PORT -o StrictHostKeyChecking=no -o ConnectTimeout=15"
+SCP_OPTS="-i $VM_KEY -P $VM_PORT -o StrictHostKeyChecking=no -o ConnectTimeout=15"
+
+log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $1" | tee -a "$LOG"; }
+
+log "🔄 Syncing platform code → $VM_USER@$VM_HOST:$VM_DEPLOY_DIR"
+
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "mkdir -p $VM_DEPLOY_DIR" || {
+    log "❌ Cannot reach VM — aborting"
+    exit 1
+}
+
+# --itemize-changes so we can tell below whether anything actually
+# changed (drives the restart-only-if-needed decision).
+CHANGES=$(rsync -a --delete -e "ssh $SSH_OPTS" \
+    --exclude venv --exclude __pycache__ --exclude '*.pyc' \
+    --exclude config.py --exclude '*.log' \
+    --itemize-changes \
+    "$LOCAL_SRC" "$VM_USER@$VM_HOST:$VM_DEPLOY_DIR/" 2>>"$LOG")
+RSYNC_RC=$?
+
+if [ $RSYNC_RC -ne 0 ]; then
+    log "❌ rsync failed (rc=$RSYNC_RC)"
+    exit 1
+fi
+
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "mkdir -p $VM_BACKUP_DIR"
+BACKUP_CHANGES=$(rsync -a --delete -e "ssh $SSH_OPTS" \
+    --itemize-changes \
+    "$LOCAL_BACKUP_SRC" "$VM_USER@$VM_HOST:$VM_BACKUP_DIR/" 2>>"$LOG")
+if [ $? -ne 0 ]; then
+    log "❌ rsync of backup/ failed"
+    exit 1
+fi
+CHANGES="${CHANGES}${BACKUP_CHANGES}"
+
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "mkdir -p $VM_ANSIBLE_DIR"
+ANSIBLE_CHANGES=$(rsync -a --delete -e "ssh $SSH_OPTS" \
+    --exclude tmp --exclude '*.retry' \
+    --itemize-changes \
+    "$LOCAL_ANSIBLE_SRC" "$VM_USER@$VM_HOST:$VM_ANSIBLE_DIR/" 2>>"$LOG")
+if [ $? -ne 0 ]; then
+    log "❌ rsync of ansible/ failed"
+    exit 1
+fi
+CHANGES="${CHANGES}${ANSIBLE_CHANGES}"
+
+CONFIG_CHANGED=""
+if ! scp $SCP_OPTS "$LIVE_CONFIG" "$VM_USER@$VM_HOST:$VM_DEPLOY_DIR/config.py.new" &>>"$LOG"; then
+    log "❌ Could not copy live config.py to VM"
+    exit 1
+fi
+if ! ssh $SSH_OPTS "$VM_USER@$VM_HOST" \
+    "cmp -s $VM_DEPLOY_DIR/config.py $VM_DEPLOY_DIR/config.py.new 2>/dev/null"; then
+    CONFIG_CHANGED="yes"
+fi
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "mv $VM_DEPLOY_DIR/config.py.new $VM_DEPLOY_DIR/config.py"
+
+if [ -z "$CHANGES" ] && [ -z "$CONFIG_CHANGED" ]; then
+    log "✅ Already up to date — nothing changed, no restart needed"
+    exit 0
+fi
+
+log "📝 Changes detected:"
+echo "$CHANGES" | tee -a "$LOG"
+[ -n "$CONFIG_CHANGED" ] && log "   (config.py changed too)"
+
+log "📦 Syncing venv deps (requirements.txt) ..."
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "
+    cd $VM_DEPLOY_DIR
+    chmod +x *.sh 2>/dev/null
+    chmod +x *.py 2>/dev/null
+    if [ ! -d venv ]; then
+        python3 -m venv venv
+        venv/bin/python -m ensurepip --upgrade 2>/dev/null || true
+    fi
+    venv/bin/pip install --quiet -r requirements.txt
+" >>"$LOG" 2>&1
+
+log "🔁 Restarting management-platform service on VM ..."
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "systemctl restart management-platform"
+sleep 3
+STATUS=$(ssh $SSH_OPTS "$VM_USER@$VM_HOST" "systemctl is-active management-platform 2>/dev/null")
+
+if [ "$STATUS" = "active" ]; then
+    log "✅ Standby platform restarted and running latest code"
+else
+    log "⚠️  Service not active after restart (status: $STATUS) — check journalctl -u management-platform on the VM"
+    exit 1
+fi