Make restore-from-standby actually work, not just look wired

Two gaps found testing this live from the standby: 1. restore.html's "Restore on This Server" was checked by default regardless of RUNNING_ON_MAIN_SERVER — on the standby that option is nonsensical (no local cluster/kubectl at all) and restore_start() would have just tried and failed confusingly. Now: that radio is disabled with an explanatory note when not on the main server, "External Machine" is checked instead and pre-filled with the tunnel details (localhost:2224, contabo-key) so restoring from the standby just targets the real main server without the user having to know any of that. Added a matching server-side guard in restore_start() for target=='local' + not RUNNING_ON_MAIN_SERVER (defense in depth — the UI already prevents it, this catches a direct API call too). Also fixed refreshSystemMetrics() in platform.js, which would have overwritten the disabled option's label with the (now-correct, see previous commit) main-server hostname — looking like a working local target when it isn't. 2. sync-standby-platform.sh only ever mirrored platform/ — but restore_start() references /root/CloudOps/backup/restore-k8s-apps.sh as a fixed absolute path to scp to the remote target, and that directory never existed on the VM at all. Every restore attempt from the standby failed immediately with "restore-k8s-apps.sh not found", regardless of target. Now mirrors /root/CloudOps/backup/ too. Verified end-to-end for real: triggered a restore of frappe/erpnext from the standby's actual web UI (target=remote, localhost:2224) — connected over the tunnel, copied the backup archive + script to the main server, ran restore-k8s-apps.sh there, scaled the deployment down/up, restored the DB. Confirmed after: 740 tables in the DB, /api/method/ping responding on the live pod. Not a dry run — a real restore, actually initiated from the standby machine.
Add SSH+kubectl remote fallback so the standby actually shows cluster/app health
2026-08-21 13:55:43 +02:00 · 2026-08-21 13:50:02 +02:00 · 2026-08-21 13:26:07 +02:00 · 2026-08-21 13:04:05 +02:00 · 2026-08-21 12:37:16 +02:00 · 2026-08-21 11:07:02 +02:00
12 changed files with 818 additions and 62 deletions
--- a/backup/backup-k8s-apps.sh
+++ b/backup/backup-k8s-apps.sh
@@ -96,6 +96,15 @@ declare -A SECRET_NAME=( [n8n]=n8n-secrets    [odoo]=odoo-secrets    [mautic]=ma
 declare -A DB_USER_KEY=( [n8n]=DB_POSTGRESDB_USER [odoo]=POSTGRES_USER [mautic]=MYSQL_USER [nextcloud]=POSTGRES_USER [frappe]="" )
 declare -A DB_PASS_KEY=( [n8n]=DB_POSTGRESDB_PASSWORD [odoo]=POSTGRES_PASSWORD [mautic]=MYSQL_ROOT_PASSWORD [nextcloud]=POSTGRES_PASSWORD [frappe]=MARIADB_ROOT_PASSWORD )

+# Reproducible framework/vendor code that lives on the same PVC as real user
+# data for these two apps (single shared mount, no separate volume). Both
+# images bake these dirs into their own filesystem layer independent of any
+# volume — Mautic has no logic to reconstruct them, so restore-k8s-apps.sh
+# re-seeds them straight from the app's own image instead of from backup.
+# Never pull request/exclude changes here without updating the matching
+# RECONSTRUCT_DIRS table in restore-k8s-apps.sh.
+declare -A TAR_EXCLUDES=( [mautic]="node_modules vendor" [nextcloud]="apps core dist 3rdparty lib" )
+
 ALL_APPS="n8n odoo mautic nextcloud frappe"

 BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
@@ -174,6 +183,83 @@ check_resources() {
    fi
 }

+# --------------------------------------------------
+# Per-backup status metadata sidecar (.meta.json) — powers the UI's
+# green/yellow/red status dot without ever needing to decompress the
+# (possibly 600MB+) archive itself. Written only after all 3 tiers are
+# known, then best-effort mirrored to VM/R2 alongside the archive, same
+# as the .sha256 sidecar already is — so status is readable from
+# whichever tier the UI is listing, not just local.
+# --------------------------------------------------
+write_meta_sidecar() {
+    local size_bytes meta_path
+    size_bytes=$(stat -c%s "$BACKUP_ARCHIVE" 2>/dev/null || echo 0)
+    meta_path="/root/backups/${BACKUP_NAME}.tar.gz.meta.json"
+
+    python3 - "$BACKUP_NAME" "$size_bytes" "$TIER1_STATUS" "$TIER2_STATUS" "$TIER3_STATUS" \
+        "${#INCLUDED_APPS[@]}" "${INCLUDED_APPS[@]}" "${ERRORS[@]}" > "$meta_path" <<'PYEOF'
+import sys, json, datetime
+
+backup_name  = sys.argv[1]
+size_bytes   = int(sys.argv[2])
+tier1, tier2, tier3 = sys.argv[3], sys.argv[4], sys.argv[5]
+n_apps       = int(sys.argv[6])
+apps         = sys.argv[7:7 + n_apps]
+errors_raw   = sys.argv[7 + n_apps:]
+
+def norm_tier(s):
+    # "ok:572M" -> "ok"; "failed:rc=2" / "skipped:no_credentials" pass through
+    # unchanged so the UI can tell a real failure apart from a deliberate skip.
+    return 'ok' if s.startswith('ok') else s
+
+errors = []
+for e in errors_raw:
+    parts = e.split(':', 2)
+    errors.append({
+        'app':    parts[0] if len(parts) > 0 else '',
+        'step':   parts[1] if len(parts) > 1 else '',
+        'detail': parts[2] if len(parts) > 2 else '',
+    })
+
+meta = {
+    'backup_name': backup_name,
+    'created_at':  datetime.datetime.now(datetime.timezone.utc).isoformat(timespec='seconds'),
+    'apps':        apps,
+    'apps_key':    ','.join(sorted(apps)),
+    'size_bytes':  size_bytes,
+    'errors':      errors,
+    'tiers': {'local': norm_tier(tier1), 'vm': norm_tier(tier2), 'r2': norm_tier(tier3)},
+}
+json.dump(meta, sys.stdout, indent=2)
+PYEOF
+
+    if [ -s "$meta_path" ]; then
+        echo "   ✅ Metadata sidecar written: $(basename "$meta_path")"
+    else
+        echo "   ⚠️  Metadata sidecar write failed"
+        return 1
+    fi
+
+    # Best-effort mirror to whichever tiers actually succeeded — never fatal,
+    # the local sidecar (already written above) is what matters most.
+    if [ "$TIER2_STATUS" = "ok" ]; then
+        scp -i "$VM_KEY" -P "$VM_PORT" -o StrictHostKeyChecking=no -o ConnectTimeout=15 \
+            "$meta_path" "${VM_USER}@${VM_HOST}:${VM_DEST}" &>/dev/null || true
+    fi
+    if [ "$TIER3_STATUS" = "ok" ] && [ -n "${AWS_ACCESS_KEY_ID:-}" ]; then
+        META_SCRIPT="/tmp/r2_meta_$$.py"
+        cat > "$META_SCRIPT" << PYEOF2
+import boto3
+client = boto3.client('s3', endpoint_url="${R2_ENDPOINT}",
+    aws_access_key_id="${AWS_ACCESS_KEY_ID}", aws_secret_access_key="${AWS_SECRET_ACCESS_KEY}",
+    region_name='auto')
+client.upload_file("$meta_path", "${R2_BUCKET}", "backups/${BACKUP_NAME}.tar.gz.meta.json")
+PYEOF2
+        "$PYTHON_BIN" "$META_SCRIPT" &>/dev/null || true
+        rm -f "$META_SCRIPT"
+    fi
+}
+
 echo "========================================="
 echo "📦 Starting k8s Backup: $BACKUP_NAME"
 if [ "$APPS_FILTER" = "all" ]; then
@@ -189,8 +275,11 @@ mkdir -p "$BACKUP_DIR" "/root/backups"
 cd "$BACKUP_DIR"

 APP_COUNT=0
+INCLUDED_APPS=()   # apps actually attempted this run — feeds the .meta.json sidecar's apps_key
+ERRORS=()          # "app:step:detail" entries — any entry here makes the sidecar's status red
 for app in $ALL_APPS; do
    app_selected "$app" || continue
+    INCLUDED_APPS+=("$app")
    ns="${NS[$app]}"
    app_deploy="${APP_DEPLOY[$app]}"
    app_ctr="${APP_CTR[$app]}"
@@ -214,13 +303,15 @@ for app in $ALL_APPS; do
    echo -n "   📄 Manifests (deployment/service/configmap/ingress) ... "
    kubectl get deployment,service,configmap,ingress -n "$ns" -l owner=ameni-boukattaya -o yaml \
        > "$APP_DIR/manifests.yaml" 2>"$APP_DIR/manifests.err" \
-        && { echo "✅"; rm -f "$APP_DIR/manifests.err"; } || echo "⚠️  FAILED (see manifests.err)"
+        && { echo "✅"; rm -f "$APP_DIR/manifests.err"; } \
+        || { echo "⚠️  FAILED (see manifests.err)"; ERRORS+=("$app:manifests:$(tr '\n' ' ' < "$APP_DIR/manifests.err" | cut -c1-200)"); }

    # ---- 2. Secret (plaintext, encryption-at-rest deferred) ----
    echo -n "   🔑 Secret ($secret_name) ... "
    kubectl get secret "$secret_name" -n "$ns" -o yaml \
        > "$APP_DIR/secret.yaml" 2>"$APP_DIR/secret.err" \
-        && { echo "✅"; rm -f "$APP_DIR/secret.err"; } || echo "⚠️  FAILED (see secret.err)"
+        && { echo "✅"; rm -f "$APP_DIR/secret.err"; } \
+        || { echo "⚠️  FAILED (see secret.err)"; ERRORS+=("$app:secret:$(tr '\n' ' ' < "$APP_DIR/secret.err" | cut -c1-200)"); }

    # ---- 3. DB dump via kubectl exec (never raw datadir copy) ----
    echo -n "   💾 DB dump ($db_engine) ... "
@@ -238,9 +329,11 @@ for app in $ALL_APPS; do
                    rm -f "$APP_DIR/db-dump.err"
                else
                    echo "⚠️  FAILED (rc=$pg_dump_rc, see db-dump.err)"
+                    ERRORS+=("$app:db_dump:rc=$pg_dump_rc $(tr '\n' ' ' < "$APP_DIR/db-dump.err" 2>/dev/null | cut -c1-160)")
                fi
            else
                echo "⚠️  could not read DB credentials from secret"
+                ERRORS+=("$app:db_dump:could not read DB credentials from secret")
            fi
            ;;
        mariadb)
@@ -266,24 +359,33 @@ for app in $ALL_APPS; do
                    rm -f "$APP_DIR/db-dump.err"
                else
                    echo "⚠️  FAILED (rc=$mysqldump_rc, see db-dump.err)"
+                    ERRORS+=("$app:db_dump:rc=$mysqldump_rc $(tr '\n' ' ' < "$APP_DIR/db-dump.err" 2>/dev/null | cut -c1-160)")
                fi
            else
                echo "⚠️  could not determine DB name/credentials"
+                ERRORS+=("$app:db_dump:could not determine DB name/credentials")
            fi
            ;;
    esac

    # ---- 4. App-data PVC contents via kubectl exec (live tar, app-data only,
    #         never raw filesystem/PVC-path access — §5 sandbox-divergence rule) ----
-    echo -n "   📦 PVC data ($pvc_path) ... "
+    exclude_args=()
+    if [ -n "${TAR_EXCLUDES[$app]:-}" ]; then
+        for _excl_dir in ${TAR_EXCLUDES[$app]}; do
+            exclude_args+=(--exclude="./$_excl_dir")
+        done
+    fi
+    echo -n "   📦 PVC data ($pvc_path${exclude_args:+, excl: ${TAR_EXCLUDES[$app]}}) ... "
    kubectl exec -n "$ns" "deploy/${app_deploy}" -c "$app_ctr" -- \
-        tar czf - -C "$pvc_path" . 2>"$APP_DIR/pvc-data.err" \
+        tar czf - "${exclude_args[@]}" -C "$pvc_path" . 2>"$APP_DIR/pvc-data.err" \
        > "$APP_DIR/pvc-data.tar.gz"
    if [ -s "$APP_DIR/pvc-data.tar.gz" ]; then
        echo "✅ ($(du -h "$APP_DIR/pvc-data.tar.gz" | cut -f1))"
        rm -f "$APP_DIR/pvc-data.err"
    else
        echo "⚠️  FAILED (see pvc-data.err)"
+        ERRORS+=("$app:pvc_data:$(tr '\n' ' ' < "$APP_DIR/pvc-data.err" 2>/dev/null | cut -c1-160)")
    fi

    APP_COUNT=$((APP_COUNT + 1))
@@ -354,7 +456,7 @@ if [ "$ARCHIVE_COUNT" -gt "$MAX_BACKUPS" ]; then
    TO_DELETE=$(echo "$ARCHIVE_LIST" | tail -n +$((MAX_BACKUPS + 1)))
    while IFS= read -r old_file; do
        [ -z "$old_file" ] && continue
-        rm -f "$old_file" "${old_file}.sha256"
+        rm -f "$old_file" "${old_file}.sha256" "${old_file}.meta.json"
        echo "   🗑️  Deleted: $(basename "$old_file")"
    done <<< "$TO_DELETE"
 else
@@ -469,10 +571,11 @@ try:
    for obj in to_delete:
        old_key = obj['Key']
        client.delete_object(Bucket=bucket, Key=old_key)
-        try:
-            client.delete_object(Bucket=bucket, Key=old_key + '.sha256')
-        except Exception:
-            pass
+        for suffix in ('.sha256', '.meta.json'):
+            try:
+                client.delete_object(Bucket=bucket, Key=old_key + suffix)
+            except Exception:
+                pass
        print(f'   🗑️  Deleted from R2: {old_key.replace("backups/", "")}')

    sys.exit(0)
@@ -495,6 +598,13 @@ PYEOF
    fi
 fi

+# --------------------------------------------------
+# Status metadata sidecar — after all 3 tiers are known
+# --------------------------------------------------
+echo ""
+echo "📝 Writing status metadata sidecar..."
+write_meta_sidecar
+
 # --------------------------------------------------
 # Final summary + log
 # --------------------------------------------------
--- a/backup/restore-k8s-apps.sh
+++ b/backup/restore-k8s-apps.sh
@@ -103,6 +103,38 @@ declare -A DB_USER_KEY=( [n8n]=DB_POSTGRESDB_USER [odoo]=POSTGRES_USER [mautic]=
 declare -A DB_PASS_KEY=( [n8n]=DB_POSTGRESDB_PASSWORD [odoo]=POSTGRES_PASSWORD [mautic]=MYSQL_ROOT_PASSWORD [nextcloud]=POSTGRES_PASSWORD [frappe]=MARIADB_ROOT_PASSWORD )
 declare -A DOMAIN=(      [n8n]="n8nwf.nav.ovh" [odoo]="odooo.nav.ovh" [mautic]="mautics.nav.ovh" [nextcloud]="next.cloud.nav.ovh" [frappe]="erpnext.navitrends.ovh" )

+# Dirs backup-k8s-apps.sh excludes from pvc-data.tar.gz for these two apps
+# (reproducible framework/vendor code, not user data — see matching
+# TAR_EXCLUDES table + comment there). They must be re-seeded here from the
+# app's own live image, NOT reconstructed by the container entrypoint:
+#   - Mautic's image has no logic at all to repopulate node_modules/vendor
+#     if missing — confirmed empty by reading /entrypoint.sh.
+#   - Nextcloud's entrypoint only rsyncs from /usr/src/nextcloud when it
+#     detects image_version > installed_version (via version.php). Since
+#     version.php itself is NOT excluded (kept, "everything else"), a
+#     restored version.php already matches the running image's version, so
+#     that rsync path never fires — relying on it would silently leave
+#     apps/core/dist/3rdparty/lib missing after restore.
+# Both images bake these dirs into their own filesystem layer independent
+# of any volume mount, so cp'ing from a loader pod running the same image
+# (PVC mounted elsewhere, so it doesn't shadow the image's own copy) is
+# reliable regardless of either app's startup logic.
+#
+# RECONSTRUCT_SRC is the path *inside the image* to copy from — NOT
+# necessarily pvc_path. Verified per app by running the actual image with no
+# volume mounted and checking where it bakes the dirs in:
+#   - Mautic bakes node_modules/vendor directly at /var/www/html (its own
+#     PVC mount path) — same path, source == destination parent.
+#   - Nextcloud's image keeps its shipped source tree at /usr/src/nextcloud
+#     (confirmed via `du -sh` in a bare nextcloud:32 pod), completely
+#     separate from /var/www/html (the PVC mount, empty at that path in the
+#     image). Assuming pvc_path here (mirroring Mautic's layout) was wrong
+#     and failed loudly in testing — cp: cannot stat '/var/www/html/apps' —
+#     caught by the reseed_failed gate above rather than silently booting
+#     nextcloud broken.
+declare -A RECONSTRUCT_DIRS=( [mautic]="node_modules vendor" [nextcloud]="apps core dist 3rdparty lib" )
+declare -A RECONSTRUCT_SRC=(  [mautic]="/var/www/html"       [nextcloud]="/usr/src/nextcloud" )
+
 ALL_APPS="n8n odoo mautic nextcloud frappe"
 LOADER_IMAGE="alpine:3.20"

@@ -168,7 +200,18 @@ for app in $ALL_APPS; do
    #         restored here already exists in a same-cluster restore) ----
    if [ -f "$APP_DIR/secret.yaml" ]; then
        echo -n "   🔑 Applying Secret ... "
-        kubectl apply -f "$APP_DIR/secret.yaml" &>/dev/null && echo "✅" || { echo "⚠️  FAILED"; app_ok=false; }
+        # secret.yaml is a raw `kubectl get -o yaml` dump, which includes the
+        # live resourceVersion/uid/creationTimestamp at capture time. Passing
+        # a stale resourceVersion to `apply` trips optimistic-concurrency
+        # control ("the object has been modified") on any restore where the
+        # secret's real resourceVersion has since moved on — which is every
+        # restore, since normal cluster activity (this backup pipeline's own
+        # earlier applies included) bumps it constantly. Strip the volatile
+        # fields before applying; they're server-assigned and never meant to
+        # round-trip through a backup.
+        sed -e '/^  resourceVersion:/d' -e '/^  uid:/d' -e '/^  creationTimestamp:/d' \
+            "$APP_DIR/secret.yaml" > "$APP_DIR/secret.apply.yaml"
+        kubectl apply -f "$APP_DIR/secret.apply.yaml" &>/dev/null && echo "✅" || { echo "⚠️  FAILED"; app_ok=false; }
    fi
    # NOTE: manifests.yaml (Deployment/Service/ConfigMap/Ingress specs) is
    # captured by backup-k8s-apps.sh but deliberately NOT re-applied here.
@@ -216,9 +259,68 @@ for app in $ALL_APPS; do
        echo "   ⏭️  No pvc-data.tar.gz in backup — skipping PVC restore"
    fi

+    # ---- 3b. Re-seed reproducible dirs excluded from backup (see
+    #          RECONSTRUCT_DIRS comment above) from the app's own live image,
+    #          never from the backup archive itself ----
+    reseed_failed=false
+    if [ -n "${RECONSTRUCT_DIRS[$app]:-}" ] && kubectl get pvc "$pvc_name" -n "$ns" &>/dev/null; then
+        app_image=$(kubectl get deployment "$app_deploy" -n "$ns" -o jsonpath='{.spec.template.spec.containers[0].image}' 2>/dev/null)
+        if [ -n "$app_image" ]; then
+            seed_loader="${app}-reseed-loader"
+            echo -n "   🧩 Re-seeding [${RECONSTRUCT_DIRS[$app]}] from $app_image ... "
+            # imagePullPolicy IfNotPresent: $app_image is read straight off the
+            # currently-running Deployment, so it's already on this node —
+            # forcing a re-pull (the ":latest"/floating-tag default) just adds
+            # a slow, needless network fetch during every restore.
+            kubectl run "$seed_loader" -n "$ns" --image="$app_image" --restart=Never \
+                --overrides="{\"spec\":{\"containers\":[{\"name\":\"loader\",\"image\":\"${app_image}\",\"imagePullPolicy\":\"IfNotPresent\",\"command\":[\"sleep\",\"3600\"],\"volumeMounts\":[{\"name\":\"data\",\"mountPath\":\"/restore\"}]}],\"volumes\":[{\"name\":\"data\",\"persistentVolumeClaim\":{\"claimName\":\"${pvc_name}\"}}]}}" \
+                &>/dev/null
+            # 240s, not 60/90s like the alpine loader above: this pulls the
+            # full app image (600MB+ for mautic/nextcloud) if it isn't cached
+            # on this node yet — a 90s timeout was observed to fire mid-pull
+            # in testing, silently leaving the app pod without its
+            # reconstructed dirs after being scaled back up.
+            if kubectl wait --for=condition=Ready "pod/$seed_loader" -n "$ns" --timeout=240s &>/dev/null; then
+                seed_ok=true
+                : > "$APP_DIR/reseed.err"
+                seed_src="${RECONSTRUCT_SRC[$app]}"
+                for _seed_dir in ${RECONSTRUCT_DIRS[$app]}; do
+                    kubectl exec -n "$ns" "$seed_loader" -- sh -c "rm -rf '/restore/$_seed_dir' && cp -a '${seed_src}/${_seed_dir}' '/restore/$_seed_dir'" \
+                        >>"$APP_DIR/reseed.err" 2>&1 || seed_ok=false
+                done
+                if $seed_ok; then
+                    echo "✅"
+                    rm -f "$APP_DIR/reseed.err"
+                else
+                    echo "⚠️  FAILED (see $APP_DIR/reseed.err)"
+                    app_ok=false
+                    reseed_failed=true
+                fi
+            else
+                echo "⚠️  loader pod not ready"
+                app_ok=false
+                reseed_failed=true
+            fi
+            kubectl delete pod "$seed_loader" -n "$ns" --wait=false &>/dev/null
+        else
+            echo "   ⚠️  Could not determine $app image — skipping re-seed of [${RECONSTRUCT_DIRS[$app]}]"
+            app_ok=false
+            reseed_failed=true
+        fi
+    fi
+
    # ---- 4. Scale app back up ----
-    echo -n "   ▶️  Scaling $app_deploy back to $prior_replicas ... "
-    kubectl scale deployment "$app_deploy" -n "$ns" --replicas="$prior_replicas" &>/dev/null && echo "✅" || echo "⚠️  FAILED"
+    # Never bring the app back with replicas>0 if a required re-seed (3b)
+    # failed — for Mautic/Nextcloud that means booting with vendor/ or
+    # core/lib missing, which is a hard crash/broken-app state, not a
+    # degraded one. Left at 0 so the operator fixes it and scales up
+    # manually rather than the restore silently serving a broken app.
+    if $reseed_failed; then
+        echo "   ⛔ Skipping scale-up — required re-seed failed (app left at 0 replicas, see above)"
+    else
+        echo -n "   ▶️  Scaling $app_deploy back to $prior_replicas ... "
+        kubectl scale deployment "$app_deploy" -n "$ns" --replicas="$prior_replicas" &>/dev/null && echo "✅" || echo "⚠️  FAILED"
+    fi

    # ---- 5. DB restore (SQL-level, never raw datadir copy) ----
    if [ -f "$APP_DIR/db-dump.sql.gz" ]; then
--- a/platform/app.py
+++ b/platform/app.py
@@ -11,12 +11,14 @@ from datetime import datetime, timezone
 from config import (
    MAIN_SERVER_IP, RUNNING_ON_MAIN_SERVER,
    VM_HOST, VM_PORT, VM_KEY, VM_USER,
-    MAIN_SERVER_KEY, MAIN_SERVER_PORT, MAIN_SERVER_USER,
+    MAIN_SERVER_KEY, MAIN_SERVER_USER,
+    MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT,
 )
 from modules.auth import login_required
 from modules.backups import (
    get_containers, get_all_root_containers,
    get_local_backups, get_vm_backups,
+    get_local_backups_with_status, get_vm_backups_with_status,
    get_all_stats, get_system_info,
    get_rootless_user_containers_remote,
    container_action, get_container_status,
@@ -169,8 +171,8 @@ def cluster_page():
 def backups_page():
    return render_template(
        'pages/backups.html',
-        backups=get_local_backups(),
-        vm_backups=get_vm_backups(),
+        backups=get_local_backups_with_status(),
+        vm_backups=get_vm_backups_with_status(),
        main_server=MAIN_SERVER_IP,
        active_page='backups',
        page_title='Backup Management',
@@ -191,6 +193,14 @@ def restore_page():
        vm_backups=get_vm_backups(),
        restore_prefill=prefill,
        main_server=MAIN_SERVER_IP,
+        running_on_main_server=RUNNING_ON_MAIN_SERVER,
+        # Only meaningful when running as the standby: "restore on this
+        # server" is nonsensical there (no local cluster/kubectl at all),
+        # so pre-fill the remote-target fields with the tunnel back to the
+        # real main server instead of leaving the user to find these values.
+        tunnel_remote_ip=MAIN_SERVER_SSH_HOST,
+        tunnel_remote_port=MAIN_SERVER_SSH_PORT,
+        tunnel_ssh_key=MAIN_SERVER_KEY,
        active_page='restore',
        page_title='Restore',
        page_subtitle='backup → target'
@@ -376,7 +386,7 @@ def api_container_status(name):
@app.route('/api/backups')
@login_required
 def api_backups():
-    return jsonify({'local': get_local_backups(), 'vm': get_vm_backups()})
+    return jsonify({'local': get_local_backups_with_status(), 'vm': get_vm_backups_with_status()})


@app.route('/api/backups/log')
@@ -694,9 +704,9 @@ def api_r2_upload():
            local_path = f"/tmp/{backup_file}"
            if not os.path.exists(local_path):
                pull_cmd = (
-                    f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_PORT} "
+                    f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_SSH_PORT} "
                    f"-o StrictHostKeyChecking=no -o ConnectTimeout=15 "
-                    f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}:/root/backups/{backup_file} "
+                    f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}:/root/backups/{backup_file} "
                    f"{local_path}"
                )
                res = subprocess.run(pull_cmd, shell=True, capture_output=True, text=True)
@@ -798,9 +808,9 @@ def restore_start():
            backup_path = f"/tmp/{backup_file}"
            if not os.path.exists(backup_path):
                pull_cmd = (
-                    f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_PORT} "
+                    f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_SSH_PORT} "
                    f"-o StrictHostKeyChecking=no -o ConnectTimeout=15 "
-                    f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}:/root/backups/{backup_file} "
+                    f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}:/root/backups/{backup_file} "
                    f"{backup_path}"
                )
                res = subprocess.run(pull_cmd, shell=True, capture_output=True, text=True)
@@ -834,6 +844,12 @@ def restore_start():
        return jsonify({'error': f'restore-k8s-apps.sh not found at {restore_script_local}'}), 500

    if target == 'local':
+        if not RUNNING_ON_MAIN_SERVER:
+            return jsonify({
+                'error': 'This is the standby — there is no local cluster here to restore '
+                         'into. Use "External Machine" to restore onto the real main server '
+                         'over the tunnel instead.'
+            }), 400
        hostname    = os.uname().nodename
        os.makedirs('/root/tmp', exist_ok=True)
        session_dir = f"/root/tmp/restore-session-{uuid.uuid4().hex[:8]}"
--- a/platform/modules/backups.py
+++ b/platform/modules/backups.py
@@ -11,7 +11,8 @@ import socket
 import psutil
 from config import (
    RUNNING_ON_MAIN_SERVER,
-    MAIN_SERVER_IP, MAIN_SERVER_USER, MAIN_SERVER_KEY, MAIN_SERVER_PORT,
+    MAIN_SERVER_USER, MAIN_SERVER_KEY,
+    MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT,
    VM_HOST, VM_PORT, VM_KEY, VM_USER,
 )

@@ -43,10 +44,10 @@ def _ssh_main(remote_cmd, timeout=30):
    else:
        escaped = remote_cmd.replace("'", "'\\''")
        ssh = (
-            f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_PORT} "
+            f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_SSH_PORT} "
            f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 "
            f"-o BatchMode=yes "
-            f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}"
+            f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}"
        )
        return _run(f"{ssh} '{escaped}'", timeout=timeout)

@@ -127,6 +128,127 @@ def get_vm_backups():
    return vm_backups


+# ────────────────────────────────────────────────────────────────
+# BACKUP STATUS DOTS (green/yellow/red/unknown)
+#
+# Reads the .meta.json sidecar backup-k8s-apps.sh writes alongside each
+# archive (never decompresses the archive itself). Legacy myapps-backup-*
+# archives and any k8s backup made before this feature shipped have no
+# sidecar and show as 'unknown' — there's no reliable way to reconstruct
+# per-step error info after the fact, and no backfill is attempted.
+# ────────────────────────────────────────────────────────────────
+
+ROLLING_WINDOW = 5
+SIZE_DEVIATION_THRESHOLD = 0.40
+
+
+def _load_local_sidecars(local_dir='/root/backups'):
+    """All .meta.json sidecars in a directory, newest first. Used both to
+    look up one backup's own metadata and as the history pool for its
+    rolling-average comparison."""
+    sidecars = {}
+    for path in glob.glob(f'{local_dir}/*.tar.gz.meta.json'):
+        try:
+            with open(path) as f:
+                m = json.load(f)
+        except Exception:
+            continue
+        backup_file = os.path.basename(path)[:-len('.meta.json')]
+        m['_backup_file'] = backup_file
+        sidecars[backup_file] = m
+    return sidecars
+
+
+def _parse_vm_sidecars(cat_output):
+    """Parse the batched `echo marker + cat` SSH output from _fetch_vm_sidecars
+    into the same {backup_file: meta} shape as _load_local_sidecars."""
+    sidecars = {}
+    if not cat_output:
+        return sidecars
+    for block in cat_output.split('===META:')[1:]:
+        try:
+            fname_line, _, rest = block.partition('===\n')
+            backup_file = fname_line.strip()[:-len('.meta.json')]
+            m = json.loads(rest)
+            m['_backup_file'] = backup_file
+            sidecars[backup_file] = m
+        except Exception:
+            continue
+    return sidecars
+
+
+def _fetch_vm_sidecars():
+    local_dir = '/backups/cloudproject'
+    if os.path.isdir(local_dir):
+        return _load_local_sidecars(local_dir)
+    cmd = (
+        f"ssh -i {VM_KEY} -p {VM_PORT} -o StrictHostKeyChecking=no "
+        f"-o ConnectTimeout=10 -o BatchMode=yes {VM_USER}@{VM_HOST} "
+        f"'for f in /backups/cloudproject/*.tar.gz.meta.json; do "
+        f"[ -f \"$f\" ] && echo \"===META:$(basename \"$f\")===\" && cat \"$f\"; "
+        f"done 2>/dev/null'"
+    )
+    stdout, _ = _run(cmd, timeout=25)
+    return _parse_vm_sidecars(stdout)
+
+
+def _compute_backup_status(meta, all_sidecars):
+    """all_sidecars: dict of {backup_file: meta} — the pool this entry's
+    rolling average is computed from (same directory/tier it came from)."""
+    tiers = meta.get('tiers', {})
+    if meta.get('errors') or any(str(v).startswith('failed') for v in tiers.values()):
+        return 'red'
+
+    apps_key = meta.get('apps_key', '')
+    size_bytes = meta.get('size_bytes', 0)
+    created_at = meta.get('created_at', '')
+
+    # Same app-combination, strictly older than this backup, most recent
+    # first — a fresh, self-calibrating baseline per app-set rather than
+    # one hardcoded global threshold (a 1-app and a 5-app backup have very
+    # different normal sizes).
+    history = sorted(
+        (m for m in all_sidecars.values()
+         if m.get('apps_key') == apps_key and m.get('created_at', '') < created_at),
+        key=lambda m: m.get('created_at', ''),
+        reverse=True,
+    )[:ROLLING_WINDOW]
+
+    if not history:
+        return 'green'  # first backup of this app-combination — nothing to compare against
+
+    avg = sum(m.get('size_bytes', 0) for m in history) / len(history)
+    if avg <= 0:
+        return 'green'
+
+    deviation = abs(size_bytes - avg) / avg
+    return 'yellow' if deviation > SIZE_DEVIATION_THRESHOLD else 'green'
+
+
+def _with_status(names, sidecars):
+    result = []
+    for name in names:
+        meta = sidecars.get(name)
+        if meta is None:
+            result.append({'name': name, 'status': 'unknown', 'size_human': None, 'apps': None})
+        else:
+            result.append({
+                'name': name,
+                'status': _compute_backup_status(meta, sidecars),
+                'size_human': _human_bytes(meta.get('size_bytes', 0)),
+                'apps': meta.get('apps'),
+            })
+    return result
+
+
+def get_local_backups_with_status():
+    return _with_status(get_local_backups(), _load_local_sidecars())
+
+
+def get_vm_backups_with_status():
+    return _with_status(get_vm_backups(), _fetch_vm_sidecars())
+
+
 # ────────────────────────────────────────────────────────────────
 # BACKUP HEALTH AUDIT
 # ────────────────────────────────────────────────────────────────
@@ -154,9 +276,9 @@ def _resolve_archive_path(backup_file, source, add):
        tmp_path = f"/tmp/audit_{backup_file}"
        if not os.path.exists(tmp_path):
            pull_cmd = (
-                f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_PORT} "
+                f"scp -i {MAIN_SERVER_KEY} -P {MAIN_SERVER_SSH_PORT} "
                f"-o StrictHostKeyChecking=no -o ConnectTimeout=15 "
-                f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}:/root/backups/{backup_file} "
+                f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}:/root/backups/{backup_file} "
                f"{tmp_path}"
            )
            _out, err = _run(pull_cmd, timeout=120)
@@ -524,11 +646,16 @@ def delete_backup(backup_file, source='local'):
            if not os.path.exists(archive_path):
                return False, f'File not found: {archive_path}'
            os.remove(archive_path)
-            sha = archive_path + '.sha256'
-            if os.path.exists(sha): os.remove(sha)
+            for suffix in ('.sha256', '.meta.json'):
+                sidecar = archive_path + suffix
+                if os.path.exists(sidecar): os.remove(sidecar)
            return True, f'Deleted {backup_file} from main server'
        else:
-            cmd = f"rm -f /root/backups/{backup_file} /root/backups/{backup_file}.sha256"
+            cmd = (
+                f"rm -f /root/backups/{backup_file} "
+                f"/root/backups/{backup_file}.sha256 "
+                f"/root/backups/{backup_file}.meta.json"
+            )
            out, err = _ssh_main(cmd)
            if err and 'No such file' not in err:
                return False, f'Remote delete error: {err}'
@@ -540,8 +667,9 @@ def delete_backup(backup_file, source='local'):
            if not os.path.exists(archive_path):
                return False, f'File not found: {archive_path}'
            os.remove(archive_path)
-            sha = archive_path + '.sha256'
-            if os.path.exists(sha): os.remove(sha)
+            for suffix in ('.sha256', '.meta.json'):
+                sidecar = archive_path + suffix
+                if os.path.exists(sidecar): os.remove(sidecar)
            return True, f'Deleted {backup_file} from VM'
        else:
            cmd = (
@@ -550,7 +678,8 @@ def delete_backup(backup_file, source='local'):
                f"-o BatchMode=yes "
                f"{VM_USER}@{VM_HOST} "
                f"'rm -f /backups/cloudproject/{backup_file} "
-                f"/backups/cloudproject/{backup_file}.sha256'"
+                f"/backups/cloudproject/{backup_file}.sha256 "
+                f"/backups/cloudproject/{backup_file}.meta.json'"
            )
            out, err = _run(cmd, timeout=30)
            if err and 'No such file' not in err:
@@ -766,8 +895,65 @@ def get_all_stats():
 # SYSTEM INFO (FIXED WITH PSUTIL)
 # ────────────────────────────────────────────────────────────────

+def _get_main_server_system_info_remote():
+    """get_system_info()'s equivalent for the standby: the main server's bare
+    host has no psutil (system python3, not this app's container image), so
+    this uses plain coreutils/procps instead — reusing _ssh_main means this
+    genuinely queries the real main server over the tunnel, not the VM
+    itself (the previous unconditional psutil.* calls silently reported the
+    VM's own stats mislabeled as "system info" whenever running as standby)."""
+    fallback = {
+        'cpu_pct': '0', 'memory': 'N/A', 'mem_pct': '0', 'disk': 'N/A',
+        'disk_pct': '0', 'load': 'N/A', 'uptime': 'N/A', 'docker_v': 'N/A',
+        'hostname': 'unreachable',
+    }
+    remote_cmd = (
+        "hostname; "
+        "vmstat 1 2 | tail -1 | awk '{print 100-$15}'; "
+        "free -m | awk 'NR==2{printf \"%d %d %d\\n\", $3, $2, $3*100/$2}'; "
+        "df -BM --output=used,size,pcent / | tail -1 | tr -d 'M%'; "
+        "awk '{printf \"%.2f %.2f %.2f\\n\", $1, $2, $3}' /proc/loadavg; "
+        "awk '{print $1}' /proc/uptime; "
+        "docker --version 2>/dev/null | awk '{print $3}' | tr -d ','"
+    )
+    out, err = _ssh_main(remote_cmd, timeout=15)
+    lines = out.split('\n') if out else []
+    if len(lines) < 7:
+        return fallback
+    try:
+        hostname = lines[0].strip()
+        cpu_pct = lines[1].strip()
+        mem_used_mb, mem_total_mb, mem_pct = lines[2].split()
+        disk_used_mb, disk_total_mb, disk_pct = lines[3].split()
+        load = lines[4].strip()
+        uptime_s = float(lines[5].strip())
+        docker_v = lines[6].strip() or 'N/A'
+
+        days = int(uptime_s // 86400)
+        hours = int((uptime_s % 86400) // 3600)
+        minutes = int((uptime_s % 3600) // 60)
+        uptime = f"{days}d {hours}h {minutes}m" if days > 0 else f"{hours}h {minutes}m"
+
+        return {
+            'cpu_pct':  cpu_pct,
+            'memory':   f"{int(mem_used_mb) / 1024:.1f}G/{int(mem_total_mb) / 1024:.1f}G",
+            'mem_pct':  mem_pct,
+            'disk':     f"{int(disk_used_mb) / 1024:.0f}G/{int(disk_total_mb) / 1024:.0f}G",
+            'disk_pct': disk_pct,
+            'load':     load,
+            'uptime':   uptime,
+            'docker_v': docker_v,
+            'hostname': hostname,
+        }
+    except Exception as e:
+        print(f"[backups] remote system info parse error: {e} (raw: {out!r}, err: {err!r})")
+        return fallback
+
+
 def get_system_info():
    """Get system information using psutil for reliable metrics inside container"""
+    if not RUNNING_ON_MAIN_SERVER:
+        return _get_main_server_system_info_remote()

    info = {
        'cpu_pct': '0',
--- a/platform/modules/commands.py
+++ b/platform/modules/commands.py
@@ -1,5 +1,5 @@
 import subprocess
-from config import RUNNING_ON_MAIN_SERVER, MAIN_SERVER_IP
+from config import RUNNING_ON_MAIN_SERVER, MAIN_SERVER_KEY, MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT

 def run_command(command):
    """Run command on main server (directly or via SSH)"""
@@ -14,10 +14,12 @@ def run_command(command):
    else:
        ssh_cmd = [
            "ssh",
+            "-i", MAIN_SERVER_KEY,
+            "-p", MAIN_SERVER_SSH_PORT,
            "-o", "StrictHostKeyChecking=no",
            "-o", "ConnectTimeout=10",
            "-o", "BatchMode=yes",
-            f"root@{MAIN_SERVER_IP}",
+            f"root@{MAIN_SERVER_SSH_HOST}",
            command
        ]
        try:
--- a/platform/modules/kubernetes.py
+++ b/platform/modules/kubernetes.py
@@ -4,6 +4,18 @@
 # deployments.apps — no secrets, no write, no exec/log), bound per-namespace
 # across the 8 app namespaces below. There is no cluster-wide RoleBinding,
 # so every list call below is scoped to one namespace at a time.
+#
+# On the standby (RUNNING_ON_MAIN_SERVER False), there is no in-cluster
+# ServiceAccount to load at all — load_incluster_config() always fails
+# there, by design, even if the standby itself later runs as a pod on a
+# *local* k3s on that VM (that cluster isn't this one; in-cluster config
+# would only ever see the wrong, essentially-empty cluster). Every function
+# below instead SSHes to the main server (over main-to-vm-tunnel.service)
+# and runs `kubectl` there directly, where real cluster-admin access
+# already exists — same real-vs-display separation as MAIN_SERVER_SSH_HOST
+# in config.py.
+import json
+import subprocess
 from datetime import datetime, timezone

 try:
@@ -13,6 +25,11 @@ try:
 except ImportError:
    _K8S_IMPORT_OK = False

+from config import (
+    RUNNING_ON_MAIN_SERVER, MAIN_SERVER_USER, MAIN_SERVER_KEY,
+    MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT,
+)
+
 NAMESPACES = [
    'n8n', 'odoo', 'mautic', 'erpnext', 'nextcloud',
    'jenkins-agents', 'jenkins', 'management-platform',
@@ -25,6 +42,78 @@ _networking_v1 = None
 K8S_AVAILABLE = False


+def _ssh_kubectl(args):
+    """Run `kubectl <args> -o json` on the main server over the reverse
+    tunnel. Returns the parsed JSON (dict) or None on any failure —
+    callers must treat None/empty the same as the in-cluster client
+    returning nothing, never as a hard error."""
+    remote_cmd = "kubectl " + " ".join(args) + " -o json"
+    ssh_cmd = (
+        f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_SSH_PORT} "
+        f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 -o BatchMode=yes "
+        f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST} '{remote_cmd}'"
+    )
+    try:
+        r = subprocess.run(ssh_cmd, shell=True, capture_output=True, text=True, timeout=20)
+        if r.returncode != 0:
+            return None
+        return json.loads(r.stdout)
+    except Exception:
+        return None
+
+
+def _ssh_kubectl_raw(path):
+    """Same as _ssh_kubectl but for `kubectl get --raw <path>` (used for
+    the metrics-server API, which isn't a normal `get <resource>` call)."""
+    ssh_cmd = (
+        f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_SSH_PORT} "
+        f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 -o BatchMode=yes "
+        f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST} 'kubectl get --raw {path}'"
+    )
+    try:
+        r = subprocess.run(ssh_cmd, shell=True, capture_output=True, text=True, timeout=20)
+        if r.returncode != 0:
+            return None
+        return json.loads(r.stdout)
+    except Exception:
+        return None
+
+
+def _parse_k8s_ts(s):
+    if not s:
+        return None
+    try:
+        return datetime.strptime(s, '%Y-%m-%dT%H:%M:%SZ').replace(tzinfo=timezone.utc)
+    except Exception:
+        return None
+
+
+def _pod_health_raw(pod):
+    statuses = (pod.get('status') or {}).get('containerStatuses') or []
+    for cs in statuses:
+        waiting = (cs.get('state') or {}).get('waiting')
+        if waiting and waiting.get('reason') in _CRASH_REASONS:
+            return 'failed'
+    phase = (pod.get('status') or {}).get('phase') or 'Unknown'
+    if phase == 'Failed':
+        return 'failed'
+    if phase == 'Running':
+        all_ready = all(cs.get('ready') for cs in statuses) if statuses else True
+        return 'healthy' if all_ready else 'degraded'
+    return 'degraded'
+
+
+def _ready_count_raw(pod):
+    statuses = (pod.get('status') or {}).get('containerStatuses') or []
+    ready = sum(1 for cs in statuses if cs.get('ready'))
+    return f'{ready}/{len(statuses)}'
+
+
+def _restart_count_raw(pod):
+    statuses = (pod.get('status') or {}).get('containerStatuses') or []
+    return sum(cs.get('restartCount', 0) for cs in statuses)
+
+
 def _init_client():
    global _core_v1, _apps_v1, _custom_objects, _networking_v1, K8S_AVAILABLE
    if _core_v1 is not None or not _K8S_IMPORT_OK:
@@ -135,6 +224,24 @@ def _parse_mem_mib(v):
 # ────────────────────────────────────────────────────────────────

 def list_pods():
+    if not RUNNING_ON_MAIN_SERVER:
+        pods = []
+        for ns in NAMESPACES:
+            data = _ssh_kubectl(['get', 'pods', '-n', ns])
+            if not data:
+                continue
+            for pod in data.get('items', []):
+                pods.append({
+                    'name':      pod['metadata']['name'],
+                    'namespace': ns,
+                    'phase':     (pod.get('status') or {}).get('phase') or 'Unknown',
+                    'health':    _pod_health_raw(pod),
+                    'ready':     _ready_count_raw(pod),
+                    'restarts':  _restart_count_raw(pod),
+                    'age':       _age_str(_parse_k8s_ts(pod['metadata'].get('creationTimestamp'))),
+                })
+        return pods
+
    _init_client()
    pods = []
    if not K8S_AVAILABLE:
@@ -159,6 +266,25 @@ def list_pods():


 def list_deployments():
+    if not RUNNING_ON_MAIN_SERVER:
+        deployments = []
+        for ns in NAMESPACES:
+            data = _ssh_kubectl(['get', 'deployments', '-n', ns])
+            if not data:
+                continue
+            for dep in data.get('items', []):
+                desired = (dep.get('spec') or {}).get('replicas') or 0
+                ready = (dep.get('status') or {}).get('readyReplicas') or 0
+                deployments.append({
+                    'name':      dep['metadata']['name'],
+                    'namespace': ns,
+                    'ready':     ready,
+                    'desired':   desired,
+                    'health':    _deploy_health(desired, ready),
+                    'age':       _age_str(_parse_k8s_ts(dep['metadata'].get('creationTimestamp'))),
+                })
+        return deployments
+
    _init_client()
    deployments = []
    if not K8S_AVAILABLE:
@@ -187,6 +313,19 @@ def get_ingress_info(name, namespace):
    """Real domain/TLS for one Ingress, or None if unavailable/not found.
    Used by modules/sites.py instead of the old hardcoded Docker-era
    domain fields — reflects actual live k8s Ingress state."""
+    if not RUNNING_ON_MAIN_SERVER:
+        data = _ssh_kubectl(['get', 'ingress', name, '-n', namespace])
+        if not data:
+            return None
+        host = None
+        rules = (data.get('spec') or {}).get('rules') or []
+        for rule in rules:
+            if rule.get('host'):
+                host = rule['host']
+                break
+        tls = bool((data.get('spec') or {}).get('tls'))
+        return {'host': host, 'tls': tls}
+
    _init_client()
    if not K8S_AVAILABLE:
        return None
@@ -207,6 +346,25 @@ def get_ingress_info(name, namespace):
 def list_deployments_for_namespace(ns):
    """Targeted single-namespace deployment list (vs. list_deployments()'s
    fixed NAMESPACES sweep) — used by modules/sites.py per app."""
+    if not RUNNING_ON_MAIN_SERVER:
+        data = _ssh_kubectl(['get', 'deployments', '-n', ns])
+        if not data:
+            return []
+        out = []
+        for dep in data.get('items', []):
+            desired = (dep.get('spec') or {}).get('replicas') or 0
+            ready = (dep.get('status') or {}).get('readyReplicas') or 0
+            containers = (((dep.get('spec') or {}).get('template') or {}).get('spec') or {}).get('containers') or []
+            image = containers[0]['image'] if containers else '—'
+            out.append({
+                'name':    dep['metadata']['name'],
+                'desired': desired,
+                'ready':   ready,
+                'health':  _deploy_health(desired, ready),
+                'image':   image,
+            })
+        return out
+
    _init_client()
    if not K8S_AVAILABLE:
        return []
@@ -235,6 +393,20 @@ def get_pod_metrics():
    """Best-effort CPU/mem per pod via metrics-server. Returns {} if it's not
    installed or unreachable — callers must treat an empty dict as 'no data',
    never as an error."""
+    if not RUNNING_ON_MAIN_SERVER:
+        metrics = {}
+        for ns in NAMESPACES:
+            resp = _ssh_kubectl_raw(f'/apis/metrics.k8s.io/v1beta1/namespaces/{ns}/pods')
+            if not resp:
+                continue
+            for item in resp.get('items', []):
+                name = item.get('metadata', {}).get('name')
+                containers = item.get('containers', [])
+                cpu = sum(_parse_cpu(c.get('usage', {}).get('cpu')) for c in containers)
+                mem = sum(_parse_mem_mib(c.get('usage', {}).get('memory')) for c in containers)
+                metrics[f'{ns}/{name}'] = {'cpu_millicores': round(cpu), 'mem_mib': round(mem)}
+        return metrics
+
    _init_client()
    metrics = {}
    if not K8S_AVAILABLE:
@@ -283,8 +455,13 @@ def get_cluster_overview():
        'deployments_healthy': sum(1 for d in deployments if d['health'] == 'healthy'),
    }

+    # K8S_AVAILABLE only reflects the in-cluster client (main server); on
+    # the standby, "available" instead means the SSH+kubectl fetch above
+    # actually returned something.
+    available = K8S_AVAILABLE if RUNNING_ON_MAIN_SERVER else bool(pods or deployments)
+
    return {
-        'available':        K8S_AVAILABLE,
+        'available':        available,
        'namespaces':        by_namespace,
        'summary':           summary,
        'metrics_available': bool(metrics),
--- a/platform/modules/users.py
+++ b/platform/modules/users.py
@@ -7,7 +7,8 @@ import json

 from config import (
    RUNNING_ON_MAIN_SERVER,
-    MAIN_SERVER_IP, MAIN_SERVER_USER, MAIN_SERVER_KEY, MAIN_SERVER_PORT,
+    MAIN_SERVER_USER, MAIN_SERVER_KEY,
+    MAIN_SERVER_SSH_HOST, MAIN_SERVER_SSH_PORT,
 )


@@ -33,10 +34,10 @@ def _ssh_main(remote_cmd, timeout=30):
        # Escape single quotes in remote_cmd for safe shell wrapping
        escaped = remote_cmd.replace("'", "'\\''")
        ssh = (
-            f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_PORT} "
+            f"ssh -i {MAIN_SERVER_KEY} -p {MAIN_SERVER_SSH_PORT} "
            f"-o StrictHostKeyChecking=no -o ConnectTimeout=10 "
            f"-o BatchMode=yes "
-            f"{MAIN_SERVER_USER}@{MAIN_SERVER_IP}"
+            f"{MAIN_SERVER_USER}@{MAIN_SERVER_SSH_HOST}"
        )
        return _run(f"{ssh} '{escaped}'", timeout=timeout)

--- a/platform/static/css/style.css
+++ b/platform/static/css/style.css
@@ -881,7 +881,7 @@ a.sidebar-brand-link {
  box-shadow: var(--glow-accent);
 }

-.backup-item-main { min-width: 0; padding-bottom: 12px; }
+.backup-item-main { min-width: 0; padding-bottom: 12px; display: flex; align-items: center; gap: 8px; }
 .backup-item:not(:has(.backup-actions)) .backup-item-main { padding-bottom: 0; }

 .backup-name {
@@ -894,6 +894,18 @@ a.sidebar-brand-link {
 }
 .backup-name i { margin-right: 6px; opacity: 0.85; }

+/* Backup status dot — green/yellow/red from the .meta.json sidecar
+   backup-k8s-apps.sh writes; gray for backups with no sidecar (legacy
+   archives, or anything made before this feature shipped). */
+.status-dot {
+  width: 8px; height: 8px; border-radius: 50%; flex-shrink: 0;
+  border: 1px solid rgba(0,0,0,0.15);
+}
+.status-dot-green  { background: var(--green); box-shadow: 0 0 6px rgba(45,212,122,0.5); }
+.status-dot-yellow { background: var(--yellow); box-shadow: 0 0 6px rgba(245,185,66,0.5); }
+.status-dot-red    { background: var(--red); box-shadow: 0 0 6px rgba(240,82,82,0.5); }
+.status-dot-unknown { background: var(--text3); box-shadow: none; }
+
 .backup-meta {
  display: block;
  margin-top: 6px;
--- a/platform/static/js/platform.js
+++ b/platform/static/js/platform.js
@@ -170,7 +170,10 @@ async function refreshSystemMetrics() {
    if (el('g-disk')) el('g-disk').style.width = Math.min(parseFloat(d.disk_pct) || 0, 100) + '%';
    setText('uptime-chip', d.uptime);
    if (el('settings-uptime')) el('settings-uptime').value = d.uptime;
-    if (d.hostname && el('this-server-desc')) el('this-server-desc').textContent = d.hostname;
+    const localTargetRadio = document.querySelector('input[name="restore_target"][value="local"]');
+    if (d.hostname && el('this-server-desc') && !(localTargetRadio && localTargetRadio.disabled)) {
+      el('this-server-desc').textContent = d.hostname;
+    }
  } catch (_) {}
 }

@@ -377,8 +380,8 @@ async function refreshBackupsList() {
    renderBackupList(d.vm, 'vm-backup-list', 'vm');
    if (el('stat-local-bk')) el('stat-local-bk').textContent = d.local.length;
    if (el('stat-vm-bk')) el('stat-vm-bk').textContent = d.vm.length;
-    if (el('local-options')) el('local-options').innerHTML = d.local.length ? d.local.map((b) => `<option value="${b}" data-source="local">${b}</option>`).join('') : '<option disabled>No local backups</option>';
-    if (el('vm-options')) el('vm-options').innerHTML = d.vm.length ? d.vm.map((b) => `<option value="${b}" data-source="vm">${b}</option>`).join('') : '<option disabled>No VM backups</option>';
+    if (el('local-options')) el('local-options').innerHTML = d.local.length ? d.local.map((b) => `<option value="${b.name}" data-source="local">${b.name}</option>`).join('') : '<option disabled>No local backups</option>';
+    if (el('vm-options')) el('vm-options').innerHTML = d.vm.length ? d.vm.map((b) => `<option value="${b.name}" data-source="vm">${b.name}</option>`).join('') : '<option disabled>No VM backups</option>';
  } catch (_) {}
 }

@@ -394,7 +397,13 @@ function renderBackupList(items, id, source) {
    return;
  }
  node.innerHTML = items.map((b) => {
-    const safe = jsStr(b);
+    const name = b.name;
+    const safe = jsStr(name);
+    const status = b.status || 'unknown';
+    const titleParts = [status];
+    if (b.apps && b.apps.length) titleParts.push(`apps: ${b.apps.join(', ')}`);
+    if (b.size_human) titleParts.push(b.size_human);
+    const title = escapeHtml(titleParts.join(' · '));
    const uploadBtn = source === 'local' && typeof quickUploadR2 === 'function'
      ? `<button class="btn btn-ghost btn-sm" onclick="quickUploadR2('${safe}',this)" title="Upload to R2"><i class="fas fa-cloud-upload-alt"></i></button>`
      : '';
@@ -402,9 +411,10 @@ function renderBackupList(items, id, source) {
      ? `<button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('${source}','${safe}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>`
      : '';
    return `
-    <div class="backup-item" id="bk-item-${source}-${b.replace(/[^a-z0-9]/gi, '_')}">
+    <div class="backup-item" id="bk-item-${source}-${name.replace(/[^a-z0-9]/gi, '_')}">
      <div class="backup-item-main">
-        <span class="backup-name">${escapeHtml(b)}</span>
+        <span class="status-dot status-dot-${status}" title="${title}"></span>
+        <span class="backup-name">${escapeHtml(name)}</span>
      </div>
      <div class="backup-actions">
        ${detailsBtn}
--- a/platform/templates/pages/backups.html
+++ b/platform/templates/pages/backups.html
@@ -286,14 +286,15 @@
        {% for b in backups %}
        <div class="backup-item">
          <div class="backup-item-main">
-            <span class="backup-name">{{ b }}</span>
+            <span class="status-dot status-dot-{{ b.status }}" title="{{ b.status }}{% if b.apps %} · apps: {{ b.apps|join(', ') }}{% endif %}{% if b.size_human %} · {{ b.size_human }}{% endif %}"></span>
+            <span class="backup-name">{{ b.name }}</span>
          </div>
          <div class="backup-actions">
-            <button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('local','{{ b }}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>
-            <button class="btn btn-ghost btn-sm btn-audit" onclick="auditBackup('local','{{ b }}',this)"><i class="fas fa-shield-check"></i> Audit</button>
-            <button class="btn btn-ghost btn-sm" onclick="quickRestore('local','{{ b }}')">↩ Restore</button>
-            <button class="btn btn-ghost btn-sm" onclick="quickUploadR2('{{ b }}',this)" title="Upload to R2"><i class="fas fa-cloud-upload-alt"></i></button>
-            <button class="btn btn-ghost btn-sm btn-delete-backup" onclick="deleteBackup('local','{{ b }}',this)"><i class="fas fa-trash"></i></button>
+            <button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('local','{{ b.name }}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>
+            <button class="btn btn-ghost btn-sm btn-audit" onclick="auditBackup('local','{{ b.name }}',this)"><i class="fas fa-shield-check"></i> Audit</button>
+            <button class="btn btn-ghost btn-sm" onclick="quickRestore('local','{{ b.name }}')">↩ Restore</button>
+            <button class="btn btn-ghost btn-sm" onclick="quickUploadR2('{{ b.name }}',this)" title="Upload to R2"><i class="fas fa-cloud-upload-alt"></i></button>
+            <button class="btn btn-ghost btn-sm btn-delete-backup" onclick="deleteBackup('local','{{ b.name }}',this)"><i class="fas fa-trash"></i></button>
          </div>
        </div>
        {% else %}<div class="empty-state"><i class="fas fa-inbox"></i>No backups</div>{% endfor %}
@@ -310,13 +311,14 @@
        {% for b in vm_backups %}
        <div class="backup-item">
          <div class="backup-item-main">
-            <span class="backup-name">{{ b }}</span>
+            <span class="status-dot status-dot-{{ b.status }}" title="{{ b.status }}{% if b.apps %} · apps: {{ b.apps|join(', ') }}{% endif %}{% if b.size_human %} · {{ b.size_human }}{% endif %}"></span>
+            <span class="backup-name">{{ b.name }}</span>
          </div>
          <div class="backup-actions">
-            <button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('vm','{{ b }}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>
-            <button class="btn btn-ghost btn-sm btn-audit" onclick="auditBackup('vm','{{ b }}',this)"><i class="fas fa-shield-check"></i> Audit</button>
-            <button class="btn btn-ghost btn-sm" onclick="quickRestore('vm','{{ b }}')">↩ Restore</button>
-            <button class="btn btn-ghost btn-sm btn-delete-backup" onclick="deleteBackup('vm','{{ b }}',this)"><i class="fas fa-trash"></i></button>
+            <button class="btn btn-ghost btn-sm" style="color:var(--text3);" onclick="showBackupDetails('vm','{{ b.name }}',this)" title="Quick details"><i class="fas fa-circle-info"></i></button>
+            <button class="btn btn-ghost btn-sm btn-audit" onclick="auditBackup('vm','{{ b.name }}',this)"><i class="fas fa-shield-check"></i> Audit</button>
+            <button class="btn btn-ghost btn-sm" onclick="quickRestore('vm','{{ b.name }}')">↩ Restore</button>
+            <button class="btn btn-ghost btn-sm btn-delete-backup" onclick="deleteBackup('vm','{{ b.name }}',this)"><i class="fas fa-trash"></i></button>
          </div>
        </div>
        {% else %}<div class="empty-state"><i class="fas fa-inbox"></i>No VM backups</div>{% endfor %}
--- a/platform/templates/pages/restore.html
+++ b/platform/templates/pages/restore.html
@@ -63,26 +63,37 @@

    <div class="form-section">
      <div class="form-section-title">STEP 2 — SELECT RESTORE TARGET</div>
+      {% if not running_on_main_server %}
+      <div style="margin-bottom:12px;padding:10px 14px;background:rgba(245,158,11,0.08);
+        border:1px solid rgba(245,158,11,0.25);border-radius:8px;font-size:12px;color:var(--yellow);">
+        <i class="fas fa-triangle-exclamation"></i>
+        Running as the standby — there's no local cluster here to restore into.
+        "External Machine" is pre-filled to reach the real main server over the standby's
+        SSH tunnel; only change these if you actually mean a different target.
+      </div>
+      {% endif %}
      <div class="radio-group">
-        <label class="radio-card">
-          <input type="radio" name="restore_target" value="local" checked onchange="toggleRemoteFields()">
-          <div class="radio-body"><span class="radio-icon">🎯</span><div><div class="radio-label">Restore on This Server</div><div class="radio-desc" id="this-server-desc">Loading hostname…</div></div></div>
+        <label class="radio-card{% if not running_on_main_server %} disabled{% endif %}">
+          <input type="radio" name="restore_target" value="local"
+            {% if running_on_main_server %}checked{% else %}disabled{% endif %}
+            onchange="toggleRemoteFields()">
+          <div class="radio-body"><span class="radio-icon">🎯</span><div><div class="radio-label">Restore on This Server</div><div class="radio-desc" id="this-server-desc">{% if running_on_main_server %}Loading hostname…{% else %}Not available on the standby{% endif %}</div></div></div>
        </label>
        <label class="radio-card">
-          <input type="radio" name="restore_target" value="remote" onchange="toggleRemoteFields()">
+          <input type="radio" name="restore_target" value="remote" {% if not running_on_main_server %}checked{% endif %} onchange="toggleRemoteFields()">
          <div class="radio-body"><span class="radio-icon">📡</span><div><div class="radio-label">External Machine</div><div class="radio-desc">via SSH — any IP</div></div></div>
        </label>
      </div>

-      <div id="remote-fields" style="display:none; margin-top:16px; max-width:560px;">
+      <div id="remote-fields" style="display:{% if running_on_main_server %}none{% else %}block{% endif %}; margin-top:16px; max-width:560px;">
        <div class="form-row">
          <div class="form-group">
            <label class="form-label">TARGET IP</label>
-            <input type="text" id="remote-ip" class="form-input" placeholder="192.168.x.x or IP">
+            <input type="text" id="remote-ip" class="form-input" placeholder="192.168.x.x or IP" value="{{ tunnel_remote_ip if not running_on_main_server else '' }}">
          </div>
          <div class="form-group" style="max-width:100px;">
            <label class="form-label">SSH PORT</label>
-            <input type="text" id="remote-port" class="form-input" value="22">
+            <input type="text" id="remote-port" class="form-input" value="{{ tunnel_remote_port if not running_on_main_server else '22' }}">
          </div>
          <div class="form-group" style="max-width:120px;">
            <label class="form-label">SSH USER</label>
--- a/scripts/sync-standby-platform.sh
+++ b/scripts/sync-standby-platform.sh
@@ -0,0 +1,127 @@
+#!/bin/bash
+# ─────────────────────────────────────────────────────────────
+# sync-standby-platform.sh
+# Keeps the warm-standby copy of management-platform on the VM
+# (178.18.243.51) in sync with the real source of truth on this
+# server (/root/CloudOps/platform), pushed directly over SSH.
+#
+# Replaces the old pull-platform-backup.sh + deploy-platform.sh
+# pair on the VM, which pulled/redeployed a platform-backup-*.tar.gz
+# tarball of /root/management-platform (the pre-migration Docker-era
+# deploy dir). That tarball generation was disabled on 2026-08-14
+# when the platform's source of truth moved to git-tracked
+# /root/CloudOps + Jenkins — nothing replaced it on the VM side, so
+# the standby silently kept re-deploying its last cached snapshot
+# (found stale at 2+ months old during PFE failover investigation).
+#
+# config.py is intentionally NOT sourced from /root/CloudOps/platform
+# (gitignored there, and can drift — see that file's own history).
+# The live /root/management-platform/config.py on THIS server is the
+# real one actually mounted into the running pod; it already contains
+# the RUNNING_ON_MAIN_SERVER auto-detect logic that makes the exact
+# same file behave correctly on both this server and the VM, so it's
+# copied byte-for-byte rather than templated.
+#
+# Run hourly via cron on this server (NOT on the VM — the VM has no
+# reason to pull, this server already pushes everywhere else it
+# manages, e.g. backup-k8s-apps.sh's VM tier).
+# ─────────────────────────────────────────────────────────────
+
+VM_HOST="178.18.243.51"
+VM_PORT="22"
+VM_USER="root"
+VM_KEY="/root/.ssh/id_rsa"
+VM_DEPLOY_DIR="/root/management-platform"
+
+LOCAL_SRC="/root/CloudOps/platform/"
+LIVE_CONFIG="/root/management-platform/config.py"
+
+# restore_start()/api_backup_run() in app.py reference this as a fixed
+# absolute path (/root/CloudOps/backup/restore-k8s-apps.sh) — it's a
+# sibling of platform/, never baked into the image, reached via the same
+# rw hostPath mount as everything else. Restoring FROM the standby (target
+# = remote, pointed at the main server over the tunnel) needs a local copy
+# here to scp across, so mirror it too, not just platform/.
+LOCAL_BACKUP_SRC="/root/CloudOps/backup/"
+VM_BACKUP_DIR="/root/CloudOps/backup"
+LOG="/root/CloudOps/scripts/sync-standby-platform.log"
+
+SSH_OPTS="-i $VM_KEY -p $VM_PORT -o StrictHostKeyChecking=no -o ConnectTimeout=15"
+SCP_OPTS="-i $VM_KEY -P $VM_PORT -o StrictHostKeyChecking=no -o ConnectTimeout=15"
+
+log() { echo "$(date '+%Y-%m-%d %H:%M:%S') $1" | tee -a "$LOG"; }
+
+log "🔄 Syncing platform code → $VM_USER@$VM_HOST:$VM_DEPLOY_DIR"
+
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "mkdir -p $VM_DEPLOY_DIR" || {
+    log "❌ Cannot reach VM — aborting"
+    exit 1
+}
+
+# --itemize-changes so we can tell below whether anything actually
+# changed (drives the restart-only-if-needed decision).
+CHANGES=$(rsync -a --delete -e "ssh $SSH_OPTS" \
+    --exclude venv --exclude __pycache__ --exclude '*.pyc' \
+    --exclude config.py --exclude '*.log' \
+    --itemize-changes \
+    "$LOCAL_SRC" "$VM_USER@$VM_HOST:$VM_DEPLOY_DIR/" 2>>"$LOG")
+RSYNC_RC=$?
+
+if [ $RSYNC_RC -ne 0 ]; then
+    log "❌ rsync failed (rc=$RSYNC_RC)"
+    exit 1
+fi
+
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "mkdir -p $VM_BACKUP_DIR"
+BACKUP_CHANGES=$(rsync -a --delete -e "ssh $SSH_OPTS" \
+    --itemize-changes \
+    "$LOCAL_BACKUP_SRC" "$VM_USER@$VM_HOST:$VM_BACKUP_DIR/" 2>>"$LOG")
+if [ $? -ne 0 ]; then
+    log "❌ rsync of backup/ failed"
+    exit 1
+fi
+CHANGES="${CHANGES}${BACKUP_CHANGES}"
+
+CONFIG_CHANGED=""
+if ! scp $SCP_OPTS "$LIVE_CONFIG" "$VM_USER@$VM_HOST:$VM_DEPLOY_DIR/config.py.new" &>>"$LOG"; then
+    log "❌ Could not copy live config.py to VM"
+    exit 1
+fi
+if ! ssh $SSH_OPTS "$VM_USER@$VM_HOST" \
+    "cmp -s $VM_DEPLOY_DIR/config.py $VM_DEPLOY_DIR/config.py.new 2>/dev/null"; then
+    CONFIG_CHANGED="yes"
+fi
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "mv $VM_DEPLOY_DIR/config.py.new $VM_DEPLOY_DIR/config.py"
+
+if [ -z "$CHANGES" ] && [ -z "$CONFIG_CHANGED" ]; then
+    log "✅ Already up to date — nothing changed, no restart needed"
+    exit 0
+fi
+
+log "📝 Changes detected:"
+echo "$CHANGES" | tee -a "$LOG"
+[ -n "$CONFIG_CHANGED" ] && log "   (config.py changed too)"
+
+log "📦 Syncing venv deps (requirements.txt) ..."
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "
+    cd $VM_DEPLOY_DIR
+    chmod +x *.sh 2>/dev/null
+    chmod +x *.py 2>/dev/null
+    if [ ! -d venv ]; then
+        python3 -m venv venv
+        venv/bin/python -m ensurepip --upgrade 2>/dev/null || true
+    fi
+    venv/bin/pip install --quiet -r requirements.txt
+" >>"$LOG" 2>&1
+
+log "🔁 Restarting management-platform service on VM ..."
+ssh $SSH_OPTS "$VM_USER@$VM_HOST" "systemctl restart management-platform"
+sleep 3
+STATUS=$(ssh $SSH_OPTS "$VM_USER@$VM_HOST" "systemctl is-active management-platform 2>/dev/null")
+
+if [ "$STATUS" = "active" ]; then
+    log "✅ Standby platform restarted and running latest code"
+else
+    log "⚠️  Service not active after restart (status: $STATUS) — check journalctl -u management-platform on the VM"
+    exit 1
+fi
Author	SHA1	Message	Date
root	9361b9c4c1	Make restore-from-standby actually work, not just look wired Two gaps found testing this live from the standby: 1. restore.html's "Restore on This Server" was checked by default regardless of RUNNING_ON_MAIN_SERVER — on the standby that option is nonsensical (no local cluster/kubectl at all) and restore_start() would have just tried and failed confusingly. Now: that radio is disabled with an explanatory note when not on the main server, "External Machine" is checked instead and pre-filled with the tunnel details (localhost:2224, contabo-key) so restoring from the standby just targets the real main server without the user having to know any of that. Added a matching server-side guard in restore_start() for target=='local' + not RUNNING_ON_MAIN_SERVER (defense in depth — the UI already prevents it, this catches a direct API call too). Also fixed refreshSystemMetrics() in platform.js, which would have overwritten the disabled option's label with the (now-correct, see previous commit) main-server hostname — looking like a working local target when it isn't. 2. sync-standby-platform.sh only ever mirrored platform/ — but restore_start() references /root/CloudOps/backup/restore-k8s-apps.sh as a fixed absolute path to scp to the remote target, and that directory never existed on the VM at all. Every restore attempt from the standby failed immediately with "restore-k8s-apps.sh not found", regardless of target. Now mirrors /root/CloudOps/backup/ too. Verified end-to-end for real: triggered a restore of frappe/erpnext from the standby's actual web UI (target=remote, localhost:2224) — connected over the tunnel, copied the backup archive + script to the main server, ran restore-k8s-apps.sh there, scaled the deployment down/up, restored the DB. Confirmed after: 740 tables in the DB, /api/method/ping responding on the live pod. Not a dry run — a real restore, actually initiated from the standby machine.	2026-08-21 13:55:43 +02:00
root	11b9f6f0c7	Add SSH+kubectl remote fallback so the standby actually shows cluster/app health Two real gaps found after testing the live standby platform: 1. modules/kubernetes.py's in-cluster client (load_incluster_config()) can only ever work inside the real k8s pod. On the standby it silently returned K8S_AVAILABLE=False with zero fallback — Cluster, Containers- via-sites, and any cluster/app health data was just empty on that page, not erroring, so it was easy to miss. This holds even if the standby later runs as a pod on a local k3s on that server (per plan for the monitoring migration) — in-cluster config there would only ever see that local, unrelated cluster, never the main one. Fixed by adding an SSH+kubectl remote path to list_pods(), list_ deployments(), get_ingress_info(), list_deployments_for_namespace(), and get_pod_metrics(): when not RUNNING_ON_MAIN_SERVER, each runs `kubectl <verb> -o json` on the main server over the tunnel (main-to-vm-tunnel.service) instead of using the python client, parses the raw (camelCase) JSON directly rather than trying to force it through the client-library's snake_case object model, and feeds the same _pod_health/_deploy_health/_age_str helpers either way. The in-cluster path for the real main-server pod is untouched. 2. get_system_info() called psutil.* unconditionally with no RUNNING_ON_MAIN_SERVER awareness at all — on the standby it was silently reporting the VM's OWN cpu/mem/disk/hostname mislabeled as "system info" (hostname field literally showed the VM's hostname). Added _get_main_server_system_info_remote(): SSHes to the main server and gathers the same stats via vmstat/free/df/procfs (the main server's bare host has no psutil — that's only in this app's own container image — so this avoids depending on it being present remotely). Verified end-to-end on the live standby: /api/system now reports the real main server's hostname/cpu/mem/disk/uptime; /api/cluster reports real counts (19 pods, 16/16 deployments healthy, 3 degraded, plus live per-pod metrics via the metrics-server raw API); /api/sites shows real per-app container/role status. All previously silently empty.	2026-08-21 13:50:02 +02:00
root	4dfd512785	Wire up a real main<->VM SSH path for the standby's remote-info feature The standby's "SSH to main server for containers/stats/system info" fallback (RUNNING_ON_MAIN_SERVER=False path) was wired in code across backups.py/users.py/commands.py/app.py but never actually worked: outbound SSH from the VM is blocked by its own firewall (deliberate hardening — ufw DENY OUT on 22/tcp and 2222/tcp, left untouched), and separately the VM's contabo-key was never added to the main server's authorized_keys in the first place. Fixed via a persistent reverse tunnel instead of opening the VM's firewall: main-to-vm-tunnel.service (systemd, auto-restart) runs on the main server and keeps `ssh -R 2224:localhost:22` open to the VM, so the VM can reach the main server's SSH via localhost:2224 without ever needing outbound access itself. Added the standby's public key to the main server's authorized_keys (chattr +i-locked — unlocked, appended, re-locked immediately). config.py (gitignored, not in this commit — edited live on both servers directly) gets new MAIN_SERVER_SSH_HOST/PORT ("localhost"/"2224"), kept deliberately separate from the existing MAIN_SERVER_IP (which stays the real IP and is still used for on-page display in several templates — overloading it for the tunnel target would have silently changed what's shown in the UI). All four call sites switched from MAIN_SERVER_IP/MAIN_SERVER_PORT to the new pair. Also fixed a pre-existing bug in commands.py's run_command(): its SSH fallback had no -i key at all, so it could never have authenticated even with the tunnel in place. Verified end-to-end on the live standby: run_command(), backups.py's and users.py's _ssh_main() all confirmed to actually execute on the real main server (hostname/whoami/backup count all matched), not erroring or silently no-op'ing.	2026-08-21 13:26:07 +02:00
root	5329799a5a	Add sync-standby-platform.sh, replacing the dead standby-sync pipeline The VM standby's management-platform was found 2+ months stale during failover investigation (last deploy: June 10). Root cause: the old pull-platform-backup.sh + deploy-platform.sh pair on the VM pulled/ redeployed platform-backup-*.tar.gz tarballs of /root/management-platform, but tarball generation was disabled 2026-08-14 when the source of truth moved to git-tracked /root/CloudOps — nothing replaced it, so the VM kept re-deploying its last cached snapshot every hour indefinitely. New script runs hourly from the main server (cron) instead of pulling from the VM — pushes /root/CloudOps/platform directly via rsync, plus the live /root/management-platform/config.py (gitignored, holds real secrets/DB creds — copied byte-for-byte since it already contains the RUNNING_ON_MAIN_SERVER auto-detect logic that makes it work correctly unmodified on both servers). Only restarts the systemd service when rsync or config actually changed. Old pull/deploy cron entries commented out on the VM (superseded, not deleted, for history). Verified end-to-end: ran it live, standby now serves current code (status- dot markers present in synced templates/app.py), service restarted clean, config auto-detect correctly logs "running on VM / backup host" and takes the SSH-fallback code path as designed.	2026-08-21 13:04:05 +02:00
root	995e07c894	Fix two restore-k8s-apps.sh bugs found verifying Nextcloud restore 1. Secret apply conflict: secret.yaml is a raw `kubectl get -o yaml` dump that includes the live resourceVersion/uid/creationTimestamp at capture time. Applying a stale resourceVersion trips optimistic-concurrency control ("the object has been modified") on effectively every restore, since normal cluster activity keeps bumping it. Strip those three volatile fields before `kubectl apply`. 2. Wrong re-seed source for Nextcloud: the re-seed step added for the PVC exclusion fix assumed both Mautic and Nextcloud bake their excluded dirs in at their own PVC mount path (/var/www/html), copying from pvc_path/$dir. True for Mautic, wrong for Nextcloud — its image keeps the shipped source tree at /usr/src/nextcloud, completely separate from /var/www/html (empty in the raw image at that path). Caught in testing ("cp: cannot stat '/var/www/html/apps'") because the reseed_failed gate from the previous fix did its job and left the app at 0 replicas instead of booting broken — but still required a live manual fix to bring Nextcloud back after the failed test. New RECONSTRUCT_SRC table makes the copy source explicit and independently verified per app instead of assumed by analogy: confirmed /usr/src/nextcloud/{apps,core,dist, 3rdparty,lib} exist in a bare nextcloud:32 pod with no volume mounted before relying on it. Verified end-to-end after both fixes: fresh Nextcloud backup -> restore -> data/ intact (184M, 89 files, unchanged) -> occ status installed/healthy -> 200 on /status.php, via the actual script run (not just the manual recovery), confirming the automated path works, not just my live fix.	2026-08-21 12:37:16 +02:00
root	fca77d92b5	Exclude reproducible framework code from Mautic/Nextcloud PVC backups Mautic (node_modules/, vendor/) and Nextcloud (apps/, core/, dist/, 3rdparty/, lib/) share one PVC mount with their real user data, and were backing up ~700MB-1GB of app/vendor code that's identical to what ships in the image, alongside the actual user data. Excluded both via tar --exclude in the PVC capture step: Mautic backup drops from ~185MB to ~90MB compressed, Nextcloud from ~379MB to ~107MB. Verified the "the image lays these down fresh on start" assumption before relying on it — it does NOT hold for either app as a naive exclude: - Mautic's entrypoint has no logic at all to reconstruct node_modules/ vendor if missing (confirmed empty by reading /entrypoint.sh). - Nextcloud's entrypoint only rsyncs from /usr/src/nextcloud when image_version > installed_version (read from version.php). Since version.php itself isn't excluded, a restored version.php already matches the running image's version, so that path never fires. So restore-k8s-apps.sh now explicitly re-seeds these dirs from the app's own live image (both images bake them in at /var/www/html independently of any volume mount) via a loader pod with the PVC mounted at a different path, after the normal PVC-data restore and before scaling the app back up. If that re-seed fails, the app is deliberately left at 0 replicas instead of coming back up broken (missing vendor/autoload.php is a hard crash, not a degraded state) — found this the hard way in testing when a 90s wait timed out mid image-pull and scale-up proceeded anyway with vendor/ missing; fixed by gating scale-up on the re-seed outcome and raising the timeout to 240s with imagePullPolicy: IfNotPresent (the image is already on-node, pulled off the live Deployment spec). Verified end-to-end: real backup + restore of Mautic on the live cluster, confirming size drop, vendor/autoload.php present post-restore, app serving 200s, and all 433,681 leads intact via the DB restore. Also found and left unfixed (pre-existing, unrelated): secret.yaml apply during restore can hit a resourceVersion conflict from kubectl apply against a captured manifest — the mautic-secrets apply failed on this restore test with a benign "object has been modified" error since the secret already existed with correct values; app was unaffected. Separate bug from this change, flagging for later.	2026-08-21 11:07:02 +02:00
root	72dba7a3a4	Fix backup status dots disappearing after page load /api/backups returned plain filename strings while pages/backups.html's server-rendered dots came from get_local_backups_with_status()/ get_vm_backups_with_status(). platform.js's DOMContentLoaded handler always calls refreshBackupsList() (backups.html has #backup-history-list, which triggers it unconditionally), which re-fetches /api/backups and rebuilds #local-backup-list/#vm-backup-list via renderBackupList() — a version with zero knowledge of status, wiping out the dots seconds after initial paint. Switch /api/backups to the *_with_status() variants and have renderBackupList() render the same status-dot markup as the Jinja template, so the dot is sourced from persisted backup metadata on every render path, not just the first one.	2026-08-21 11:07:01 +02:00
root	d498c81d5b	Add per-backup status dot (green/yellow/red/unknown) to the UI backup-k8s-apps.sh now tracks step-level failures (manifests, secret, db_dump, pvc_data) into an ERRORS array during the per-app loop, and writes a .meta.json sidecar alongside each archive once all 3 storage tiers are known - apps included, final size, the error list, and per-tier ok/failed/skipped status. Sidecar is mirrored to VM/R2 the same best-effort way the .sha256 sidecar already is, and cleaned up by both local and R2 retention pruning plus manual delete. modules/backups.py reads these sidecars (never decompresses the archive) to compute a status per backup at list-render time: - red: any recorded error, or any tier status starting with "failed" - yellow: no errors, but size deviates >40% from the rolling average of the last 5 backups sharing the same app-combination (apps_key) - green: no errors, size within range (or first backup of its app-combination - nothing to compare against yet) - unknown: no sidecar at all (legacy myapps-backup-* archives, or any k8s backup made before this shipped) - no backfill attempted, old runs never recorded step-level failures to reconstruct from /backups route now passes get_local_backups_with_status()/ get_vm_backups_with_status() instead of the plain filename lists (get_local_backups()/get_vm_backups() themselves are untouched - /restore and /api/backups still use the plain versions, they don't need the dot). Template renders a colored dot next to each entry with a tooltip showing apps/size. Verified: real n8n backup produces a correct sidecar synced to all 3 tiers; JSON-writer argv parsing and the red/yellow/green/unknown decision logic each checked against synthetic cases; full Jinja render checked against real local + VM data pulled from the live pod. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA	2026-08-21 02:46:50 +02:00