Two gaps found testing this live from the standby:
1. restore.html's "Restore on This Server" was checked by default
regardless of RUNNING_ON_MAIN_SERVER — on the standby that option is
nonsensical (no local cluster/kubectl at all) and restore_start()
would have just tried and failed confusingly. Now: that radio is
disabled with an explanatory note when not on the main server,
"External Machine" is checked instead and pre-filled with the tunnel
details (localhost:2224, contabo-key) so restoring from the standby
just targets the real main server without the user having to know
any of that. Added a matching server-side guard in restore_start()
for target=='local' + not RUNNING_ON_MAIN_SERVER (defense in depth —
the UI already prevents it, this catches a direct API call too).
Also fixed refreshSystemMetrics() in platform.js, which would have
overwritten the disabled option's label with the (now-correct, see
previous commit) main-server hostname — looking like a working local
target when it isn't.
2. sync-standby-platform.sh only ever mirrored platform/ — but
restore_start() references /root/CloudOps/backup/restore-k8s-apps.sh
as a fixed absolute path to scp to the remote target, and that
directory never existed on the VM at all. Every restore attempt from
the standby failed immediately with "restore-k8s-apps.sh not found",
regardless of target. Now mirrors /root/CloudOps/backup/ too.
Verified end-to-end for real: triggered a restore of frappe/erpnext from
the standby's actual web UI (target=remote, localhost:2224) — connected
over the tunnel, copied the backup archive + script to the main server,
ran restore-k8s-apps.sh there, scaled the deployment down/up, restored
the DB. Confirmed after: 740 tables in the DB, /api/method/ping
responding on the live pod. Not a dry run — a real restore, actually
initiated from the standby machine.
backup-k8s-apps.sh now tracks step-level failures (manifests, secret,
db_dump, pvc_data) into an ERRORS array during the per-app loop, and
writes a .meta.json sidecar alongside each archive once all 3 storage
tiers are known - apps included, final size, the error list, and
per-tier ok/failed/skipped status. Sidecar is mirrored to VM/R2 the
same best-effort way the .sha256 sidecar already is, and cleaned up
by both local and R2 retention pruning plus manual delete.
modules/backups.py reads these sidecars (never decompresses the
archive) to compute a status per backup at list-render time:
- red: any recorded error, or any tier status starting with "failed"
- yellow: no errors, but size deviates >40% from the rolling average
of the last 5 backups sharing the same app-combination (apps_key)
- green: no errors, size within range (or first backup of its
app-combination - nothing to compare against yet)
- unknown: no sidecar at all (legacy myapps-backup-* archives, or
any k8s backup made before this shipped) - no backfill attempted,
old runs never recorded step-level failures to reconstruct from
/backups route now passes get_local_backups_with_status()/
get_vm_backups_with_status() instead of the plain filename lists
(get_local_backups()/get_vm_backups() themselves are untouched -
/restore and /api/backups still use the plain versions, they don't
need the dot). Template renders a colored dot next to each entry with
a tooltip showing apps/size.
Verified: real n8n backup produces a correct sidecar synced to all 3
tiers; JSON-writer argv parsing and the red/yellow/green/unknown
decision logic each checked against synthetic cases; full Jinja render
checked against real local + VM data pulled from the live pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
New modules/kubernetes.py lists pods and deployments across the 8 app
namespaces via the in-cluster management-platform-viewer-sa (read-only,
no secrets/exec/log), with metrics-server usage as a best-effort extra
that degrades silently when unreachable. New /cluster page and
/api/cluster endpoint, nav entry, and templates render pods/deployments
grouped by namespace using the existing card/badge design system.
Additive only — no existing Docker routes or views touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
