Two gaps found testing this live from the standby:
1. restore.html's "Restore on This Server" was checked by default
regardless of RUNNING_ON_MAIN_SERVER — on the standby that option is
nonsensical (no local cluster/kubectl at all) and restore_start()
would have just tried and failed confusingly. Now: that radio is
disabled with an explanatory note when not on the main server,
"External Machine" is checked instead and pre-filled with the tunnel
details (localhost:2224, contabo-key) so restoring from the standby
just targets the real main server without the user having to know
any of that. Added a matching server-side guard in restore_start()
for target=='local' + not RUNNING_ON_MAIN_SERVER (defense in depth —
the UI already prevents it, this catches a direct API call too).
Also fixed refreshSystemMetrics() in platform.js, which would have
overwritten the disabled option's label with the (now-correct, see
previous commit) main-server hostname — looking like a working local
target when it isn't.
2. sync-standby-platform.sh only ever mirrored platform/ — but
restore_start() references /root/CloudOps/backup/restore-k8s-apps.sh
as a fixed absolute path to scp to the remote target, and that
directory never existed on the VM at all. Every restore attempt from
the standby failed immediately with "restore-k8s-apps.sh not found",
regardless of target. Now mirrors /root/CloudOps/backup/ too.
Verified end-to-end for real: triggered a restore of frappe/erpnext from
the standby's actual web UI (target=remote, localhost:2224) — connected
over the tunnel, copied the backup archive + script to the main server,
ran restore-k8s-apps.sh there, scaled the deployment down/up, restored
the DB. Confirmed after: 740 tables in the DB, /api/method/ping
responding on the live pod. Not a dry run — a real restore, actually
initiated from the standby machine.
The standby's "SSH to main server for containers/stats/system info"
fallback (RUNNING_ON_MAIN_SERVER=False path) was wired in code across
backups.py/users.py/commands.py/app.py but never actually worked:
outbound SSH from the VM is blocked by its own firewall (deliberate
hardening — ufw DENY OUT on 22/tcp and 2222/tcp, left untouched), and
separately the VM's contabo-key was never added to the main server's
authorized_keys in the first place.
Fixed via a persistent reverse tunnel instead of opening the VM's
firewall: main-to-vm-tunnel.service (systemd, auto-restart) runs on the
main server and keeps `ssh -R 2224:localhost:22` open to the VM, so the
VM can reach the main server's SSH via localhost:2224 without ever
needing outbound access itself. Added the standby's public key to the
main server's authorized_keys (chattr +i-locked — unlocked, appended,
re-locked immediately).
config.py (gitignored, not in this commit — edited live on both servers
directly) gets new MAIN_SERVER_SSH_HOST/PORT ("localhost"/"2224"), kept
deliberately separate from the existing MAIN_SERVER_IP (which stays the
real IP and is still used for on-page display in several templates —
overloading it for the tunnel target would have silently changed what's
shown in the UI). All four call sites switched from
MAIN_SERVER_IP/MAIN_SERVER_PORT to the new pair. Also fixed a pre-existing
bug in commands.py's run_command(): its SSH fallback had no -i key at all,
so it could never have authenticated even with the tunnel in place.
Verified end-to-end on the live standby: run_command(), backups.py's and
users.py's _ssh_main() all confirmed to actually execute on the real main
server (hostname/whoami/backup count all matched), not erroring or
silently no-op'ing.
/api/backups returned plain filename strings while pages/backups.html's
server-rendered dots came from get_local_backups_with_status()/
get_vm_backups_with_status(). platform.js's DOMContentLoaded handler always
calls refreshBackupsList() (backups.html has #backup-history-list, which
triggers it unconditionally), which re-fetches /api/backups and rebuilds
#local-backup-list/#vm-backup-list via renderBackupList() — a version with
zero knowledge of status, wiping out the dots seconds after initial paint.
Switch /api/backups to the *_with_status() variants and have
renderBackupList() render the same status-dot markup as the Jinja template,
so the dot is sourced from persisted backup metadata on every render path,
not just the first one.
backup-k8s-apps.sh now tracks step-level failures (manifests, secret,
db_dump, pvc_data) into an ERRORS array during the per-app loop, and
writes a .meta.json sidecar alongside each archive once all 3 storage
tiers are known - apps included, final size, the error list, and
per-tier ok/failed/skipped status. Sidecar is mirrored to VM/R2 the
same best-effort way the .sha256 sidecar already is, and cleaned up
by both local and R2 retention pruning plus manual delete.
modules/backups.py reads these sidecars (never decompresses the
archive) to compute a status per backup at list-render time:
- red: any recorded error, or any tier status starting with "failed"
- yellow: no errors, but size deviates >40% from the rolling average
of the last 5 backups sharing the same app-combination (apps_key)
- green: no errors, size within range (or first backup of its
app-combination - nothing to compare against yet)
- unknown: no sidecar at all (legacy myapps-backup-* archives, or
any k8s backup made before this shipped) - no backfill attempted,
old runs never recorded step-level failures to reconstruct from
/backups route now passes get_local_backups_with_status()/
get_vm_backups_with_status() instead of the plain filename lists
(get_local_backups()/get_vm_backups() themselves are untouched -
/restore and /api/backups still use the plain versions, they don't
need the dot). Template renders a colored dot next to each entry with
a tooltip showing apps/size.
Verified: real n8n backup produces a correct sidecar synced to all 3
tiers; JSON-writer argv parsing and the red/yellow/green/unknown
decision logic each checked against synthetic cases; full Jinja render
checked against real local + VM data pulled from the live pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
Two separate bugs found while testing the Audit/Details UI against a
real k8s-format backup (myapps-k8s-backup-*):
1. Format-hardcoded checks. audit_backup()'s Internal Structure and
Volume Count checks assumed the legacy Docker-era archive layout
(volumes/*.tar.gz, compose-files/) and would fail every k8s-format
backup (per-app dirs with manifests.yaml/db-dump.sql.gz/pvc-data.tar.gz)
even when it's perfectly healthy - same root cause as the
myapps-backup-*/myapps-k8s-backup-* prefix bug fixed earlier, just in
the audit checks instead of the listing/delete glob. cloud_backup.py's
r2_audit_backup() had the identical unpatched filename regex, and
app.py's /api/backups/details route had its own, which outright
400'd any k8s-format filename before even looking at it.
2. A bigger, separate bug this surfaced: get_local_backups(),
get_vm_backups(), and _resolve_archive_path() all branch on
RUNNING_ON_MAIN_SERVER to decide between direct filesystem access and
SSH-to-self - and from inside the management-platform pod that flag's
hostname check can be wrong for the pod's own ephemeral hostname
depending on how it's evaluated, sending these down the SSH branch
using a topology (separate warm-standby-on-the-VM SSH path) that
doesn't apply to a pod that already has /root hostPath-mounted in.
Fixed by trying direct filesystem access first wherever the archive
would already be locally reachable, falling back to the existing SSH
paths otherwise - this preserves the original warm-standby-on-VM
failover design (a real second deployment of this platform on the VM
host, kept reachable via SSH when the main server is down) completely
unchanged; it only adds the fast local path for the case where the
caller already has direct filesystem access.
Verified against both a real k8s-format backup (odoo, single-app) and a
real legacy-format backup on disk - both audit correctly now, with
format-appropriate checks and labels.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
api_backup_run and restore_start were still hardcoded to the Docker-era
backup-myapps.sh/restore-myapps.sh, which target Docker volumes that no
longer exist for the 5 apps now running in k3s (n8n, odoo, mautic,
nextcloud, frappe/erpnext) — every backup/restore triggered from the UI
has been silently hollow for these apps since the migration.
- api_backup_run: /root/backup-myapps.sh -> /root/CloudOps/backup/backup-k8s-apps.sh
- restore_start: switched from the old image-relative path (which resolved
to /app/restore-myapps.sh, baked into the Docker image from platform/)
to an absolute /root/CloudOps/backup/restore-k8s-apps.sh path — the new
backup/ folder is a sibling of platform/, not part of the Docker build
context, so it's only reachable via the pod's existing /root hostPath
mount, the same way api_backup_run already reaches its script.
--apps flag shape and the job-log streaming contract are unchanged, so no
frontend/template changes needed for the core flow. modules/backups.py's
three hardcoded myapps-backup-* references (get_local_backups, get_vm_backups
x2, delete_backup's filename regex) now also recognize the new
myapps-k8s-backup-* prefix, so both backup lineages stay visible/manageable
in the UI.
Depends on the previous commit (scoped ServiceAccount + k3s kubectl access
for the pod) to actually function when triggered from the UI.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New modules/kubernetes.py lists pods and deployments across the 8 app
namespaces via the in-cluster management-platform-viewer-sa (read-only,
no secrets/exec/log), with metrics-server usage as a best-effort extra
that degrades silently when unreachable. New /cluster page and
/api/cluster endpoint, nav entry, and templates render pods/deployments
grouped by namespace using the existing card/badge design system.
Additive only — no existing Docker routes or views touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
