Prompted by today's testing: running backup-k8s-apps.sh --apps X repeatedly
for isolated per-app verification left a bunch of single-app archives in
the list, all named identically (myapps-k8s-backup-TIMESTAMP.tar.gz) with
no visible indication of which app(s) each one actually contains — the
data was already there (the .meta.json sidecar's `apps` field, used for
the status-dot rolling average) but only surfaced in a hover tooltip.
Cleaned up today's test archives (local + VM + R2) and ran one fresh full
backup so the list reflects real state.
Two things added, backup/restore logic itself untouched per explicit
instruction:
1. A small visible badge next to each backup's name — "All apps" for a
full bundle, or the specific list (e.g. "odoo, n8n") for a partial one
— in both the Jinja-rendered list and platform.js's refresh path.
2. An optional app-picker on the "Run Backup Now" manual trigger, same
checkbox-grid pattern already used on the Restore page's app selector.
All checked (default) = exactly today's existing behavior, no --apps
flag, everything bundled into one archive. Unchecking some = a
deliberate one-off partial backup (--apps a,b) for e.g. backing up just
odoo before a risky change without waiting on the other 4. The nightly
cron and "backup all together" behavior are completely unaffected —
this only touches the manual/UI-triggered path.
Verified: manual full backup still produces one bundled archive (249M, 5
apps); the apps-badge renders correctly ("All apps") on the live standby;
the --apps arg-construction logic unit-tested directly (partial selection
-> ['--apps', 'a,b'], all-selected -> [] i.e. default bundle).
Two gaps found testing this live from the standby:
1. restore.html's "Restore on This Server" was checked by default
regardless of RUNNING_ON_MAIN_SERVER — on the standby that option is
nonsensical (no local cluster/kubectl at all) and restore_start()
would have just tried and failed confusingly. Now: that radio is
disabled with an explanatory note when not on the main server,
"External Machine" is checked instead and pre-filled with the tunnel
details (localhost:2224, contabo-key) so restoring from the standby
just targets the real main server without the user having to know
any of that. Added a matching server-side guard in restore_start()
for target=='local' + not RUNNING_ON_MAIN_SERVER (defense in depth —
the UI already prevents it, this catches a direct API call too).
Also fixed refreshSystemMetrics() in platform.js, which would have
overwritten the disabled option's label with the (now-correct, see
previous commit) main-server hostname — looking like a working local
target when it isn't.
2. sync-standby-platform.sh only ever mirrored platform/ — but
restore_start() references /root/CloudOps/backup/restore-k8s-apps.sh
as a fixed absolute path to scp to the remote target, and that
directory never existed on the VM at all. Every restore attempt from
the standby failed immediately with "restore-k8s-apps.sh not found",
regardless of target. Now mirrors /root/CloudOps/backup/ too.
Verified end-to-end for real: triggered a restore of frappe/erpnext from
the standby's actual web UI (target=remote, localhost:2224) — connected
over the tunnel, copied the backup archive + script to the main server,
ran restore-k8s-apps.sh there, scaled the deployment down/up, restored
the DB. Confirmed after: 740 tables in the DB, /api/method/ping
responding on the live pod. Not a dry run — a real restore, actually
initiated from the standby machine.
/api/backups returned plain filename strings while pages/backups.html's
server-rendered dots came from get_local_backups_with_status()/
get_vm_backups_with_status(). platform.js's DOMContentLoaded handler always
calls refreshBackupsList() (backups.html has #backup-history-list, which
triggers it unconditionally), which re-fetches /api/backups and rebuilds
#local-backup-list/#vm-backup-list via renderBackupList() — a version with
zero knowledge of status, wiping out the dots seconds after initial paint.
Switch /api/backups to the *_with_status() variants and have
renderBackupList() render the same status-dot markup as the Jinja template,
so the dot is sourced from persisted backup metadata on every render path,
not just the first one.
backup-k8s-apps.sh now tracks step-level failures (manifests, secret,
db_dump, pvc_data) into an ERRORS array during the per-app loop, and
writes a .meta.json sidecar alongside each archive once all 3 storage
tiers are known - apps included, final size, the error list, and
per-tier ok/failed/skipped status. Sidecar is mirrored to VM/R2 the
same best-effort way the .sha256 sidecar already is, and cleaned up
by both local and R2 retention pruning plus manual delete.
modules/backups.py reads these sidecars (never decompresses the
archive) to compute a status per backup at list-render time:
- red: any recorded error, or any tier status starting with "failed"
- yellow: no errors, but size deviates >40% from the rolling average
of the last 5 backups sharing the same app-combination (apps_key)
- green: no errors, size within range (or first backup of its
app-combination - nothing to compare against yet)
- unknown: no sidecar at all (legacy myapps-backup-* archives, or
any k8s backup made before this shipped) - no backfill attempted,
old runs never recorded step-level failures to reconstruct from
/backups route now passes get_local_backups_with_status()/
get_vm_backups_with_status() instead of the plain filename lists
(get_local_backups()/get_vm_backups() themselves are untouched -
/restore and /api/backups still use the plain versions, they don't
need the dot). Template renders a colored dot next to each entry with
a tooltip showing apps/size.
Verified: real n8n backup produces a correct sidecar synced to all 3
tiers; JSON-writer argv parsing and the red/yellow/green/unknown
decision logic each checked against synthetic cases; full Jinja render
checked against real local + VM data pulled from the live pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
New modules/kubernetes.py lists pods and deployments across the 8 app
namespaces via the in-cluster management-platform-viewer-sa (read-only,
no secrets/exec/log), with metrics-server usage as a best-effort extra
that degrades silently when unreachable. New /cluster page and
/api/cluster endpoint, nav entry, and templates render pods/deployments
grouped by namespace using the existing card/badge design system.
Additive only — no existing Docker routes or views touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
