backup-k8s-apps.sh now tracks step-level failures (manifests, secret,
db_dump, pvc_data) into an ERRORS array during the per-app loop, and
writes a .meta.json sidecar alongside each archive once all 3 storage
tiers are known - apps included, final size, the error list, and
per-tier ok/failed/skipped status. Sidecar is mirrored to VM/R2 the
same best-effort way the .sha256 sidecar already is, and cleaned up
by both local and R2 retention pruning plus manual delete.
modules/backups.py reads these sidecars (never decompresses the
archive) to compute a status per backup at list-render time:
- red: any recorded error, or any tier status starting with "failed"
- yellow: no errors, but size deviates >40% from the rolling average
of the last 5 backups sharing the same app-combination (apps_key)
- green: no errors, size within range (or first backup of its
app-combination - nothing to compare against yet)
- unknown: no sidecar at all (legacy myapps-backup-* archives, or
any k8s backup made before this shipped) - no backfill attempted,
old runs never recorded step-level failures to reconstruct from
/backups route now passes get_local_backups_with_status()/
get_vm_backups_with_status() instead of the plain filename lists
(get_local_backups()/get_vm_backups() themselves are untouched -
/restore and /api/backups still use the plain versions, they don't
need the dot). Template renders a colored dot next to each entry with
a tooltip showing apps/size.
Verified: real n8n backup produces a correct sidecar synced to all 3
tiers; JSON-writer argv parsing and the red/yellow/green/unknown
decision logic each checked against synthetic cases; full Jinja render
checked against real local + VM data pulled from the live pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
Investigated a resource-safety abort during Odoo restore testing:
available RAM was hovering ~1.3-1.5GB even under normal conditions on
this host (k3s + Jenkins + Docker monitoring/wazuh stack + dev tooling
baseline), not a leak from any single process. 1536MB was tripping on
ordinary single-app backups; 1024MB still leaves real headroom above
what a pg_dump/mysqldump/tar step actually needs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
The pod running app.py had no kubectl binary and only a read-only SA
(management-platform-viewer-sa) — backup-k8s-apps.sh/restore-k8s-apps.sh
could never actually run from inside it. Adds:
- management-platform-backup-role (ClusterRole, bound via RoleBinding only
in n8n/odoo/mautic/nextcloud/erpnext — not cluster-wide): get/list/watch
on pods/deployments/configmaps/ingresses/PVCs, pods/exec create, pods
create+delete (needed for restore's populate-before-scale-up loader pod),
secrets get/list/update/patch, deployments/scale update/patch only (no
write on full Deployment/Service/ConfigMap/Ingress specs).
- management-platform-sa, replacing management-platform-viewer-sa as the
pod's identity — inherits the old viewer-role's read-only bindings too
(retargeted in management-platform-viewer-rbac.yaml) so the existing
cluster-view page keeps working under one SA.
- k3s binary hostPath-mounted read-only into the pod (same pattern as the
Jenkins agent's docker-cli container), so `k3s kubectl` is available
where no separate kubectl binary exists.
Both scripts updated to fall back to k3s kubectl when no kubectl is on
PATH, and to use /proc/meminfo instead of `free` for the resource-safety
check (not present in the pod's minimal image). Also fixes a real issue
found while live-testing this from inside the pod: the pod's pre-existing
/root hostPath mount also exposes the host's own admin ~/.kube/config,
which k3s kubectl was silently preferring over the scoped SA token —
forcing --kubeconfig=/dev/null in both scripts closes that.
Verified end-to-end from inside the actual management-platform pod:
manifests/secret/DB-dump/PVC-data backup for n8n succeeds under the new
SA's scoped permissions, and a kube-system access attempt is correctly
rejected (Forbidden) once the kubeconfig leak is closed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
backup-myapps.sh/restore-myapps.sh only ever knew about Docker volumes,
which no longer exist for n8n/odoo/mautic/nextcloud/frappe now that
they're all running in k3s — nightly backups have been silently hollow
for these apps since the migration. These new scripts capture/restore
each app's k8s manifests, Secret, DB (pg_dump/mysqldump via kubectl exec),
and app-data PVC contents (kubectl exec tar, never raw filesystem access),
reusing the existing local+VM+R2 3-2-1 storage/retention logic unchanged.
Tested standalone: full backup of all 5 apps produces verified non-empty
manifests/secrets/DB dumps/PVC data; restore-in-place tested end-to-end
against n8n (DB row counts and PVC data matched exactly pre/post, app
verified healthy over HTTPS).
Not wired into management-platform yet (app.py stays untouched) — restore
script also doesn't yet handle a fresh/DR cluster where the PVCs and their
manifests don't already exist (same-cluster restore only for now).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
