Mautic (node_modules/, vendor/) and Nextcloud (apps/, core/, dist/,
3rdparty/, lib/) share one PVC mount with their real user data, and were
backing up ~700MB-1GB of app/vendor code that's identical to what ships in
the image, alongside the actual user data. Excluded both via tar --exclude
in the PVC capture step: Mautic backup drops from ~185MB to ~90MB
compressed, Nextcloud from ~379MB to ~107MB.
Verified the "the image lays these down fresh on start" assumption before
relying on it — it does NOT hold for either app as a naive exclude:
- Mautic's entrypoint has no logic at all to reconstruct node_modules/
vendor if missing (confirmed empty by reading /entrypoint.sh).
- Nextcloud's entrypoint only rsyncs from /usr/src/nextcloud when
image_version > installed_version (read from version.php). Since
version.php itself isn't excluded, a restored version.php already
matches the running image's version, so that path never fires.
So restore-k8s-apps.sh now explicitly re-seeds these dirs from the app's
own live image (both images bake them in at /var/www/html independently of
any volume mount) via a loader pod with the PVC mounted at a different
path, after the normal PVC-data restore and before scaling the app back
up. If that re-seed fails, the app is deliberately left at 0 replicas
instead of coming back up broken (missing vendor/autoload.php is a hard
crash, not a degraded state) — found this the hard way in testing when a
90s wait timed out mid image-pull and scale-up proceeded anyway with
vendor/ missing; fixed by gating scale-up on the re-seed outcome and
raising the timeout to 240s with imagePullPolicy: IfNotPresent (the image
is already on-node, pulled off the live Deployment spec).
Verified end-to-end: real backup + restore of Mautic on the live cluster,
confirming size drop, vendor/autoload.php present post-restore, app
serving 200s, and all 433,681 leads intact via the DB restore.
Also found and left unfixed (pre-existing, unrelated): secret.yaml apply
during restore can hit a resourceVersion conflict from kubectl apply
against a captured manifest — the mautic-secrets apply failed on this
restore test with a benign "object has been modified" error since the
secret already existed with correct values; app was unaffected. Separate
bug from this change, flagging for later.
/api/backups returned plain filename strings while pages/backups.html's
server-rendered dots came from get_local_backups_with_status()/
get_vm_backups_with_status(). platform.js's DOMContentLoaded handler always
calls refreshBackupsList() (backups.html has #backup-history-list, which
triggers it unconditionally), which re-fetches /api/backups and rebuilds
#local-backup-list/#vm-backup-list via renderBackupList() — a version with
zero knowledge of status, wiping out the dots seconds after initial paint.
Switch /api/backups to the *_with_status() variants and have
renderBackupList() render the same status-dot markup as the Jinja template,
so the dot is sourced from persisted backup metadata on every render path,
not just the first one.
backup-k8s-apps.sh now tracks step-level failures (manifests, secret,
db_dump, pvc_data) into an ERRORS array during the per-app loop, and
writes a .meta.json sidecar alongside each archive once all 3 storage
tiers are known - apps included, final size, the error list, and
per-tier ok/failed/skipped status. Sidecar is mirrored to VM/R2 the
same best-effort way the .sha256 sidecar already is, and cleaned up
by both local and R2 retention pruning plus manual delete.
modules/backups.py reads these sidecars (never decompresses the
archive) to compute a status per backup at list-render time:
- red: any recorded error, or any tier status starting with "failed"
- yellow: no errors, but size deviates >40% from the rolling average
of the last 5 backups sharing the same app-combination (apps_key)
- green: no errors, size within range (or first backup of its
app-combination - nothing to compare against yet)
- unknown: no sidecar at all (legacy myapps-backup-* archives, or
any k8s backup made before this shipped) - no backfill attempted,
old runs never recorded step-level failures to reconstruct from
/backups route now passes get_local_backups_with_status()/
get_vm_backups_with_status() instead of the plain filename lists
(get_local_backups()/get_vm_backups() themselves are untouched -
/restore and /api/backups still use the plain versions, they don't
need the dot). Template renders a colored dot next to each entry with
a tooltip showing apps/size.
Verified: real n8n backup produces a correct sidecar synced to all 3
tiers; JSON-writer argv parsing and the red/yellow/green/unknown
decision logic each checked against synthetic cases; full Jinja render
checked against real local + VM data pulled from the live pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
Two separate bugs found while testing the Audit/Details UI against a
real k8s-format backup (myapps-k8s-backup-*):
1. Format-hardcoded checks. audit_backup()'s Internal Structure and
Volume Count checks assumed the legacy Docker-era archive layout
(volumes/*.tar.gz, compose-files/) and would fail every k8s-format
backup (per-app dirs with manifests.yaml/db-dump.sql.gz/pvc-data.tar.gz)
even when it's perfectly healthy - same root cause as the
myapps-backup-*/myapps-k8s-backup-* prefix bug fixed earlier, just in
the audit checks instead of the listing/delete glob. cloud_backup.py's
r2_audit_backup() had the identical unpatched filename regex, and
app.py's /api/backups/details route had its own, which outright
400'd any k8s-format filename before even looking at it.
2. A bigger, separate bug this surfaced: get_local_backups(),
get_vm_backups(), and _resolve_archive_path() all branch on
RUNNING_ON_MAIN_SERVER to decide between direct filesystem access and
SSH-to-self - and from inside the management-platform pod that flag's
hostname check can be wrong for the pod's own ephemeral hostname
depending on how it's evaluated, sending these down the SSH branch
using a topology (separate warm-standby-on-the-VM SSH path) that
doesn't apply to a pod that already has /root hostPath-mounted in.
Fixed by trying direct filesystem access first wherever the archive
would already be locally reachable, falling back to the existing SSH
paths otherwise - this preserves the original warm-standby-on-VM
failover design (a real second deployment of this platform on the VM
host, kept reachable via SSH when the main server is down) completely
unchanged; it only adds the fast local path for the case where the
caller already has direct filesystem access.
Verified against both a real k8s-format backup (odoo, single-app) and a
real legacy-format backup on disk - both audit correctly now, with
format-appropriate checks and labels.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
modules/sites.py was still a fully Docker-era hardcoded registry:
container names (odoo-clean-odoo-1, frappe-erpnext, nextcloud-app,
mautic-app, n8n-app) that no longer exist post-migration, checked via
`docker inspect` over SSH, plus hardcoded domain/port fields -
nextcloud and mautic even had domain: None, silently falling back to
dead Docker host-port URLs while their real Ingress domains
(next.cloud.nav.ovh, mautics.nav.ovh) sat unused.
Now sources everything live from the cluster:
- Domain + TLS from the actual Ingress object per app (via the new
get_ingress_info() in modules/kubernetes.py), not a static guess.
Odoo/Nextcloud's Ingress lives in `default` (see prior commit for the
RBAC this needed); n8n/mautic/erpnext's lives in their own namespace.
- App/DB/cache/worker/scheduler status from real Deployment state
(list_deployments_for_namespace(), new in modules/kubernetes.py)
instead of `docker inspect`.
- Health probe now hits the real domain directly from the pod (it has
normal internet egress) instead of SSH-ing back out to the host to
curl a public HTTPS URL, which is what the RUNNING_ON_MAIN_SERVER
branch would have done for a pod whose hostname never matches the
main-server hostname check.
Same API/template contract as before (get_sites_list/get_site_health
return the same field shapes) - templates/pages/sites.html needs no
changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
Odoo's and Nextcloud's Ingress objects live in the default namespace
(historical, from the original Docker->k8s migration) rather than their
own app namespace like n8n/mautic/erpnext. management-platform-sa had
no binding there at all. Granting the full backup ClusterRole in
default would be way too broad (default also hosts unrelated live
services, and that role includes secrets/pod writes) - so this adds a
separate ClusterRole scoped to just ingresses get/list/watch, bound
only in default. Needed for the Application Sites page fix (next
commit) to read real domain/TLS state instead of guessing.
Applied and verified live: can read ingresses in default, still denied
secrets/pod-delete there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
Investigated a resource-safety abort during Odoo restore testing:
available RAM was hovering ~1.3-1.5GB even under normal conditions on
this host (k3s + Jenkins + Docker monitoring/wazuh stack + dev tooling
baseline), not a leak from any single process. 1536MB was tripping on
ordinary single-app backups; 1024MB still leaves real headroom above
what a pg_dump/mysqldump/tar step actually needs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
18 straight clone failures today (early EOF / invalid index-pack) even with
http.postBuffer/http.version tuning applied — Gitea's HTTP endpoint itself is
degraded right now, not something fixable client-side. Sidesteps it entirely:
the docker-agent pod template now hostPath-mounts /root/CloudOps (read-only,
see k3s-cloud-init.groovy) and Checkout just `cp -r`s from it, no network
fetch at all. Build/SonarQube paths revert to ./platform/ (no more repo/
subdirectory, since we're not cloning into one).
Real tradeoff, accepted deliberately: builds now reflect whatever's on the
VPS's disk at build time, not necessarily what's pushed to Gitea. Depends on
/root/CloudOps being kept current via the normal git pull workflow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Git plugin's checkout([...]) step doesn't expose http.postBuffer/
http.version tuning, which is what's reliably fixed every stuck
push/clone today (VPS and both laptops) — meanwhile checkout was failing
~10 times in a row, consistently dying at 84-95% during the shallow clone
(early EOF / invalid index-pack). Bypasses the plugin entirely: a plain
`git clone` with those settings applied, wrapped in its own 8-attempt
bash retry loop, using the existing dd13f593-06b1-4a5a-9477-2df0d186b490
credential (confirmed via credentials.xml to be a UsernamePasswordCredentialsImpl,
compatible with the usernamePassword binding as-is).
Code now lands in a repo/ subdirectory instead of the workspace root, so
every stage referencing a repo-relative path was updated to match:
- Build Docker Image: ./platform/ -> ./repo/platform/
- SonarQube Analysis: -Dsonar.sources=platform -> -Dsonar.sources=repo/platform
Deploy to Kubernetes needed no changes — it only references the already-built
image tag and hardcoded kubectl args, no filesystem paths into the checkout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
api_backup_run and restore_start were still hardcoded to the Docker-era
backup-myapps.sh/restore-myapps.sh, which target Docker volumes that no
longer exist for the 5 apps now running in k3s (n8n, odoo, mautic,
nextcloud, frappe/erpnext) — every backup/restore triggered from the UI
has been silently hollow for these apps since the migration.
- api_backup_run: /root/backup-myapps.sh -> /root/CloudOps/backup/backup-k8s-apps.sh
- restore_start: switched from the old image-relative path (which resolved
to /app/restore-myapps.sh, baked into the Docker image from platform/)
to an absolute /root/CloudOps/backup/restore-k8s-apps.sh path — the new
backup/ folder is a sibling of platform/, not part of the Docker build
context, so it's only reachable via the pod's existing /root hostPath
mount, the same way api_backup_run already reaches its script.
--apps flag shape and the job-log streaming contract are unchanged, so no
frontend/template changes needed for the core flow. modules/backups.py's
three hardcoded myapps-backup-* references (get_local_backups, get_vm_backups
x2, delete_backup's filename regex) now also recognize the new
myapps-k8s-backup-* prefix, so both backup lineages stay visible/manageable
in the UI.
Depends on the previous commit (scoped ServiceAccount + k3s kubectl access
for the pod) to actually function when triggered from the UI.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The pod running app.py had no kubectl binary and only a read-only SA
(management-platform-viewer-sa) — backup-k8s-apps.sh/restore-k8s-apps.sh
could never actually run from inside it. Adds:
- management-platform-backup-role (ClusterRole, bound via RoleBinding only
in n8n/odoo/mautic/nextcloud/erpnext — not cluster-wide): get/list/watch
on pods/deployments/configmaps/ingresses/PVCs, pods/exec create, pods
create+delete (needed for restore's populate-before-scale-up loader pod),
secrets get/list/update/patch, deployments/scale update/patch only (no
write on full Deployment/Service/ConfigMap/Ingress specs).
- management-platform-sa, replacing management-platform-viewer-sa as the
pod's identity — inherits the old viewer-role's read-only bindings too
(retargeted in management-platform-viewer-rbac.yaml) so the existing
cluster-view page keeps working under one SA.
- k3s binary hostPath-mounted read-only into the pod (same pattern as the
Jenkins agent's docker-cli container), so `k3s kubectl` is available
where no separate kubectl binary exists.
Both scripts updated to fall back to k3s kubectl when no kubectl is on
PATH, and to use /proc/meminfo instead of `free` for the resource-safety
check (not present in the pod's minimal image). Also fixes a real issue
found while live-testing this from inside the pod: the pod's pre-existing
/root hostPath mount also exposes the host's own admin ~/.kube/config,
which k3s kubectl was silently preferring over the scoped SA token —
forcing --kubeconfig=/dev/null in both scripts closes that.
Verified end-to-end from inside the actual management-platform pod:
manifests/secret/DB-dump/PVC-data backup for n8n succeeds under the new
SA's scoped permissions, and a kube-system access attempt is correctly
rejected (Forbidden) once the kubeconfig leak is closed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
backup-myapps.sh/restore-myapps.sh only ever knew about Docker volumes,
which no longer exist for n8n/odoo/mautic/nextcloud/frappe now that
they're all running in k3s — nightly backups have been silently hollow
for these apps since the migration. These new scripts capture/restore
each app's k8s manifests, Secret, DB (pg_dump/mysqldump via kubectl exec),
and app-data PVC contents (kubectl exec tar, never raw filesystem access),
reusing the existing local+VM+R2 3-2-1 storage/retention logic unchanged.
Tested standalone: full backup of all 5 apps produces verified non-empty
manifests/secrets/DB dumps/PVC data; restore-in-place tested end-to-end
against n8n (DB row counts and PVC data matched exactly pre/post, app
verified healthy over HTTPS).
Not wired into management-platform yet (app.py stays untouched) — restore
script also doesn't yet handle a fresh/DR cluster where the PVCs and their
manifests don't already exist (same-cluster restore only for now).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New modules/kubernetes.py lists pods and deployments across the 8 app
namespaces via the in-cluster management-platform-viewer-sa (read-only,
no secrets/exec/log), with metrics-server usage as a best-effort extra
that degrades silently when unreachable. New /cluster page and
/api/cluster endpoint, nav entry, and templates render pods/deployments
grouped by namespace using the existing card/badge design system.
Additive only — no existing Docker routes or views touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ServiceAccount management-platform-viewer-sa (in management-platform ns),
ClusterRole management-platform-viewer-role (get/list/watch on pods,
services, deployments.apps only - no secrets, no write verbs), and a
RoleBinding per app namespace (n8n, odoo, mautic, erpnext, nextcloud,
jenkins-agents, jenkins, management-platform). Already applied to the
live cluster and verified via kubectl auth can-i; committing here for
source-of-truth tracking.
Source-of-truth manifest for the management-platform Deployment, with
serviceAccountName: management-platform-viewer-sa added so the pod can
authenticate for the read-only k8s cluster-view feature. Not yet applied
to the cluster.
