Prompted by today's testing: running backup-k8s-apps.sh --apps X repeatedly
for isolated per-app verification left a bunch of single-app archives in
the list, all named identically (myapps-k8s-backup-TIMESTAMP.tar.gz) with
no visible indication of which app(s) each one actually contains — the
data was already there (the .meta.json sidecar's `apps` field, used for
the status-dot rolling average) but only surfaced in a hover tooltip.
Cleaned up today's test archives (local + VM + R2) and ran one fresh full
backup so the list reflects real state.
Two things added, backup/restore logic itself untouched per explicit
instruction:
1. A small visible badge next to each backup's name — "All apps" for a
full bundle, or the specific list (e.g. "odoo, n8n") for a partial one
— in both the Jinja-rendered list and platform.js's refresh path.
2. An optional app-picker on the "Run Backup Now" manual trigger, same
checkbox-grid pattern already used on the Restore page's app selector.
All checked (default) = exactly today's existing behavior, no --apps
flag, everything bundled into one archive. Unchecking some = a
deliberate one-off partial backup (--apps a,b) for e.g. backing up just
odoo before a risky change without waiting on the other 4. The nightly
cron and "backup all together" behavior are completely unaffected —
this only touches the manual/UI-triggered path.
Verified: manual full backup still produces one bundled archive (249M, 5
apps); the apps-badge renders correctly ("All apps") on the live standby;
the --apps arg-construction logic unit-tested directly (partial selection
-> ['--apps', 'a,b'], all-selected -> [] i.e. default bundle).
Previously restore_start()'s 'remote' target did its own manual scp+ssh+
restore-k8s-apps.sh dance, entirely separate from the new DR bootstrap
playbook (ansible/dr-bootstrap.yml) — meaning restoring onto a genuinely
empty server would just fail (no namespace/PVC/Deployments, and
restore-k8s-apps.sh assumes those exist). Every "External Machine" restore
now runs the same playbook instead: it's safe for both cases, not just the
empty-server one — create-namespace-if-missing and `kubectl apply` for the
sanitized PVC/Secret/manifests are no-ops against a target that already
has this app running with a matching spec (apply only reconciles
differences, and a backup's own captured manifests are by definition
identical to what's already live) — so "restore onto an existing cluster"
and "restore onto nothing" are the same command now; the playbook's own
checks-then-acts steps decide how much of it actually needs to do
anything.
Three real bugs found getting this working, not just wiring it up blind:
1. dr-bootstrap.yml's `hosts: dr_target` only matches a named inventory
group — the dynamic single-host inventory app.py builds per-request
(`-i '<ip>,'`) doesn't create one, so nothing matched and the play
silently skipped. Changed to `hosts: all`, which both invocation styles
satisfy.
2. ansible-playbook is a pip console-script installed next to whichever
python is running — the main pod's system python (no venv there) or
this server's own venv/bin on the standby. A bare "ansible-playbook" in
the shelled-out command only resolves on the main pod; the standby's
venv/bin is never on PATH when its python is invoked directly rather
than through `activate`, so it'd fail there. Resolved relative to
sys.executable instead, which is correct in both.
3. Passing connection options via `-e ansible_ssh_common_args='-o
StrictHostKeyChecking=no ...'` hit a real bug in this ansible-core
version's SSH connection plugin: its own internal tty-detection
re-parses that string with a strict argparse and throws "argument -o:
expected one argument" even for one well-formed -o KEY=VALUE — verified
directly on the CLI, not a shell-quoting artifact from this code.
Switched to ANSIBLE_HOST_KEY_CHECKING=False / ANSIBLE_TIMEOUT=15 env
vars, ansible's own dedicated mechanism for the same effect, which
bypasses that code path entirely.
Also added ansible-core to requirements.txt (installed automatically by
sync-standby-platform.sh's existing `pip install -r requirements.txt`
step; needs a Jenkins rebuild to reach the main pod's image), and synced
ansible/ to the standby the same way backup/ already was — restore_start()
references dr-bootstrap.yml as a fixed absolute path, and that directory
didn't exist there at all before this.
Verified for real end-to-end: triggered a restore via the standby's
actual web UI (target=remote, localhost:2224 tunnel) — Ansible ran env
checks, found k3s already present, reconciled the namespace/PVC/Secret/
manifests (all no-ops against the live cluster), then restore-k8s-apps.sh
restored the data. n8n on the real main server came back healthy
(healthz ok) with all 11 workflows intact in the restored Postgres DB.
get_local_backups_with_status() called _load_local_sidecars() directly,
which is a plain glob.glob('/root/backups/*.tar.gz.meta.json') with no
SSH-fallback awareness at all — unlike get_local_backups() itself (fixed
earlier this session) and _fetch_vm_sidecars() (already had this pattern
from day one). On the standby, /root/backups doesn't exist locally, so
the glob silently returned {} and every local-column entry fell through
to the 'unknown' (gray) default regardless of its real status, while the
VM column worked correctly since _fetch_vm_sidecars() already had the
fallback.
Added _fetch_local_sidecars() mirroring _fetch_vm_sidecars()'s exact
local-dir-first-else-SSH pattern, targeting /root/backups on the main
server via the now-working tunnel instead of the VM's own backup dir.
Verified on the live standby: local column now shows green for backups
with a sidecar and gray only for the legacine ones that genuinely
predate the sidecar feature (documented, intended behavior) — not gray
across the board.
restore-k8s-apps.sh's own header explicitly scoped out fresh-cluster
provisioning ("assumes the target cluster/namespaces/PVCs already
exist"). This adds exactly that missing piece as an Ansible playbook
rather than another bash script — a better fit for "provision a target
into a working state" than for the actual data-restore logic, which stays
in restore-k8s-apps.sh unchanged and is just invoked as the final step.
dr-bootstrap.yml, given an app name + a backup archive:
1. Checks the target's environment (disk/mem/cpu) against k3s minimums
before touching anything
2. Installs k3s only if not already present (checked, not assumed)
3. Creates the namespace, PVC(s), Secret, and Deployments/Services/
Ingress from the backup's captured manifests — sanitized first
(sanitize_k8s_manifest.py strips resourceVersion/uid/status/
volumeName, the same class of bug as the earlier Secret-apply fix,
generalized) since a raw `kubectl get -o yaml` dump can't be
reapplied to a different cluster as-is
4. Hands off to the existing, already-validated restore-k8s-apps.sh for
the actual data population
backup-k8s-apps.sh gets a small but necessary addition: PVC specs were
never captured at all before (only Deployment/Service/ConfigMap/Ingress),
so there was nothing for a fresh-cluster bootstrap to provision the PVC
from. Captured with the same label selector as the existing manifests
capture, so it stays in lockstep automatically.
Verified for real, not just syntax-checked: ran the full playbook against
the standby VM (freshly installed k3s, empty namespace) using a live n8n
backup. Namespace/PVCs/Secret/Deployments provisioned from nothing, data
restored, pod healthy, and the workflow count in the restored Postgres DB
(11) matched the real source server exactly.
Checking disk usage on the VM (95% full, blocking the planned k3s
install) found /backups/cloudproject had 85 legacy Docker-era
myapps-backup-* archives totaling 131GB, dating back to May — the whole
tier had never had any retention at all, only local (TIER 1) and R2
(TIER 3) did. Those 85 are from before the k8s migration entirely (the
Docker volumes they captured don't exist anymore), so deleted all of
them directly on the VM rather than just trimming a few.
Added retention to the VM upload step mirroring the existing local
policy exactly: same MAX_BACKUPS, own myapps-k8s-backup-* glob only —
never touches the other unrelated backup directories that also live on
this VM (imcbackups/mpack/garage, other servers' backups, already
separately reviewed and left alone except a manual one-off trim of
garage's uncapped 31-day history down to 10).
Disk usage: 183G/193G (95%) -> 50G/193G (26%) after cleanup.
Two gaps found testing this live from the standby:
1. restore.html's "Restore on This Server" was checked by default
regardless of RUNNING_ON_MAIN_SERVER — on the standby that option is
nonsensical (no local cluster/kubectl at all) and restore_start()
would have just tried and failed confusingly. Now: that radio is
disabled with an explanatory note when not on the main server,
"External Machine" is checked instead and pre-filled with the tunnel
details (localhost:2224, contabo-key) so restoring from the standby
just targets the real main server without the user having to know
any of that. Added a matching server-side guard in restore_start()
for target=='local' + not RUNNING_ON_MAIN_SERVER (defense in depth —
the UI already prevents it, this catches a direct API call too).
Also fixed refreshSystemMetrics() in platform.js, which would have
overwritten the disabled option's label with the (now-correct, see
previous commit) main-server hostname — looking like a working local
target when it isn't.
2. sync-standby-platform.sh only ever mirrored platform/ — but
restore_start() references /root/CloudOps/backup/restore-k8s-apps.sh
as a fixed absolute path to scp to the remote target, and that
directory never existed on the VM at all. Every restore attempt from
the standby failed immediately with "restore-k8s-apps.sh not found",
regardless of target. Now mirrors /root/CloudOps/backup/ too.
Verified end-to-end for real: triggered a restore of frappe/erpnext from
the standby's actual web UI (target=remote, localhost:2224) — connected
over the tunnel, copied the backup archive + script to the main server,
ran restore-k8s-apps.sh there, scaled the deployment down/up, restored
the DB. Confirmed after: 740 tables in the DB, /api/method/ping
responding on the live pod. Not a dry run — a real restore, actually
initiated from the standby machine.
Two real gaps found after testing the live standby platform:
1. modules/kubernetes.py's in-cluster client (load_incluster_config())
can only ever work inside the real k8s pod. On the standby it silently
returned K8S_AVAILABLE=False with zero fallback — Cluster, Containers-
via-sites, and any cluster/app health data was just empty on that page,
not erroring, so it was easy to miss. This holds even if the standby
later runs as a pod on a *local* k3s on that server (per plan for the
monitoring migration) — in-cluster config there would only ever see
that local, unrelated cluster, never the main one.
Fixed by adding an SSH+kubectl remote path to list_pods(), list_
deployments(), get_ingress_info(), list_deployments_for_namespace(),
and get_pod_metrics(): when not RUNNING_ON_MAIN_SERVER, each runs
`kubectl <verb> -o json` on the main server over the tunnel
(main-to-vm-tunnel.service) instead of using the python client, parses
the raw (camelCase) JSON directly rather than trying to force it
through the client-library's snake_case object model, and feeds the
same _pod_health/_deploy_health/_age_str helpers either way. The
in-cluster path for the real main-server pod is untouched.
2. get_system_info() called psutil.* unconditionally with no
RUNNING_ON_MAIN_SERVER awareness at all — on the standby it was
silently reporting the VM's OWN cpu/mem/disk/hostname mislabeled as
"system info" (hostname field literally showed the VM's hostname).
Added _get_main_server_system_info_remote(): SSHes to the main server
and gathers the same stats via vmstat/free/df/procfs (the main
server's bare host has no psutil — that's only in this app's own
container image — so this avoids depending on it being present
remotely).
Verified end-to-end on the live standby: /api/system now reports the
real main server's hostname/cpu/mem/disk/uptime; /api/cluster reports
real counts (19 pods, 16/16 deployments healthy, 3 degraded, plus live
per-pod metrics via the metrics-server raw API); /api/sites shows real
per-app container/role status. All previously silently empty.
The standby's "SSH to main server for containers/stats/system info"
fallback (RUNNING_ON_MAIN_SERVER=False path) was wired in code across
backups.py/users.py/commands.py/app.py but never actually worked:
outbound SSH from the VM is blocked by its own firewall (deliberate
hardening — ufw DENY OUT on 22/tcp and 2222/tcp, left untouched), and
separately the VM's contabo-key was never added to the main server's
authorized_keys in the first place.
Fixed via a persistent reverse tunnel instead of opening the VM's
firewall: main-to-vm-tunnel.service (systemd, auto-restart) runs on the
main server and keeps `ssh -R 2224:localhost:22` open to the VM, so the
VM can reach the main server's SSH via localhost:2224 without ever
needing outbound access itself. Added the standby's public key to the
main server's authorized_keys (chattr +i-locked — unlocked, appended,
re-locked immediately).
config.py (gitignored, not in this commit — edited live on both servers
directly) gets new MAIN_SERVER_SSH_HOST/PORT ("localhost"/"2224"), kept
deliberately separate from the existing MAIN_SERVER_IP (which stays the
real IP and is still used for on-page display in several templates —
overloading it for the tunnel target would have silently changed what's
shown in the UI). All four call sites switched from
MAIN_SERVER_IP/MAIN_SERVER_PORT to the new pair. Also fixed a pre-existing
bug in commands.py's run_command(): its SSH fallback had no -i key at all,
so it could never have authenticated even with the tunnel in place.
Verified end-to-end on the live standby: run_command(), backups.py's and
users.py's _ssh_main() all confirmed to actually execute on the real main
server (hostname/whoami/backup count all matched), not erroring or
silently no-op'ing.
The VM standby's management-platform was found 2+ months stale during
failover investigation (last deploy: June 10). Root cause: the old
pull-platform-backup.sh + deploy-platform.sh pair on the VM pulled/
redeployed platform-backup-*.tar.gz tarballs of /root/management-platform,
but tarball generation was disabled 2026-08-14 when the source of truth
moved to git-tracked /root/CloudOps — nothing replaced it, so the VM kept
re-deploying its last cached snapshot every hour indefinitely.
New script runs hourly from the main server (cron) instead of pulling from
the VM — pushes /root/CloudOps/platform directly via rsync, plus the live
/root/management-platform/config.py (gitignored, holds real secrets/DB
creds — copied byte-for-byte since it already contains the
RUNNING_ON_MAIN_SERVER auto-detect logic that makes it work correctly
unmodified on both servers). Only restarts the systemd service when rsync
or config actually changed. Old pull/deploy cron entries commented out on
the VM (superseded, not deleted, for history).
Verified end-to-end: ran it live, standby now serves current code (status-
dot markers present in synced templates/app.py), service restarted clean,
config auto-detect correctly logs "running on VM / backup host" and takes
the SSH-fallback code path as designed.
1. Secret apply conflict: secret.yaml is a raw `kubectl get -o yaml` dump
that includes the live resourceVersion/uid/creationTimestamp at capture
time. Applying a stale resourceVersion trips optimistic-concurrency
control ("the object has been modified") on effectively every restore,
since normal cluster activity keeps bumping it. Strip those three
volatile fields before `kubectl apply`.
2. Wrong re-seed source for Nextcloud: the re-seed step added for the PVC
exclusion fix assumed both Mautic and Nextcloud bake their excluded dirs
in at their own PVC mount path (/var/www/html), copying from
pvc_path/$dir. True for Mautic, wrong for Nextcloud — its image keeps
the shipped source tree at /usr/src/nextcloud, completely separate from
/var/www/html (empty in the raw image at that path). Caught in testing
("cp: cannot stat '/var/www/html/apps'") because the reseed_failed gate
from the previous fix did its job and left the app at 0 replicas instead
of booting broken — but still required a live manual fix to bring
Nextcloud back after the failed test. New RECONSTRUCT_SRC table makes
the copy source explicit and independently verified per app instead of
assumed by analogy: confirmed /usr/src/nextcloud/{apps,core,dist,
3rdparty,lib} exist in a bare nextcloud:32 pod with no volume mounted
before relying on it.
Verified end-to-end after both fixes: fresh Nextcloud backup -> restore ->
data/ intact (184M, 89 files, unchanged) -> occ status installed/healthy ->
200 on /status.php, via the actual script run (not just the manual
recovery), confirming the automated path works, not just my live fix.
Mautic (node_modules/, vendor/) and Nextcloud (apps/, core/, dist/,
3rdparty/, lib/) share one PVC mount with their real user data, and were
backing up ~700MB-1GB of app/vendor code that's identical to what ships in
the image, alongside the actual user data. Excluded both via tar --exclude
in the PVC capture step: Mautic backup drops from ~185MB to ~90MB
compressed, Nextcloud from ~379MB to ~107MB.
Verified the "the image lays these down fresh on start" assumption before
relying on it — it does NOT hold for either app as a naive exclude:
- Mautic's entrypoint has no logic at all to reconstruct node_modules/
vendor if missing (confirmed empty by reading /entrypoint.sh).
- Nextcloud's entrypoint only rsyncs from /usr/src/nextcloud when
image_version > installed_version (read from version.php). Since
version.php itself isn't excluded, a restored version.php already
matches the running image's version, so that path never fires.
So restore-k8s-apps.sh now explicitly re-seeds these dirs from the app's
own live image (both images bake them in at /var/www/html independently of
any volume mount) via a loader pod with the PVC mounted at a different
path, after the normal PVC-data restore and before scaling the app back
up. If that re-seed fails, the app is deliberately left at 0 replicas
instead of coming back up broken (missing vendor/autoload.php is a hard
crash, not a degraded state) — found this the hard way in testing when a
90s wait timed out mid image-pull and scale-up proceeded anyway with
vendor/ missing; fixed by gating scale-up on the re-seed outcome and
raising the timeout to 240s with imagePullPolicy: IfNotPresent (the image
is already on-node, pulled off the live Deployment spec).
Verified end-to-end: real backup + restore of Mautic on the live cluster,
confirming size drop, vendor/autoload.php present post-restore, app
serving 200s, and all 433,681 leads intact via the DB restore.
Also found and left unfixed (pre-existing, unrelated): secret.yaml apply
during restore can hit a resourceVersion conflict from kubectl apply
against a captured manifest — the mautic-secrets apply failed on this
restore test with a benign "object has been modified" error since the
secret already existed with correct values; app was unaffected. Separate
bug from this change, flagging for later.
/api/backups returned plain filename strings while pages/backups.html's
server-rendered dots came from get_local_backups_with_status()/
get_vm_backups_with_status(). platform.js's DOMContentLoaded handler always
calls refreshBackupsList() (backups.html has #backup-history-list, which
triggers it unconditionally), which re-fetches /api/backups and rebuilds
#local-backup-list/#vm-backup-list via renderBackupList() — a version with
zero knowledge of status, wiping out the dots seconds after initial paint.
Switch /api/backups to the *_with_status() variants and have
renderBackupList() render the same status-dot markup as the Jinja template,
so the dot is sourced from persisted backup metadata on every render path,
not just the first one.
backup-k8s-apps.sh now tracks step-level failures (manifests, secret,
db_dump, pvc_data) into an ERRORS array during the per-app loop, and
writes a .meta.json sidecar alongside each archive once all 3 storage
tiers are known - apps included, final size, the error list, and
per-tier ok/failed/skipped status. Sidecar is mirrored to VM/R2 the
same best-effort way the .sha256 sidecar already is, and cleaned up
by both local and R2 retention pruning plus manual delete.
modules/backups.py reads these sidecars (never decompresses the
archive) to compute a status per backup at list-render time:
- red: any recorded error, or any tier status starting with "failed"
- yellow: no errors, but size deviates >40% from the rolling average
of the last 5 backups sharing the same app-combination (apps_key)
- green: no errors, size within range (or first backup of its
app-combination - nothing to compare against yet)
- unknown: no sidecar at all (legacy myapps-backup-* archives, or
any k8s backup made before this shipped) - no backfill attempted,
old runs never recorded step-level failures to reconstruct from
/backups route now passes get_local_backups_with_status()/
get_vm_backups_with_status() instead of the plain filename lists
(get_local_backups()/get_vm_backups() themselves are untouched -
/restore and /api/backups still use the plain versions, they don't
need the dot). Template renders a colored dot next to each entry with
a tooltip showing apps/size.
Verified: real n8n backup produces a correct sidecar synced to all 3
tiers; JSON-writer argv parsing and the red/yellow/green/unknown
decision logic each checked against synthetic cases; full Jinja render
checked against real local + VM data pulled from the live pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
Two separate bugs found while testing the Audit/Details UI against a
real k8s-format backup (myapps-k8s-backup-*):
1. Format-hardcoded checks. audit_backup()'s Internal Structure and
Volume Count checks assumed the legacy Docker-era archive layout
(volumes/*.tar.gz, compose-files/) and would fail every k8s-format
backup (per-app dirs with manifests.yaml/db-dump.sql.gz/pvc-data.tar.gz)
even when it's perfectly healthy - same root cause as the
myapps-backup-*/myapps-k8s-backup-* prefix bug fixed earlier, just in
the audit checks instead of the listing/delete glob. cloud_backup.py's
r2_audit_backup() had the identical unpatched filename regex, and
app.py's /api/backups/details route had its own, which outright
400'd any k8s-format filename before even looking at it.
2. A bigger, separate bug this surfaced: get_local_backups(),
get_vm_backups(), and _resolve_archive_path() all branch on
RUNNING_ON_MAIN_SERVER to decide between direct filesystem access and
SSH-to-self - and from inside the management-platform pod that flag's
hostname check can be wrong for the pod's own ephemeral hostname
depending on how it's evaluated, sending these down the SSH branch
using a topology (separate warm-standby-on-the-VM SSH path) that
doesn't apply to a pod that already has /root hostPath-mounted in.
Fixed by trying direct filesystem access first wherever the archive
would already be locally reachable, falling back to the existing SSH
paths otherwise - this preserves the original warm-standby-on-VM
failover design (a real second deployment of this platform on the VM
host, kept reachable via SSH when the main server is down) completely
unchanged; it only adds the fast local path for the case where the
caller already has direct filesystem access.
Verified against both a real k8s-format backup (odoo, single-app) and a
real legacy-format backup on disk - both audit correctly now, with
format-appropriate checks and labels.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
modules/sites.py was still a fully Docker-era hardcoded registry:
container names (odoo-clean-odoo-1, frappe-erpnext, nextcloud-app,
mautic-app, n8n-app) that no longer exist post-migration, checked via
`docker inspect` over SSH, plus hardcoded domain/port fields -
nextcloud and mautic even had domain: None, silently falling back to
dead Docker host-port URLs while their real Ingress domains
(next.cloud.nav.ovh, mautics.nav.ovh) sat unused.
Now sources everything live from the cluster:
- Domain + TLS from the actual Ingress object per app (via the new
get_ingress_info() in modules/kubernetes.py), not a static guess.
Odoo/Nextcloud's Ingress lives in `default` (see prior commit for the
RBAC this needed); n8n/mautic/erpnext's lives in their own namespace.
- App/DB/cache/worker/scheduler status from real Deployment state
(list_deployments_for_namespace(), new in modules/kubernetes.py)
instead of `docker inspect`.
- Health probe now hits the real domain directly from the pod (it has
normal internet egress) instead of SSH-ing back out to the host to
curl a public HTTPS URL, which is what the RUNNING_ON_MAIN_SERVER
branch would have done for a pod whose hostname never matches the
main-server hostname check.
Same API/template contract as before (get_sites_list/get_site_health
return the same field shapes) - templates/pages/sites.html needs no
changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
Odoo's and Nextcloud's Ingress objects live in the default namespace
(historical, from the original Docker->k8s migration) rather than their
own app namespace like n8n/mautic/erpnext. management-platform-sa had
no binding there at all. Granting the full backup ClusterRole in
default would be way too broad (default also hosts unrelated live
services, and that role includes secrets/pod writes) - so this adds a
separate ClusterRole scoped to just ingresses get/list/watch, bound
only in default. Needed for the Application Sites page fix (next
commit) to read real domain/TLS state instead of guessing.
Applied and verified live: can read ingresses in default, still denied
secrets/pod-delete there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
Investigated a resource-safety abort during Odoo restore testing:
available RAM was hovering ~1.3-1.5GB even under normal conditions on
this host (k3s + Jenkins + Docker monitoring/wazuh stack + dev tooling
baseline), not a leak from any single process. 1536MB was tripping on
ordinary single-app backups; 1024MB still leaves real headroom above
what a pg_dump/mysqldump/tar step actually needs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
18 straight clone failures today (early EOF / invalid index-pack) even with
http.postBuffer/http.version tuning applied — Gitea's HTTP endpoint itself is
degraded right now, not something fixable client-side. Sidesteps it entirely:
the docker-agent pod template now hostPath-mounts /root/CloudOps (read-only,
see k3s-cloud-init.groovy) and Checkout just `cp -r`s from it, no network
fetch at all. Build/SonarQube paths revert to ./platform/ (no more repo/
subdirectory, since we're not cloning into one).
Real tradeoff, accepted deliberately: builds now reflect whatever's on the
VPS's disk at build time, not necessarily what's pushed to Gitea. Depends on
/root/CloudOps being kept current via the normal git pull workflow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Git plugin's checkout([...]) step doesn't expose http.postBuffer/
http.version tuning, which is what's reliably fixed every stuck
push/clone today (VPS and both laptops) — meanwhile checkout was failing
~10 times in a row, consistently dying at 84-95% during the shallow clone
(early EOF / invalid index-pack). Bypasses the plugin entirely: a plain
`git clone` with those settings applied, wrapped in its own 8-attempt
bash retry loop, using the existing dd13f593-06b1-4a5a-9477-2df0d186b490
credential (confirmed via credentials.xml to be a UsernamePasswordCredentialsImpl,
compatible with the usernamePassword binding as-is).
Code now lands in a repo/ subdirectory instead of the workspace root, so
every stage referencing a repo-relative path was updated to match:
- Build Docker Image: ./platform/ -> ./repo/platform/
- SonarQube Analysis: -Dsonar.sources=platform -> -Dsonar.sources=repo/platform
Deploy to Kubernetes needed no changes — it only references the already-built
image tag and hardcoded kubectl args, no filesystem paths into the checkout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
api_backup_run and restore_start were still hardcoded to the Docker-era
backup-myapps.sh/restore-myapps.sh, which target Docker volumes that no
longer exist for the 5 apps now running in k3s (n8n, odoo, mautic,
nextcloud, frappe/erpnext) — every backup/restore triggered from the UI
has been silently hollow for these apps since the migration.
- api_backup_run: /root/backup-myapps.sh -> /root/CloudOps/backup/backup-k8s-apps.sh
- restore_start: switched from the old image-relative path (which resolved
to /app/restore-myapps.sh, baked into the Docker image from platform/)
to an absolute /root/CloudOps/backup/restore-k8s-apps.sh path — the new
backup/ folder is a sibling of platform/, not part of the Docker build
context, so it's only reachable via the pod's existing /root hostPath
mount, the same way api_backup_run already reaches its script.
--apps flag shape and the job-log streaming contract are unchanged, so no
frontend/template changes needed for the core flow. modules/backups.py's
three hardcoded myapps-backup-* references (get_local_backups, get_vm_backups
x2, delete_backup's filename regex) now also recognize the new
myapps-k8s-backup-* prefix, so both backup lineages stay visible/manageable
in the UI.
Depends on the previous commit (scoped ServiceAccount + k3s kubectl access
for the pod) to actually function when triggered from the UI.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The pod running app.py had no kubectl binary and only a read-only SA
(management-platform-viewer-sa) — backup-k8s-apps.sh/restore-k8s-apps.sh
could never actually run from inside it. Adds:
- management-platform-backup-role (ClusterRole, bound via RoleBinding only
in n8n/odoo/mautic/nextcloud/erpnext — not cluster-wide): get/list/watch
on pods/deployments/configmaps/ingresses/PVCs, pods/exec create, pods
create+delete (needed for restore's populate-before-scale-up loader pod),
secrets get/list/update/patch, deployments/scale update/patch only (no
write on full Deployment/Service/ConfigMap/Ingress specs).
- management-platform-sa, replacing management-platform-viewer-sa as the
pod's identity — inherits the old viewer-role's read-only bindings too
(retargeted in management-platform-viewer-rbac.yaml) so the existing
cluster-view page keeps working under one SA.
- k3s binary hostPath-mounted read-only into the pod (same pattern as the
Jenkins agent's docker-cli container), so `k3s kubectl` is available
where no separate kubectl binary exists.
Both scripts updated to fall back to k3s kubectl when no kubectl is on
PATH, and to use /proc/meminfo instead of `free` for the resource-safety
check (not present in the pod's minimal image). Also fixes a real issue
found while live-testing this from inside the pod: the pod's pre-existing
/root hostPath mount also exposes the host's own admin ~/.kube/config,
which k3s kubectl was silently preferring over the scoped SA token —
forcing --kubeconfig=/dev/null in both scripts closes that.
Verified end-to-end from inside the actual management-platform pod:
manifests/secret/DB-dump/PVC-data backup for n8n succeeds under the new
SA's scoped permissions, and a kube-system access attempt is correctly
rejected (Forbidden) once the kubeconfig leak is closed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
backup-myapps.sh/restore-myapps.sh only ever knew about Docker volumes,
which no longer exist for n8n/odoo/mautic/nextcloud/frappe now that
they're all running in k3s — nightly backups have been silently hollow
for these apps since the migration. These new scripts capture/restore
each app's k8s manifests, Secret, DB (pg_dump/mysqldump via kubectl exec),
and app-data PVC contents (kubectl exec tar, never raw filesystem access),
reusing the existing local+VM+R2 3-2-1 storage/retention logic unchanged.
Tested standalone: full backup of all 5 apps produces verified non-empty
manifests/secrets/DB dumps/PVC data; restore-in-place tested end-to-end
against n8n (DB row counts and PVC data matched exactly pre/post, app
verified healthy over HTTPS).
Not wired into management-platform yet (app.py stays untouched) — restore
script also doesn't yet handle a fresh/DR cluster where the PVCs and their
manifests don't already exist (same-cluster restore only for now).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New modules/kubernetes.py lists pods and deployments across the 8 app
namespaces via the in-cluster management-platform-viewer-sa (read-only,
no secrets/exec/log), with metrics-server usage as a best-effort extra
that degrades silently when unreachable. New /cluster page and
/api/cluster endpoint, nav entry, and templates render pods/deployments
grouped by namespace using the existing card/badge design system.
Additive only — no existing Docker routes or views touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ServiceAccount management-platform-viewer-sa (in management-platform ns),
ClusterRole management-platform-viewer-role (get/list/watch on pods,
services, deployments.apps only - no secrets, no write verbs), and a
RoleBinding per app namespace (n8n, odoo, mautic, erpnext, nextcloud,
jenkins-agents, jenkins, management-platform). Already applied to the
live cluster and verified via kubectl auth can-i; committing here for
source-of-truth tracking.
Source-of-truth manifest for the management-platform Deployment, with
serviceAccountName: management-platform-viewer-sa added so the pod can
authenticate for the read-only k8s cluster-view feature. Not yet applied
to the cluster.
