Prompted by migrating SonarQube to k8s mid-session, which briefly broke
Jenkins' connectivity to it (server URL config points at a fixed
IP:port that had to move). That's exactly the kind of outage this stage
shouldn't be able to turn into a failed deploy: analysis quality isn't a
deploy gate the way the build/Sonar-unrelated stages are.
Wrapped in catchError(buildResult: 'UNSTABLE', stageResult: 'FAILURE') —
if SonarQube is unreachable for any reason, the stage fails visibly (shows
red in the stage view) and the build is marked UNSTABLE (not hidden), but
the pipeline continues to Deploy instead of stopping there.
Prompted by today's testing: running backup-k8s-apps.sh --apps X repeatedly
for isolated per-app verification left a bunch of single-app archives in
the list, all named identically (myapps-k8s-backup-TIMESTAMP.tar.gz) with
no visible indication of which app(s) each one actually contains — the
data was already there (the .meta.json sidecar's `apps` field, used for
the status-dot rolling average) but only surfaced in a hover tooltip.
Cleaned up today's test archives (local + VM + R2) and ran one fresh full
backup so the list reflects real state.
Two things added, backup/restore logic itself untouched per explicit
instruction:
1. A small visible badge next to each backup's name — "All apps" for a
full bundle, or the specific list (e.g. "odoo, n8n") for a partial one
— in both the Jinja-rendered list and platform.js's refresh path.
2. An optional app-picker on the "Run Backup Now" manual trigger, same
checkbox-grid pattern already used on the Restore page's app selector.
All checked (default) = exactly today's existing behavior, no --apps
flag, everything bundled into one archive. Unchecking some = a
deliberate one-off partial backup (--apps a,b) for e.g. backing up just
odoo before a risky change without waiting on the other 4. The nightly
cron and "backup all together" behavior are completely unaffected —
this only touches the manual/UI-triggered path.
Verified: manual full backup still produces one bundled archive (249M, 5
apps); the apps-badge renders correctly ("All apps") on the live standby;
the --apps arg-construction logic unit-tested directly (partial selection
-> ['--apps', 'a,b'], all-selected -> [] i.e. default bundle).
Previously restore_start()'s 'remote' target did its own manual scp+ssh+
restore-k8s-apps.sh dance, entirely separate from the new DR bootstrap
playbook (ansible/dr-bootstrap.yml) — meaning restoring onto a genuinely
empty server would just fail (no namespace/PVC/Deployments, and
restore-k8s-apps.sh assumes those exist). Every "External Machine" restore
now runs the same playbook instead: it's safe for both cases, not just the
empty-server one — create-namespace-if-missing and `kubectl apply` for the
sanitized PVC/Secret/manifests are no-ops against a target that already
has this app running with a matching spec (apply only reconciles
differences, and a backup's own captured manifests are by definition
identical to what's already live) — so "restore onto an existing cluster"
and "restore onto nothing" are the same command now; the playbook's own
checks-then-acts steps decide how much of it actually needs to do
anything.
Three real bugs found getting this working, not just wiring it up blind:
1. dr-bootstrap.yml's `hosts: dr_target` only matches a named inventory
group — the dynamic single-host inventory app.py builds per-request
(`-i '<ip>,'`) doesn't create one, so nothing matched and the play
silently skipped. Changed to `hosts: all`, which both invocation styles
satisfy.
2. ansible-playbook is a pip console-script installed next to whichever
python is running — the main pod's system python (no venv there) or
this server's own venv/bin on the standby. A bare "ansible-playbook" in
the shelled-out command only resolves on the main pod; the standby's
venv/bin is never on PATH when its python is invoked directly rather
than through `activate`, so it'd fail there. Resolved relative to
sys.executable instead, which is correct in both.
3. Passing connection options via `-e ansible_ssh_common_args='-o
StrictHostKeyChecking=no ...'` hit a real bug in this ansible-core
version's SSH connection plugin: its own internal tty-detection
re-parses that string with a strict argparse and throws "argument -o:
expected one argument" even for one well-formed -o KEY=VALUE — verified
directly on the CLI, not a shell-quoting artifact from this code.
Switched to ANSIBLE_HOST_KEY_CHECKING=False / ANSIBLE_TIMEOUT=15 env
vars, ansible's own dedicated mechanism for the same effect, which
bypasses that code path entirely.
Also added ansible-core to requirements.txt (installed automatically by
sync-standby-platform.sh's existing `pip install -r requirements.txt`
step; needs a Jenkins rebuild to reach the main pod's image), and synced
ansible/ to the standby the same way backup/ already was — restore_start()
references dr-bootstrap.yml as a fixed absolute path, and that directory
didn't exist there at all before this.
Verified for real end-to-end: triggered a restore via the standby's
actual web UI (target=remote, localhost:2224 tunnel) — Ansible ran env
checks, found k3s already present, reconciled the namespace/PVC/Secret/
manifests (all no-ops against the live cluster), then restore-k8s-apps.sh
restored the data. n8n on the real main server came back healthy
(healthz ok) with all 11 workflows intact in the restored Postgres DB.
get_local_backups_with_status() called _load_local_sidecars() directly,
which is a plain glob.glob('/root/backups/*.tar.gz.meta.json') with no
SSH-fallback awareness at all — unlike get_local_backups() itself (fixed
earlier this session) and _fetch_vm_sidecars() (already had this pattern
from day one). On the standby, /root/backups doesn't exist locally, so
the glob silently returned {} and every local-column entry fell through
to the 'unknown' (gray) default regardless of its real status, while the
VM column worked correctly since _fetch_vm_sidecars() already had the
fallback.
Added _fetch_local_sidecars() mirroring _fetch_vm_sidecars()'s exact
local-dir-first-else-SSH pattern, targeting /root/backups on the main
server via the now-working tunnel instead of the VM's own backup dir.
Verified on the live standby: local column now shows green for backups
with a sidecar and gray only for the legacine ones that genuinely
predate the sidecar feature (documented, intended behavior) — not gray
across the board.
Two gaps found testing this live from the standby:
1. restore.html's "Restore on This Server" was checked by default
regardless of RUNNING_ON_MAIN_SERVER — on the standby that option is
nonsensical (no local cluster/kubectl at all) and restore_start()
would have just tried and failed confusingly. Now: that radio is
disabled with an explanatory note when not on the main server,
"External Machine" is checked instead and pre-filled with the tunnel
details (localhost:2224, contabo-key) so restoring from the standby
just targets the real main server without the user having to know
any of that. Added a matching server-side guard in restore_start()
for target=='local' + not RUNNING_ON_MAIN_SERVER (defense in depth —
the UI already prevents it, this catches a direct API call too).
Also fixed refreshSystemMetrics() in platform.js, which would have
overwritten the disabled option's label with the (now-correct, see
previous commit) main-server hostname — looking like a working local
target when it isn't.
2. sync-standby-platform.sh only ever mirrored platform/ — but
restore_start() references /root/CloudOps/backup/restore-k8s-apps.sh
as a fixed absolute path to scp to the remote target, and that
directory never existed on the VM at all. Every restore attempt from
the standby failed immediately with "restore-k8s-apps.sh not found",
regardless of target. Now mirrors /root/CloudOps/backup/ too.
Verified end-to-end for real: triggered a restore of frappe/erpnext from
the standby's actual web UI (target=remote, localhost:2224) — connected
over the tunnel, copied the backup archive + script to the main server,
ran restore-k8s-apps.sh there, scaled the deployment down/up, restored
the DB. Confirmed after: 740 tables in the DB, /api/method/ping
responding on the live pod. Not a dry run — a real restore, actually
initiated from the standby machine.
Two real gaps found after testing the live standby platform:
1. modules/kubernetes.py's in-cluster client (load_incluster_config())
can only ever work inside the real k8s pod. On the standby it silently
returned K8S_AVAILABLE=False with zero fallback — Cluster, Containers-
via-sites, and any cluster/app health data was just empty on that page,
not erroring, so it was easy to miss. This holds even if the standby
later runs as a pod on a *local* k3s on that server (per plan for the
monitoring migration) — in-cluster config there would only ever see
that local, unrelated cluster, never the main one.
Fixed by adding an SSH+kubectl remote path to list_pods(), list_
deployments(), get_ingress_info(), list_deployments_for_namespace(),
and get_pod_metrics(): when not RUNNING_ON_MAIN_SERVER, each runs
`kubectl <verb> -o json` on the main server over the tunnel
(main-to-vm-tunnel.service) instead of using the python client, parses
the raw (camelCase) JSON directly rather than trying to force it
through the client-library's snake_case object model, and feeds the
same _pod_health/_deploy_health/_age_str helpers either way. The
in-cluster path for the real main-server pod is untouched.
2. get_system_info() called psutil.* unconditionally with no
RUNNING_ON_MAIN_SERVER awareness at all — on the standby it was
silently reporting the VM's OWN cpu/mem/disk/hostname mislabeled as
"system info" (hostname field literally showed the VM's hostname).
Added _get_main_server_system_info_remote(): SSHes to the main server
and gathers the same stats via vmstat/free/df/procfs (the main
server's bare host has no psutil — that's only in this app's own
container image — so this avoids depending on it being present
remotely).
Verified end-to-end on the live standby: /api/system now reports the
real main server's hostname/cpu/mem/disk/uptime; /api/cluster reports
real counts (19 pods, 16/16 deployments healthy, 3 degraded, plus live
per-pod metrics via the metrics-server raw API); /api/sites shows real
per-app container/role status. All previously silently empty.
The standby's "SSH to main server for containers/stats/system info"
fallback (RUNNING_ON_MAIN_SERVER=False path) was wired in code across
backups.py/users.py/commands.py/app.py but never actually worked:
outbound SSH from the VM is blocked by its own firewall (deliberate
hardening — ufw DENY OUT on 22/tcp and 2222/tcp, left untouched), and
separately the VM's contabo-key was never added to the main server's
authorized_keys in the first place.
Fixed via a persistent reverse tunnel instead of opening the VM's
firewall: main-to-vm-tunnel.service (systemd, auto-restart) runs on the
main server and keeps `ssh -R 2224:localhost:22` open to the VM, so the
VM can reach the main server's SSH via localhost:2224 without ever
needing outbound access itself. Added the standby's public key to the
main server's authorized_keys (chattr +i-locked — unlocked, appended,
re-locked immediately).
config.py (gitignored, not in this commit — edited live on both servers
directly) gets new MAIN_SERVER_SSH_HOST/PORT ("localhost"/"2224"), kept
deliberately separate from the existing MAIN_SERVER_IP (which stays the
real IP and is still used for on-page display in several templates —
overloading it for the tunnel target would have silently changed what's
shown in the UI). All four call sites switched from
MAIN_SERVER_IP/MAIN_SERVER_PORT to the new pair. Also fixed a pre-existing
bug in commands.py's run_command(): its SSH fallback had no -i key at all,
so it could never have authenticated even with the tunnel in place.
Verified end-to-end on the live standby: run_command(), backups.py's and
users.py's _ssh_main() all confirmed to actually execute on the real main
server (hostname/whoami/backup count all matched), not erroring or
silently no-op'ing.
/api/backups returned plain filename strings while pages/backups.html's
server-rendered dots came from get_local_backups_with_status()/
get_vm_backups_with_status(). platform.js's DOMContentLoaded handler always
calls refreshBackupsList() (backups.html has #backup-history-list, which
triggers it unconditionally), which re-fetches /api/backups and rebuilds
#local-backup-list/#vm-backup-list via renderBackupList() — a version with
zero knowledge of status, wiping out the dots seconds after initial paint.
Switch /api/backups to the *_with_status() variants and have
renderBackupList() render the same status-dot markup as the Jinja template,
so the dot is sourced from persisted backup metadata on every render path,
not just the first one.
backup-k8s-apps.sh now tracks step-level failures (manifests, secret,
db_dump, pvc_data) into an ERRORS array during the per-app loop, and
writes a .meta.json sidecar alongside each archive once all 3 storage
tiers are known - apps included, final size, the error list, and
per-tier ok/failed/skipped status. Sidecar is mirrored to VM/R2 the
same best-effort way the .sha256 sidecar already is, and cleaned up
by both local and R2 retention pruning plus manual delete.
modules/backups.py reads these sidecars (never decompresses the
archive) to compute a status per backup at list-render time:
- red: any recorded error, or any tier status starting with "failed"
- yellow: no errors, but size deviates >40% from the rolling average
of the last 5 backups sharing the same app-combination (apps_key)
- green: no errors, size within range (or first backup of its
app-combination - nothing to compare against yet)
- unknown: no sidecar at all (legacy myapps-backup-* archives, or
any k8s backup made before this shipped) - no backfill attempted,
old runs never recorded step-level failures to reconstruct from
/backups route now passes get_local_backups_with_status()/
get_vm_backups_with_status() instead of the plain filename lists
(get_local_backups()/get_vm_backups() themselves are untouched -
/restore and /api/backups still use the plain versions, they don't
need the dot). Template renders a colored dot next to each entry with
a tooltip showing apps/size.
Verified: real n8n backup produces a correct sidecar synced to all 3
tiers; JSON-writer argv parsing and the red/yellow/green/unknown
decision logic each checked against synthetic cases; full Jinja render
checked against real local + VM data pulled from the live pod.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
Two separate bugs found while testing the Audit/Details UI against a
real k8s-format backup (myapps-k8s-backup-*):
1. Format-hardcoded checks. audit_backup()'s Internal Structure and
Volume Count checks assumed the legacy Docker-era archive layout
(volumes/*.tar.gz, compose-files/) and would fail every k8s-format
backup (per-app dirs with manifests.yaml/db-dump.sql.gz/pvc-data.tar.gz)
even when it's perfectly healthy - same root cause as the
myapps-backup-*/myapps-k8s-backup-* prefix bug fixed earlier, just in
the audit checks instead of the listing/delete glob. cloud_backup.py's
r2_audit_backup() had the identical unpatched filename regex, and
app.py's /api/backups/details route had its own, which outright
400'd any k8s-format filename before even looking at it.
2. A bigger, separate bug this surfaced: get_local_backups(),
get_vm_backups(), and _resolve_archive_path() all branch on
RUNNING_ON_MAIN_SERVER to decide between direct filesystem access and
SSH-to-self - and from inside the management-platform pod that flag's
hostname check can be wrong for the pod's own ephemeral hostname
depending on how it's evaluated, sending these down the SSH branch
using a topology (separate warm-standby-on-the-VM SSH path) that
doesn't apply to a pod that already has /root hostPath-mounted in.
Fixed by trying direct filesystem access first wherever the archive
would already be locally reachable, falling back to the existing SSH
paths otherwise - this preserves the original warm-standby-on-VM
failover design (a real second deployment of this platform on the VM
host, kept reachable via SSH when the main server is down) completely
unchanged; it only adds the fast local path for the case where the
caller already has direct filesystem access.
Verified against both a real k8s-format backup (odoo, single-app) and a
real legacy-format backup on disk - both audit correctly now, with
format-appropriate checks and labels.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
modules/sites.py was still a fully Docker-era hardcoded registry:
container names (odoo-clean-odoo-1, frappe-erpnext, nextcloud-app,
mautic-app, n8n-app) that no longer exist post-migration, checked via
`docker inspect` over SSH, plus hardcoded domain/port fields -
nextcloud and mautic even had domain: None, silently falling back to
dead Docker host-port URLs while their real Ingress domains
(next.cloud.nav.ovh, mautics.nav.ovh) sat unused.
Now sources everything live from the cluster:
- Domain + TLS from the actual Ingress object per app (via the new
get_ingress_info() in modules/kubernetes.py), not a static guess.
Odoo/Nextcloud's Ingress lives in `default` (see prior commit for the
RBAC this needed); n8n/mautic/erpnext's lives in their own namespace.
- App/DB/cache/worker/scheduler status from real Deployment state
(list_deployments_for_namespace(), new in modules/kubernetes.py)
instead of `docker inspect`.
- Health probe now hits the real domain directly from the pod (it has
normal internet egress) instead of SSH-ing back out to the host to
curl a public HTTPS URL, which is what the RUNNING_ON_MAIN_SERVER
branch would have done for a pod whose hostname never matches the
main-server hostname check.
Same API/template contract as before (get_sites_list/get_site_health
return the same field shapes) - templates/pages/sites.html needs no
changes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017avLHFqkiti3g62Anq9sVA
18 straight clone failures today (early EOF / invalid index-pack) even with
http.postBuffer/http.version tuning applied — Gitea's HTTP endpoint itself is
degraded right now, not something fixable client-side. Sidesteps it entirely:
the docker-agent pod template now hostPath-mounts /root/CloudOps (read-only,
see k3s-cloud-init.groovy) and Checkout just `cp -r`s from it, no network
fetch at all. Build/SonarQube paths revert to ./platform/ (no more repo/
subdirectory, since we're not cloning into one).
Real tradeoff, accepted deliberately: builds now reflect whatever's on the
VPS's disk at build time, not necessarily what's pushed to Gitea. Depends on
/root/CloudOps being kept current via the normal git pull workflow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Git plugin's checkout([...]) step doesn't expose http.postBuffer/
http.version tuning, which is what's reliably fixed every stuck
push/clone today (VPS and both laptops) — meanwhile checkout was failing
~10 times in a row, consistently dying at 84-95% during the shallow clone
(early EOF / invalid index-pack). Bypasses the plugin entirely: a plain
`git clone` with those settings applied, wrapped in its own 8-attempt
bash retry loop, using the existing dd13f593-06b1-4a5a-9477-2df0d186b490
credential (confirmed via credentials.xml to be a UsernamePasswordCredentialsImpl,
compatible with the usernamePassword binding as-is).
Code now lands in a repo/ subdirectory instead of the workspace root, so
every stage referencing a repo-relative path was updated to match:
- Build Docker Image: ./platform/ -> ./repo/platform/
- SonarQube Analysis: -Dsonar.sources=platform -> -Dsonar.sources=repo/platform
Deploy to Kubernetes needed no changes — it only references the already-built
image tag and hardcoded kubectl args, no filesystem paths into the checkout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
api_backup_run and restore_start were still hardcoded to the Docker-era
backup-myapps.sh/restore-myapps.sh, which target Docker volumes that no
longer exist for the 5 apps now running in k3s (n8n, odoo, mautic,
nextcloud, frappe/erpnext) — every backup/restore triggered from the UI
has been silently hollow for these apps since the migration.
- api_backup_run: /root/backup-myapps.sh -> /root/CloudOps/backup/backup-k8s-apps.sh
- restore_start: switched from the old image-relative path (which resolved
to /app/restore-myapps.sh, baked into the Docker image from platform/)
to an absolute /root/CloudOps/backup/restore-k8s-apps.sh path — the new
backup/ folder is a sibling of platform/, not part of the Docker build
context, so it's only reachable via the pod's existing /root hostPath
mount, the same way api_backup_run already reaches its script.
--apps flag shape and the job-log streaming contract are unchanged, so no
frontend/template changes needed for the core flow. modules/backups.py's
three hardcoded myapps-backup-* references (get_local_backups, get_vm_backups
x2, delete_backup's filename regex) now also recognize the new
myapps-k8s-backup-* prefix, so both backup lineages stay visible/manageable
in the UI.
Depends on the previous commit (scoped ServiceAccount + k3s kubectl access
for the pod) to actually function when triggered from the UI.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New modules/kubernetes.py lists pods and deployments across the 8 app
namespaces via the in-cluster management-platform-viewer-sa (read-only,
no secrets/exec/log), with metrics-server usage as a best-effort extra
that degrades silently when unreachable. New /cluster page and
/api/cluster endpoint, nav entry, and templates render pods/deployments
grouped by namespace using the existing card/badge design system.
Additive only — no existing Docker routes or views touched.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
