Previously restore_start()'s 'remote' target did its own manual scp+ssh+
restore-k8s-apps.sh dance, entirely separate from the new DR bootstrap
playbook (ansible/dr-bootstrap.yml) — meaning restoring onto a genuinely
empty server would just fail (no namespace/PVC/Deployments, and
restore-k8s-apps.sh assumes those exist). Every "External Machine" restore
now runs the same playbook instead: it's safe for both cases, not just the
empty-server one — create-namespace-if-missing and `kubectl apply` for the
sanitized PVC/Secret/manifests are no-ops against a target that already
has this app running with a matching spec (apply only reconciles
differences, and a backup's own captured manifests are by definition
identical to what's already live) — so "restore onto an existing cluster"
and "restore onto nothing" are the same command now; the playbook's own
checks-then-acts steps decide how much of it actually needs to do
anything.
Three real bugs found getting this working, not just wiring it up blind:
1. dr-bootstrap.yml's `hosts: dr_target` only matches a named inventory
group — the dynamic single-host inventory app.py builds per-request
(`-i '<ip>,'`) doesn't create one, so nothing matched and the play
silently skipped. Changed to `hosts: all`, which both invocation styles
satisfy.
2. ansible-playbook is a pip console-script installed next to whichever
python is running — the main pod's system python (no venv there) or
this server's own venv/bin on the standby. A bare "ansible-playbook" in
the shelled-out command only resolves on the main pod; the standby's
venv/bin is never on PATH when its python is invoked directly rather
than through `activate`, so it'd fail there. Resolved relative to
sys.executable instead, which is correct in both.
3. Passing connection options via `-e ansible_ssh_common_args='-o
StrictHostKeyChecking=no ...'` hit a real bug in this ansible-core
version's SSH connection plugin: its own internal tty-detection
re-parses that string with a strict argparse and throws "argument -o:
expected one argument" even for one well-formed -o KEY=VALUE — verified
directly on the CLI, not a shell-quoting artifact from this code.
Switched to ANSIBLE_HOST_KEY_CHECKING=False / ANSIBLE_TIMEOUT=15 env
vars, ansible's own dedicated mechanism for the same effect, which
bypasses that code path entirely.
Also added ansible-core to requirements.txt (installed automatically by
sync-standby-platform.sh's existing `pip install -r requirements.txt`
step; needs a Jenkins rebuild to reach the main pod's image), and synced
ansible/ to the standby the same way backup/ already was — restore_start()
references dr-bootstrap.yml as a fixed absolute path, and that directory
didn't exist there at all before this.
Verified for real end-to-end: triggered a restore via the standby's
actual web UI (target=remote, localhost:2224 tunnel) — Ansible ran env
checks, found k3s already present, reconciled the namespace/PVC/Secret/
manifests (all no-ops against the live cluster), then restore-k8s-apps.sh
restored the data. n8n on the real main server came back healthy
(healthz ok) with all 11 workflows intact in the restored Postgres DB.
Two gaps found testing this live from the standby:
1. restore.html's "Restore on This Server" was checked by default
regardless of RUNNING_ON_MAIN_SERVER — on the standby that option is
nonsensical (no local cluster/kubectl at all) and restore_start()
would have just tried and failed confusingly. Now: that radio is
disabled with an explanatory note when not on the main server,
"External Machine" is checked instead and pre-filled with the tunnel
details (localhost:2224, contabo-key) so restoring from the standby
just targets the real main server without the user having to know
any of that. Added a matching server-side guard in restore_start()
for target=='local' + not RUNNING_ON_MAIN_SERVER (defense in depth —
the UI already prevents it, this catches a direct API call too).
Also fixed refreshSystemMetrics() in platform.js, which would have
overwritten the disabled option's label with the (now-correct, see
previous commit) main-server hostname — looking like a working local
target when it isn't.
2. sync-standby-platform.sh only ever mirrored platform/ — but
restore_start() references /root/CloudOps/backup/restore-k8s-apps.sh
as a fixed absolute path to scp to the remote target, and that
directory never existed on the VM at all. Every restore attempt from
the standby failed immediately with "restore-k8s-apps.sh not found",
regardless of target. Now mirrors /root/CloudOps/backup/ too.
Verified end-to-end for real: triggered a restore of frappe/erpnext from
the standby's actual web UI (target=remote, localhost:2224) — connected
over the tunnel, copied the backup archive + script to the main server,
ran restore-k8s-apps.sh there, scaled the deployment down/up, restored
the DB. Confirmed after: 740 tables in the DB, /api/method/ping
responding on the live pod. Not a dry run — a real restore, actually
initiated from the standby machine.
The VM standby's management-platform was found 2+ months stale during
failover investigation (last deploy: June 10). Root cause: the old
pull-platform-backup.sh + deploy-platform.sh pair on the VM pulled/
redeployed platform-backup-*.tar.gz tarballs of /root/management-platform,
but tarball generation was disabled 2026-08-14 when the source of truth
moved to git-tracked /root/CloudOps — nothing replaced it, so the VM kept
re-deploying its last cached snapshot every hour indefinitely.
New script runs hourly from the main server (cron) instead of pulling from
the VM — pushes /root/CloudOps/platform directly via rsync, plus the live
/root/management-platform/config.py (gitignored, holds real secrets/DB
creds — copied byte-for-byte since it already contains the
RUNNING_ON_MAIN_SERVER auto-detect logic that makes it work correctly
unmodified on both servers). Only restarts the systemd service when rsync
or config actually changed. Old pull/deploy cron entries commented out on
the VM (superseded, not deleted, for history).
Verified end-to-end: ran it live, standby now serves current code (status-
dot markers present in synced templates/app.py), service restarted clean,
config auto-detect correctly logs "running on VM / backup host" and takes
the SSH-fallback code path as designed.
