Previously restore_start()'s 'remote' target did its own manual scp+ssh+
restore-k8s-apps.sh dance, entirely separate from the new DR bootstrap
playbook (ansible/dr-bootstrap.yml) — meaning restoring onto a genuinely
empty server would just fail (no namespace/PVC/Deployments, and
restore-k8s-apps.sh assumes those exist). Every "External Machine" restore
now runs the same playbook instead: it's safe for both cases, not just the
empty-server one — create-namespace-if-missing and `kubectl apply` for the
sanitized PVC/Secret/manifests are no-ops against a target that already
has this app running with a matching spec (apply only reconciles
differences, and a backup's own captured manifests are by definition
identical to what's already live) — so "restore onto an existing cluster"
and "restore onto nothing" are the same command now; the playbook's own
checks-then-acts steps decide how much of it actually needs to do
anything.
Three real bugs found getting this working, not just wiring it up blind:
1. dr-bootstrap.yml's `hosts: dr_target` only matches a named inventory
group — the dynamic single-host inventory app.py builds per-request
(`-i '<ip>,'`) doesn't create one, so nothing matched and the play
silently skipped. Changed to `hosts: all`, which both invocation styles
satisfy.
2. ansible-playbook is a pip console-script installed next to whichever
python is running — the main pod's system python (no venv there) or
this server's own venv/bin on the standby. A bare "ansible-playbook" in
the shelled-out command only resolves on the main pod; the standby's
venv/bin is never on PATH when its python is invoked directly rather
than through `activate`, so it'd fail there. Resolved relative to
sys.executable instead, which is correct in both.
3. Passing connection options via `-e ansible_ssh_common_args='-o
StrictHostKeyChecking=no ...'` hit a real bug in this ansible-core
version's SSH connection plugin: its own internal tty-detection
re-parses that string with a strict argparse and throws "argument -o:
expected one argument" even for one well-formed -o KEY=VALUE — verified
directly on the CLI, not a shell-quoting artifact from this code.
Switched to ANSIBLE_HOST_KEY_CHECKING=False / ANSIBLE_TIMEOUT=15 env
vars, ansible's own dedicated mechanism for the same effect, which
bypasses that code path entirely.
Also added ansible-core to requirements.txt (installed automatically by
sync-standby-platform.sh's existing `pip install -r requirements.txt`
step; needs a Jenkins rebuild to reach the main pod's image), and synced
ansible/ to the standby the same way backup/ already was — restore_start()
references dr-bootstrap.yml as a fixed absolute path, and that directory
didn't exist there at all before this.
Verified for real end-to-end: triggered a restore via the standby's
actual web UI (target=remote, localhost:2224 tunnel) — Ansible ran env
checks, found k3s already present, reconciled the namespace/PVC/Secret/
manifests (all no-ops against the live cluster), then restore-k8s-apps.sh
restored the data. n8n on the real main server came back healthy
(healthz ok) with all 11 workflows intact in the restored Postgres DB.
restore-k8s-apps.sh's own header explicitly scoped out fresh-cluster
provisioning ("assumes the target cluster/namespaces/PVCs already
exist"). This adds exactly that missing piece as an Ansible playbook
rather than another bash script — a better fit for "provision a target
into a working state" than for the actual data-restore logic, which stays
in restore-k8s-apps.sh unchanged and is just invoked as the final step.
dr-bootstrap.yml, given an app name + a backup archive:
1. Checks the target's environment (disk/mem/cpu) against k3s minimums
before touching anything
2. Installs k3s only if not already present (checked, not assumed)
3. Creates the namespace, PVC(s), Secret, and Deployments/Services/
Ingress from the backup's captured manifests — sanitized first
(sanitize_k8s_manifest.py strips resourceVersion/uid/status/
volumeName, the same class of bug as the earlier Secret-apply fix,
generalized) since a raw `kubectl get -o yaml` dump can't be
reapplied to a different cluster as-is
4. Hands off to the existing, already-validated restore-k8s-apps.sh for
the actual data population
backup-k8s-apps.sh gets a small but necessary addition: PVC specs were
never captured at all before (only Deployment/Service/ConfigMap/Ingress),
so there was nothing for a fresh-cluster bootstrap to provision the PVC
from. Captured with the same label selector as the existing manifests
capture, so it stays in lockstep automatically.
Verified for real, not just syntax-checked: ran the full playbook against
the standby VM (freshly installed k3s, empty namespace) using a live n8n
backup. Namespace/PVCs/Secret/Deployments provisioned from nothing, data
restored, pod healthy, and the workflow count in the restored Postgres DB
(11) matched the real source server exactly.
