Make restore-from-standby actually work, not just look wired
Two gaps found testing this live from the standby: 1. restore.html's "Restore on This Server" was checked by default regardless of RUNNING_ON_MAIN_SERVER — on the standby that option is nonsensical (no local cluster/kubectl at all) and restore_start() would have just tried and failed confusingly. Now: that radio is disabled with an explanatory note when not on the main server, "External Machine" is checked instead and pre-filled with the tunnel details (localhost:2224, contabo-key) so restoring from the standby just targets the real main server without the user having to know any of that. Added a matching server-side guard in restore_start() for target=='local' + not RUNNING_ON_MAIN_SERVER (defense in depth — the UI already prevents it, this catches a direct API call too). Also fixed refreshSystemMetrics() in platform.js, which would have overwritten the disabled option's label with the (now-correct, see previous commit) main-server hostname — looking like a working local target when it isn't. 2. sync-standby-platform.sh only ever mirrored platform/ — but restore_start() references /root/CloudOps/backup/restore-k8s-apps.sh as a fixed absolute path to scp to the remote target, and that directory never existed on the VM at all. Every restore attempt from the standby failed immediately with "restore-k8s-apps.sh not found", regardless of target. Now mirrors /root/CloudOps/backup/ too. Verified end-to-end for real: triggered a restore of frappe/erpnext from the standby's actual web UI (target=remote, localhost:2224) — connected over the tunnel, copied the backup archive + script to the main server, ran restore-k8s-apps.sh there, scaled the deployment down/up, restored the DB. Confirmed after: 740 tables in the DB, /api/method/ping responding on the live pod. Not a dry run — a real restore, actually initiated from the standby machine.
This commit is contained in:
@@ -193,6 +193,14 @@ def restore_page():
|
||||
vm_backups=get_vm_backups(),
|
||||
restore_prefill=prefill,
|
||||
main_server=MAIN_SERVER_IP,
|
||||
running_on_main_server=RUNNING_ON_MAIN_SERVER,
|
||||
# Only meaningful when running as the standby: "restore on this
|
||||
# server" is nonsensical there (no local cluster/kubectl at all),
|
||||
# so pre-fill the remote-target fields with the tunnel back to the
|
||||
# real main server instead of leaving the user to find these values.
|
||||
tunnel_remote_ip=MAIN_SERVER_SSH_HOST,
|
||||
tunnel_remote_port=MAIN_SERVER_SSH_PORT,
|
||||
tunnel_ssh_key=MAIN_SERVER_KEY,
|
||||
active_page='restore',
|
||||
page_title='Restore',
|
||||
page_subtitle='backup → target'
|
||||
@@ -836,6 +844,12 @@ def restore_start():
|
||||
return jsonify({'error': f'restore-k8s-apps.sh not found at {restore_script_local}'}), 500
|
||||
|
||||
if target == 'local':
|
||||
if not RUNNING_ON_MAIN_SERVER:
|
||||
return jsonify({
|
||||
'error': 'This is the standby — there is no local cluster here to restore '
|
||||
'into. Use "External Machine" to restore onto the real main server '
|
||||
'over the tunnel instead.'
|
||||
}), 400
|
||||
hostname = os.uname().nodename
|
||||
os.makedirs('/root/tmp', exist_ok=True)
|
||||
session_dir = f"/root/tmp/restore-session-{uuid.uuid4().hex[:8]}"
|
||||
|
||||
@@ -170,7 +170,10 @@ async function refreshSystemMetrics() {
|
||||
if (el('g-disk')) el('g-disk').style.width = Math.min(parseFloat(d.disk_pct) || 0, 100) + '%';
|
||||
setText('uptime-chip', d.uptime);
|
||||
if (el('settings-uptime')) el('settings-uptime').value = d.uptime;
|
||||
if (d.hostname && el('this-server-desc')) el('this-server-desc').textContent = d.hostname;
|
||||
const localTargetRadio = document.querySelector('input[name="restore_target"][value="local"]');
|
||||
if (d.hostname && el('this-server-desc') && !(localTargetRadio && localTargetRadio.disabled)) {
|
||||
el('this-server-desc').textContent = d.hostname;
|
||||
}
|
||||
} catch (_) {}
|
||||
}
|
||||
|
||||
|
||||
@@ -63,26 +63,37 @@
|
||||
|
||||
<div class="form-section">
|
||||
<div class="form-section-title">STEP 2 — SELECT RESTORE TARGET</div>
|
||||
{% if not running_on_main_server %}
|
||||
<div style="margin-bottom:12px;padding:10px 14px;background:rgba(245,158,11,0.08);
|
||||
border:1px solid rgba(245,158,11,0.25);border-radius:8px;font-size:12px;color:var(--yellow);">
|
||||
<i class="fas fa-triangle-exclamation"></i>
|
||||
Running as the standby — there's no local cluster here to restore into.
|
||||
"External Machine" is pre-filled to reach the real main server over the standby's
|
||||
SSH tunnel; only change these if you actually mean a different target.
|
||||
</div>
|
||||
{% endif %}
|
||||
<div class="radio-group">
|
||||
<label class="radio-card">
|
||||
<input type="radio" name="restore_target" value="local" checked onchange="toggleRemoteFields()">
|
||||
<div class="radio-body"><span class="radio-icon">🎯</span><div><div class="radio-label">Restore on This Server</div><div class="radio-desc" id="this-server-desc">Loading hostname…</div></div></div>
|
||||
<label class="radio-card{% if not running_on_main_server %} disabled{% endif %}">
|
||||
<input type="radio" name="restore_target" value="local"
|
||||
{% if running_on_main_server %}checked{% else %}disabled{% endif %}
|
||||
onchange="toggleRemoteFields()">
|
||||
<div class="radio-body"><span class="radio-icon">🎯</span><div><div class="radio-label">Restore on This Server</div><div class="radio-desc" id="this-server-desc">{% if running_on_main_server %}Loading hostname…{% else %}Not available on the standby{% endif %}</div></div></div>
|
||||
</label>
|
||||
<label class="radio-card">
|
||||
<input type="radio" name="restore_target" value="remote" onchange="toggleRemoteFields()">
|
||||
<input type="radio" name="restore_target" value="remote" {% if not running_on_main_server %}checked{% endif %} onchange="toggleRemoteFields()">
|
||||
<div class="radio-body"><span class="radio-icon">📡</span><div><div class="radio-label">External Machine</div><div class="radio-desc">via SSH — any IP</div></div></div>
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div id="remote-fields" style="display:none; margin-top:16px; max-width:560px;">
|
||||
<div id="remote-fields" style="display:{% if running_on_main_server %}none{% else %}block{% endif %}; margin-top:16px; max-width:560px;">
|
||||
<div class="form-row">
|
||||
<div class="form-group">
|
||||
<label class="form-label">TARGET IP</label>
|
||||
<input type="text" id="remote-ip" class="form-input" placeholder="192.168.x.x or IP">
|
||||
<input type="text" id="remote-ip" class="form-input" placeholder="192.168.x.x or IP" value="{{ tunnel_remote_ip if not running_on_main_server else '' }}">
|
||||
</div>
|
||||
<div class="form-group" style="max-width:100px;">
|
||||
<label class="form-label">SSH PORT</label>
|
||||
<input type="text" id="remote-port" class="form-input" value="22">
|
||||
<input type="text" id="remote-port" class="form-input" value="{{ tunnel_remote_port if not running_on_main_server else '22' }}">
|
||||
</div>
|
||||
<div class="form-group" style="max-width:120px;">
|
||||
<label class="form-label">SSH USER</label>
|
||||
|
||||
@@ -35,6 +35,15 @@ VM_DEPLOY_DIR="/root/management-platform"
|
||||
|
||||
LOCAL_SRC="/root/CloudOps/platform/"
|
||||
LIVE_CONFIG="/root/management-platform/config.py"
|
||||
|
||||
# restore_start()/api_backup_run() in app.py reference this as a fixed
|
||||
# absolute path (/root/CloudOps/backup/restore-k8s-apps.sh) — it's a
|
||||
# sibling of platform/, never baked into the image, reached via the same
|
||||
# rw hostPath mount as everything else. Restoring FROM the standby (target
|
||||
# = remote, pointed at the main server over the tunnel) needs a local copy
|
||||
# here to scp across, so mirror it too, not just platform/.
|
||||
LOCAL_BACKUP_SRC="/root/CloudOps/backup/"
|
||||
VM_BACKUP_DIR="/root/CloudOps/backup"
|
||||
LOG="/root/CloudOps/scripts/sync-standby-platform.log"
|
||||
|
||||
SSH_OPTS="-i $VM_KEY -p $VM_PORT -o StrictHostKeyChecking=no -o ConnectTimeout=15"
|
||||
@@ -63,6 +72,16 @@ if [ $RSYNC_RC -ne 0 ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ssh $SSH_OPTS "$VM_USER@$VM_HOST" "mkdir -p $VM_BACKUP_DIR"
|
||||
BACKUP_CHANGES=$(rsync -a --delete -e "ssh $SSH_OPTS" \
|
||||
--itemize-changes \
|
||||
"$LOCAL_BACKUP_SRC" "$VM_USER@$VM_HOST:$VM_BACKUP_DIR/" 2>>"$LOG")
|
||||
if [ $? -ne 0 ]; then
|
||||
log "❌ rsync of backup/ failed"
|
||||
exit 1
|
||||
fi
|
||||
CHANGES="${CHANGES}${BACKUP_CHANGES}"
|
||||
|
||||
CONFIG_CHANGED=""
|
||||
if ! scp $SCP_OPTS "$LIVE_CONFIG" "$VM_USER@$VM_HOST:$VM_DEPLOY_DIR/config.py.new" &>>"$LOG"; then
|
||||
log "❌ Could not copy live config.py to VM"
|
||||
|
||||
Reference in New Issue
Block a user