Major infrastructure migration and Vaultwarden PostgreSQL troubleshooting
COMPREHENSIVE CHANGES: INFRASTRUCTURE MIGRATION: - Migrated services to Docker Swarm on OMV800 (192.168.50.229) - Deployed PostgreSQL database for Vaultwarden migration - Updated all stack configurations for Docker Swarm compatibility - Added comprehensive monitoring stack (Prometheus, Grafana, Blackbox) - Implemented proper secret management for all services VAULTWARDEN POSTGRESQL MIGRATION: - Attempted migration from SQLite to PostgreSQL for NFS compatibility - Created PostgreSQL stack with proper user/password configuration - Built custom Vaultwarden image with PostgreSQL support - Troubleshot persistent SQLite fallback issue despite PostgreSQL config - Identified known issue where Vaultwarden silently falls back to SQLite - Added ENABLE_DB_WAL=false to prevent filesystem compatibility issues - Current status: Old Vaultwarden on lenovo410 still working, new one has config issues PAPERLESS SERVICES: - Successfully deployed Paperless-NGX and Paperless-AI on OMV800 - Both services running on ports 8000 and 3000 respectively - Caddy configuration updated for external access - Services accessible via paperless.pressmess.duckdns.org and paperless-ai.pressmess.duckdns.org CADDY CONFIGURATION: - Updated Caddyfile on Surface (192.168.50.254) for new service locations - Fixed Vaultwarden reverse proxy to point to new Docker Swarm service - Removed old notification hub reference that was causing conflicts - All services properly configured for external access via DuckDNS BACKUP AND DISCOVERY: - Created comprehensive backup system for all hosts - Generated detailed discovery reports for infrastructure analysis - Implemented automated backup validation scripts - Created migration progress tracking and verification reports MONITORING STACK: - Deployed Prometheus, Grafana, and Blackbox monitoring - Created infrastructure and system overview dashboards - Added proper service discovery and alerting configuration - Implemented performance monitoring for all critical services DOCUMENTATION: - Reorganized documentation into logical structure - Created comprehensive migration playbook and troubleshooting guides - Added hardware specifications and optimization recommendations - Documented all configuration changes and service dependencies CURRENT STATUS: - Paperless services: ✅ Working and accessible externally - Vaultwarden: ❌ PostgreSQL configuration issues, old instance still working - Monitoring: ✅ Deployed and operational - Caddy: ✅ Updated and working for external access - PostgreSQL: ✅ Database running, connection issues with Vaultwarden NEXT STEPS: - Continue troubleshooting Vaultwarden PostgreSQL configuration - Consider alternative approaches for Vaultwarden migration - Validate all external service access - Complete final migration validation TECHNICAL NOTES: - Used Docker Swarm for orchestration on OMV800 - Implemented proper secret management for sensitive data - Added comprehensive logging and monitoring - Created automated backup and validation scripts
This commit is contained in:
admin
185
scripts/backup_hosts_individually.sh
Executable file
185
scripts/backup_hosts_individually.sh
Executable file
@@ -0,0 +1,185 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Individual host backup script
|
||||
# Runs backup for each host one by one with detailed output
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
# Load passwords
|
||||
source secrets/ssh_passwords.env
|
||||
|
||||
# Configuration
|
||||
BACKUP_TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
BACKUP_DIR="/export/omv800_backup/pre_migration_${BACKUP_TIMESTAMP}"
|
||||
|
||||
echo "=== INDIVIDUAL HOST BACKUP SCRIPT ==="
|
||||
echo "Backup directory: $BACKUP_DIR"
|
||||
echo "Timestamp: $BACKUP_TIMESTAMP"
|
||||
echo
|
||||
|
||||
# Create backup directory
|
||||
ssh jon@raspberrypi "mkdir -p $BACKUP_DIR"
|
||||
|
||||
# Function to backup a single host
|
||||
backup_host() {
|
||||
local host="$1"
|
||||
local user="$2"
|
||||
|
||||
echo "🔄 BACKING UP: $host (user: $user)"
|
||||
echo "=================================="
|
||||
|
||||
# Get password for this host
|
||||
case "$host" in
|
||||
"fedora")
|
||||
password="$FEDORA_PASSWORD"
|
||||
;;
|
||||
"lenovo")
|
||||
password="$LENOVO_PASSWORD"
|
||||
;;
|
||||
"lenovo420")
|
||||
password="$LENOVO420_PASSWORD"
|
||||
;;
|
||||
"omv800")
|
||||
password="$OMV800_PASSWORD"
|
||||
;;
|
||||
"surface")
|
||||
password="$SURFACE_PASSWORD"
|
||||
;;
|
||||
"audrey")
|
||||
password="$AUDREY_PASSWORD"
|
||||
;;
|
||||
"raspberrypi")
|
||||
password="$RASPBERRYPI_PASSWORD"
|
||||
;;
|
||||
*)
|
||||
echo "❌ No password configured for $host"
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# Test connectivity first
|
||||
echo " Testing connectivity..."
|
||||
if sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "echo 'Connection test successful'" 2>/dev/null; then
|
||||
echo "✅ Connectivity: SUCCESS"
|
||||
else
|
||||
echo "❌ Connectivity: FAILED"
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Create host-specific directories
|
||||
ssh jon@raspberrypi "mkdir -p $BACKUP_DIR/configurations $BACKUP_DIR/secrets $BACKUP_DIR/user_data $BACKUP_DIR/system_configs"
|
||||
|
||||
# Backup configurations
|
||||
echo " Backing up configurations..."
|
||||
if sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "tar czf /tmp/config_backup.tar.gz -C /etc . -C /home . 2>/dev/null || true" 2>/dev/null; then
|
||||
if sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "test -f /tmp/config_backup.tar.gz" 2>/dev/null; then
|
||||
scp "$user@$host:/tmp/config_backup.tar.gz" "/tmp/config_backup_temp.tar.gz" 2>/dev/null || true
|
||||
if [[ -f "/tmp/config_backup_temp.tar.gz" ]]; then
|
||||
rsync -avz "/tmp/config_backup_temp.tar.gz" "jon@raspberrypi:$BACKUP_DIR/configurations/${host}_configs_${BACKUP_TIMESTAMP}.tar.gz"
|
||||
local size=$(stat -c%s "/tmp/config_backup_temp.tar.gz" 2>/dev/null || echo "0")
|
||||
echo "✅ Configs: SUCCESS ($size bytes)"
|
||||
rm -f "/tmp/config_backup_temp.tar.gz" 2>/dev/null || true
|
||||
sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "rm -f /tmp/config_backup.tar.gz" 2>/dev/null || true
|
||||
else
|
||||
echo "❌ Configs: Failed to copy"
|
||||
fi
|
||||
else
|
||||
echo "❌ Configs: Failed to create"
|
||||
fi
|
||||
else
|
||||
echo "❌ Configs: Failed to connect"
|
||||
fi
|
||||
|
||||
# Backup secrets
|
||||
echo " Backing up secrets..."
|
||||
if sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "tar czf /tmp/secrets_backup.tar.gz -C /etc/ssl . -C /etc/letsencrypt . 2>/dev/null || true" 2>/dev/null; then
|
||||
if sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "test -f /tmp/secrets_backup.tar.gz" 2>/dev/null; then
|
||||
scp "$user@$host:/tmp/secrets_backup.tar.gz" "/tmp/secrets_backup_temp.tar.gz" 2>/dev/null || true
|
||||
if [[ -f "/tmp/secrets_backup_temp.tar.gz" ]]; then
|
||||
rsync -avz "/tmp/secrets_backup_temp.tar.gz" "jon@raspberrypi:$BACKUP_DIR/secrets/${host}_secrets_${BACKUP_TIMESTAMP}.tar.gz"
|
||||
local size=$(stat -c%s "/tmp/secrets_backup_temp.tar.gz" 2>/dev/null || echo "0")
|
||||
echo "✅ Secrets: SUCCESS ($size bytes)"
|
||||
rm -f "/tmp/secrets_backup_temp.tar.gz" 2>/dev/null || true
|
||||
sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "rm -f /tmp/secrets_backup.tar.gz" 2>/dev/null || true
|
||||
else
|
||||
echo "❌ Secrets: Failed to copy"
|
||||
fi
|
||||
else
|
||||
echo "❌ Secrets: Failed to create"
|
||||
fi
|
||||
else
|
||||
echo "❌ Secrets: Failed to connect"
|
||||
fi
|
||||
|
||||
# Backup user data
|
||||
echo " Backing up user data..."
|
||||
if sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "tar czf /tmp/user_data_backup.tar.gz --exclude='*/node_modules' --exclude='*/.git' --exclude='*/Downloads' --exclude='*/Videos' --exclude='*/Music' -C /home . -C /srv . 2>/dev/null || true" 2>/dev/null; then
|
||||
if sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "test -f /tmp/user_data_backup.tar.gz" 2>/dev/null; then
|
||||
scp "$user@$host:/tmp/user_data_backup.tar.gz" "/tmp/user_data_backup_temp.tar.gz" 2>/dev/null || true
|
||||
if [[ -f "/tmp/user_data_backup_temp.tar.gz" ]]; then
|
||||
rsync -avz "/tmp/user_data_backup_temp.tar.gz" "jon@raspberrypi:$BACKUP_DIR/user_data/${host}_user_data_${BACKUP_TIMESTAMP}.tar.gz"
|
||||
local size=$(stat -c%s "/tmp/user_data_backup_temp.tar.gz" 2>/dev/null || echo "0")
|
||||
echo "✅ User Data: SUCCESS ($size bytes)"
|
||||
rm -f "/tmp/user_data_backup_temp.tar.gz" 2>/dev/null || true
|
||||
sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "rm -f /tmp/user_data_backup.tar.gz" 2>/dev/null || true
|
||||
else
|
||||
echo "❌ User Data: Failed to copy"
|
||||
fi
|
||||
else
|
||||
echo "❌ User Data: Failed to create"
|
||||
fi
|
||||
else
|
||||
echo "❌ User Data: Failed to connect"
|
||||
fi
|
||||
|
||||
# Backup system configs
|
||||
echo " Backing up system configs..."
|
||||
if sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "tar czf /tmp/system_configs_backup.tar.gz -C /etc/systemd . -C /etc/network . -C /etc/docker . 2>/dev/null || true" 2>/dev/null; then
|
||||
if sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "test -f /tmp/system_configs_backup.tar.gz" 2>/dev/null; then
|
||||
scp "$user@$host:/tmp/system_configs_backup.tar.gz" "/tmp/system_configs_backup_temp.tar.gz" 2>/dev/null || true
|
||||
if [[ -f "/tmp/system_configs_backup_temp.tar.gz" ]]; then
|
||||
rsync -avz "/tmp/system_configs_backup_temp.tar.gz" "jon@raspberrypi:$BACKUP_DIR/system_configs/${host}_system_configs_${BACKUP_TIMESTAMP}.tar.gz"
|
||||
local size=$(stat -c%s "/tmp/system_configs_backup_temp.tar.gz" 2>/dev/null || echo "0")
|
||||
echo "✅ System Configs: SUCCESS ($size bytes)"
|
||||
rm -f "/tmp/system_configs_backup_temp.tar.gz" 2>/dev/null || true
|
||||
sshpass -p "$password" ssh -o ConnectTimeout=10 -o StrictHostKeyChecking=no "$user@$host" "rm -f /tmp/system_configs_backup.tar.gz" 2>/dev/null || true
|
||||
else
|
||||
echo "❌ System Configs: Failed to copy"
|
||||
fi
|
||||
else
|
||||
echo "❌ System Configs: Failed to create"
|
||||
fi
|
||||
else
|
||||
echo "❌ System Configs: Failed to connect"
|
||||
fi
|
||||
|
||||
echo "✅ COMPLETED: $host"
|
||||
echo "=================================="
|
||||
echo
|
||||
}
|
||||
|
||||
# Backup each accessible host
|
||||
echo "Starting individual host backups..."
|
||||
echo
|
||||
|
||||
# Test each host from all_hosts.txt
|
||||
while IFS=: read -r host user; do
|
||||
if [[ -z "$host" || "$host" == "localhost" ]]; then
|
||||
continue
|
||||
fi
|
||||
|
||||
# Skip omvbackup (DNS resolution issue)
|
||||
if [[ "$host" == "omvbackup" ]]; then
|
||||
echo "⏭️ SKIPPING: $host (DNS resolution issue)"
|
||||
echo
|
||||
continue
|
||||
fi
|
||||
|
||||
# Backup this host
|
||||
backup_host "$host" "$user"
|
||||
|
||||
done < "comprehensive_discovery_results/all_hosts.txt"
|
||||
|
||||
echo "=== INDIVIDUAL HOST BACKUP COMPLETE ==="
|
||||
echo "Backup directory: $BACKUP_DIR"
|
||||
echo "Check the backup results with: ssh jon@raspberrypi 'ls -la $BACKUP_DIR/'"
|
||||
Reference in New Issue
Block a user