Files
HomeAudit/migration_scripts/prepare_seamless_migration.sh
admin 705a2757c1 Major infrastructure migration and Vaultwarden PostgreSQL troubleshooting
COMPREHENSIVE CHANGES:

INFRASTRUCTURE MIGRATION:
- Migrated services to Docker Swarm on OMV800 (192.168.50.229)
- Deployed PostgreSQL database for Vaultwarden migration
- Updated all stack configurations for Docker Swarm compatibility
- Added comprehensive monitoring stack (Prometheus, Grafana, Blackbox)
- Implemented proper secret management for all services

VAULTWARDEN POSTGRESQL MIGRATION:
- Attempted migration from SQLite to PostgreSQL for NFS compatibility
- Created PostgreSQL stack with proper user/password configuration
- Built custom Vaultwarden image with PostgreSQL support
- Troubleshot persistent SQLite fallback issue despite PostgreSQL config
- Identified known issue where Vaultwarden silently falls back to SQLite
- Added ENABLE_DB_WAL=false to prevent filesystem compatibility issues
- Current status: Old Vaultwarden on lenovo410 still working, new one has config issues

PAPERLESS SERVICES:
- Successfully deployed Paperless-NGX and Paperless-AI on OMV800
- Both services running on ports 8000 and 3000 respectively
- Caddy configuration updated for external access
- Services accessible via paperless.pressmess.duckdns.org and paperless-ai.pressmess.duckdns.org

CADDY CONFIGURATION:
- Updated Caddyfile on Surface (192.168.50.254) for new service locations
- Fixed Vaultwarden reverse proxy to point to new Docker Swarm service
- Removed old notification hub reference that was causing conflicts
- All services properly configured for external access via DuckDNS

BACKUP AND DISCOVERY:
- Created comprehensive backup system for all hosts
- Generated detailed discovery reports for infrastructure analysis
- Implemented automated backup validation scripts
- Created migration progress tracking and verification reports

MONITORING STACK:
- Deployed Prometheus, Grafana, and Blackbox monitoring
- Created infrastructure and system overview dashboards
- Added proper service discovery and alerting configuration
- Implemented performance monitoring for all critical services

DOCUMENTATION:
- Reorganized documentation into logical structure
- Created comprehensive migration playbook and troubleshooting guides
- Added hardware specifications and optimization recommendations
- Documented all configuration changes and service dependencies

CURRENT STATUS:
- Paperless services:  Working and accessible externally
- Vaultwarden:  PostgreSQL configuration issues, old instance still working
- Monitoring:  Deployed and operational
- Caddy:  Updated and working for external access
- PostgreSQL:  Database running, connection issues with Vaultwarden

NEXT STEPS:
- Continue troubleshooting Vaultwarden PostgreSQL configuration
- Consider alternative approaches for Vaultwarden migration
- Validate all external service access
- Complete final migration validation

TECHNICAL NOTES:
- Used Docker Swarm for orchestration on OMV800
- Implemented proper secret management for sensitive data
- Added comprehensive logging and monitoring
- Created automated backup and validation scripts
2025-08-30 20:18:44 -04:00

196 lines
5.4 KiB
Bash
Executable File

#!/bin/bash
# PREPARE SEAMLESS MIGRATION
# Sets up infrastructure for zero-downtime service migration
set -e
echo "🎯 PREPARING SEAMLESS MIGRATION INFRASTRUCTURE"
echo "=============================================="
# Check current service status
echo ""
echo "📊 CURRENT DOCKER SWARM STATUS"
echo "-------------------------------"
ssh root@omv800.local "docker service ls"
echo ""
# Check node status
echo "📊 NODE STATUS"
echo "--------------"
ssh root@omv800.local "docker node ls"
echo ""
# Check network status
echo "📊 NETWORK STATUS"
echo "-----------------"
ssh root@omv800.local "docker network ls --filter driver=overlay"
echo ""
# Check secrets status
echo "📊 SECRETS STATUS"
echo "-----------------"
ssh root@omv800.local "docker secret ls"
echo ""
# Prepare Caddyfile backup
echo ""
echo "🔧 PREPARING CADDYFILE BACKUP"
echo "-----------------------------"
echo "Creating backup of current Caddyfile..."
ssh jon@192.168.50.254 "sudo cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.backup.$(date +%Y%m%d_%H%M%S)"
echo "✅ Caddyfile backup created"
# Create migration-ready Caddyfile template
echo ""
echo "📝 CREATING MIGRATION CADDYFILE TEMPLATE"
echo "----------------------------------------"
cat > /tmp/migration_caddyfile_template.txt << 'EOF'
# MIGRATION-READY CADDYFILE TEMPLATE
# This template supports both old and new service endpoints
# Reusable snippet for standard TLS configuration
(standard_tls) {
tls {
dns duckdns {env.DUCKDNS_TOKEN}
resolvers 1.1.1.1 8.8.8.8
}
}
# --- MIGRATION PHASE 1: PARALLEL DEPLOYMENT ---
# Services running on both old and new infrastructure
# Nextcloud - OLD ENDPOINT (keep for fallback)
nextcloud-old.pressmess.duckdns.org {
reverse_proxy 192.168.50.229:8080
import standard_tls
}
# Nextcloud - NEW ENDPOINT (Docker Swarm)
nextcloud.pressmess.duckdns.org {
reverse_proxy 192.168.50.229:8080 # Will change to swarm endpoint
import standard_tls
}
# AppFlowy - OLD ENDPOINT (keep for fallback)
appflowy-old.pressmess.duckdns.org {
reverse_proxy 192.168.50.254:8080
import standard_tls
}
# AppFlowy - NEW ENDPOINT (Docker Swarm)
appflowy-server.pressmess.duckdns.org {
reverse_proxy 192.168.50.254:8080 # Will change to swarm endpoint
import standard_tls
}
# --- MIGRATION PHASE 2: CUTOVER ---
# After validation, update these to point to swarm services
# Example swarm service endpoints (to be updated during migration):
# nextcloud.pressmess.duckdns.org {
# reverse_proxy swarm-service-name:8080
# import standard_tls
# }
EOF
echo "✅ Migration Caddyfile template created at /tmp/migration_caddyfile_template.txt"
# Create service migration checklist
echo ""
echo "📋 CREATING MIGRATION CHECKLIST"
echo "-------------------------------"
cat > /tmp/migration_checklist.txt << 'EOF'
# SEAMLESS MIGRATION CHECKLIST
## PHASE 1: PARALLEL DEPLOYMENT (Zero Downtime)
□ Deploy all services to Docker Swarm
□ Test each service individually
□ Verify data integrity
□ Update Caddyfile with new endpoints
□ Test new endpoints while keeping old ones
## PHASE 2: TRAFFIC MIGRATION (Gradual)
□ Start with low-traffic services
□ Monitor performance and stability
□ Gradually shift traffic to new services
□ Keep old services as fallback
## PHASE 3: CUTOVER (Minimal Downtime)
□ Update Caddyfile to point to swarm services
□ Reload Caddy configuration
□ Verify all services accessible
□ Stop old services
□ Clean up old infrastructure
## SERVICES TO MIGRATE:
□ Nextcloud (Port 8080)
□ AppFlowy (Port 8080)
□ Jellyfin (Port 8096)
□ Immich (Port 2283)
□ Gitea (Port 3001)
□ Joplin (Port 22300)
□ Vikunja (Port 3456)
□ n8n (Port 5678)
□ Home Assistant (Port 8123)
□ Paperless (Port 8001)
□ Vaultwarden (Port 8088)
## VALIDATION CHECKLIST:
□ All services accessible via new endpoints
□ Data integrity verified
□ Performance acceptable
□ SSL certificates working
□ Backup procedures updated
□ Monitoring configured
□ Documentation updated
EOF
echo "✅ Migration checklist created at /tmp/migration_checklist.txt"
# Create rollback script
echo ""
echo "🔄 CREATING ROLLBACK SCRIPT"
echo "---------------------------"
cat > /tmp/rollback_migration.sh << 'EOF'
#!/bin/bash
# ROLLBACK MIGRATION SCRIPT
# Emergency rollback to previous state
set -e
echo "🚨 EMERGENCY ROLLBACK - MIGRATION"
echo "================================="
# Restore Caddyfile from backup
echo "Restoring Caddyfile from backup..."
ssh jon@192.168.50.254 "sudo cp /etc/caddy/Caddyfile.backup.* /etc/caddy/Caddyfile"
ssh jon@192.168.50.254 "sudo systemctl reload caddy"
# Stop swarm services
echo "Stopping Docker Swarm services..."
ssh root@omv800.local "docker stack rm postgresql mariadb redis" || true
# Restart old services
echo "Restarting original services..."
# Add commands to restart original services here
echo "✅ Rollback completed - services restored to previous state"
EOF
chmod +x /tmp/rollback_migration.sh
echo "✅ Rollback script created at /tmp/rollback_migration.sh"
echo ""
echo "🎯 MIGRATION PREPARATION COMPLETE"
echo "================================="
echo ""
echo "Next steps:"
echo "1. Review migration checklist: /tmp/migration_checklist.txt"
echo "2. Test rollback procedure: /tmp/rollback_migration.sh"
echo "3. Deploy application services to swarm"
echo "4. Begin parallel deployment phase"
echo ""
echo "✅ Ready for seamless migration!"