Files
HomeAudit/scripts/migrate_sqlite_to_postgres.sh
admin 705a2757c1 Major infrastructure migration and Vaultwarden PostgreSQL troubleshooting
COMPREHENSIVE CHANGES:

INFRASTRUCTURE MIGRATION:
- Migrated services to Docker Swarm on OMV800 (192.168.50.229)
- Deployed PostgreSQL database for Vaultwarden migration
- Updated all stack configurations for Docker Swarm compatibility
- Added comprehensive monitoring stack (Prometheus, Grafana, Blackbox)
- Implemented proper secret management for all services

VAULTWARDEN POSTGRESQL MIGRATION:
- Attempted migration from SQLite to PostgreSQL for NFS compatibility
- Created PostgreSQL stack with proper user/password configuration
- Built custom Vaultwarden image with PostgreSQL support
- Troubleshot persistent SQLite fallback issue despite PostgreSQL config
- Identified known issue where Vaultwarden silently falls back to SQLite
- Added ENABLE_DB_WAL=false to prevent filesystem compatibility issues
- Current status: Old Vaultwarden on lenovo410 still working, new one has config issues

PAPERLESS SERVICES:
- Successfully deployed Paperless-NGX and Paperless-AI on OMV800
- Both services running on ports 8000 and 3000 respectively
- Caddy configuration updated for external access
- Services accessible via paperless.pressmess.duckdns.org and paperless-ai.pressmess.duckdns.org

CADDY CONFIGURATION:
- Updated Caddyfile on Surface (192.168.50.254) for new service locations
- Fixed Vaultwarden reverse proxy to point to new Docker Swarm service
- Removed old notification hub reference that was causing conflicts
- All services properly configured for external access via DuckDNS

BACKUP AND DISCOVERY:
- Created comprehensive backup system for all hosts
- Generated detailed discovery reports for infrastructure analysis
- Implemented automated backup validation scripts
- Created migration progress tracking and verification reports

MONITORING STACK:
- Deployed Prometheus, Grafana, and Blackbox monitoring
- Created infrastructure and system overview dashboards
- Added proper service discovery and alerting configuration
- Implemented performance monitoring for all critical services

DOCUMENTATION:
- Reorganized documentation into logical structure
- Created comprehensive migration playbook and troubleshooting guides
- Added hardware specifications and optimization recommendations
- Documented all configuration changes and service dependencies

CURRENT STATUS:
- Paperless services:  Working and accessible externally
- Vaultwarden:  PostgreSQL configuration issues, old instance still working
- Monitoring:  Deployed and operational
- Caddy:  Updated and working for external access
- PostgreSQL:  Database running, connection issues with Vaultwarden

NEXT STEPS:
- Continue troubleshooting Vaultwarden PostgreSQL configuration
- Consider alternative approaches for Vaultwarden migration
- Validate all external service access
- Complete final migration validation

TECHNICAL NOTES:
- Used Docker Swarm for orchestration on OMV800
- Implemented proper secret management for sensitive data
- Added comprehensive logging and monitoring
- Created automated backup and validation scripts
2025-08-30 20:18:44 -04:00

118 lines
3.8 KiB
Bash
Executable File

#!/bin/bash
# Migrate Vaultwarden from SQLite to PostgreSQL
# This script migrates the existing SQLite database to PostgreSQL
set -euo pipefail
# Configuration
SOURCE_HOST="jonathan@192.168.50.181"
SWARM_MANAGER="root@192.168.50.229"
LOG_FILE="./logs/sqlite_to_postgres_migration.log"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Logging function
log() {
echo -e "${BLUE}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $1" | tee -a "$LOG_FILE"
}
log_success() {
echo -e "${GREEN}[$(date +'%Y-%m-%d %H:%M:%S')] SUCCESS:${NC} $1" | tee -a "$LOG_FILE"
}
log_warning() {
echo -e "${YELLOW}[$(date +'%Y-%m-%d %H:%M:%S')] WARNING:${NC} $1" | tee -a "$LOG_FILE"
}
log_error() {
echo -e "${RED}[$(date +'%Y-%m-%d %H:%M:%S')] ERROR:${NC} $1" | tee -a "$LOG_FILE"
}
# Create log directory
mkdir -p "$(dirname "$LOG_FILE")"
log "Starting Vaultwarden SQLite to PostgreSQL migration"
# Step 1: Stop the current Vaultwarden service
log "Step 1: Stopping current Vaultwarden service"
ssh "$SWARM_MANAGER" "docker stack rm vaultwarden" || true
sleep 10
# Step 2: Create a temporary container to run the migration
log "Step 2: Creating migration container"
ssh "$SWARM_MANAGER" "docker run -d --name vaultwarden_migration --network caddy-public -v /export/vaultwarden:/data vaultwarden/server:1.30.5 sleep infinity"
# Step 3: Install pgloader in the migration container
log "Step 3: Installing pgloader in migration container"
ssh "$SWARM_MANAGER" "docker exec vaultwarden_migration sh -c 'apt-get update && apt-get install -y pgloader'"
# Step 4: Create migration script
log "Step 4: Creating migration script"
ssh "$SWARM_MANAGER" "docker exec vaultwarden_migration sh -c 'cat > /tmp/migrate.sql << \"EOF\"
LOAD DATABASE
FROM sqlite:///data/db.sqlite3
INTO postgresql://vaultwarden:vaultwarden_secure_password_2024@postgres_postgres:5432/vaultwarden
WITH include drop, create tables, create indexes, reset sequences
SET work_mem to \"128MB\", maintenance_work_mem to \"512 MB\";
EOF'"
# Step 5: Run the migration
log "Step 5: Running database migration"
if ssh "$SWARM_MANAGER" "docker exec vaultwarden_migration pgloader /tmp/migrate.sql"; then
log_success "Database migration completed successfully"
else
log_error "Database migration failed"
exit 1
fi
# Step 6: Clean up migration container
log "Step 6: Cleaning up migration container"
ssh "$SWARM_MANAGER" "docker rm -f vaultwarden_migration"
# Step 7: Update Vaultwarden configuration to use PostgreSQL
log "Step 7: Deploying Vaultwarden with PostgreSQL configuration"
ssh "$SWARM_MANAGER" "docker stack deploy -c /opt/stacks/apps/vaultwarden.yml vaultwarden"
# Step 8: Wait for service to be ready
log "Step 8: Waiting for Vaultwarden service to be ready"
for i in {1..60}; do
if ssh "$SWARM_MANAGER" "docker service ls | grep vaultwarden | grep -q '1/1'"; then
log_success "Vaultwarden service is running"
break
fi
if [ $i -eq 60 ]; then
log_error "Vaultwarden service failed to start"
exit 1
fi
sleep 5
done
# Step 9: Verify the service is working
log "Step 9: Verifying service functionality"
sleep 10
if ssh "$SWARM_MANAGER" "curl -f http://localhost:8088/"; then
log_success "Vaultwarden is responding to HTTP requests"
else
log_warning "Vaultwarden is not responding to HTTP requests yet"
fi
log ""
log "=== MIGRATION COMPLETED SUCCESSFULLY ==="
log "✅ SQLite database migrated to PostgreSQL"
log "✅ Vaultwarden service deployed with PostgreSQL"
log "✅ Service is running and accessible"
log ""
log "Your Vaultwarden data has been successfully migrated to PostgreSQL!"
log "The service should now work properly without NFS/SQLite issues."
log_success "Vaultwarden SQLite to PostgreSQL migration completed successfully!"