Files
HomeAudit/scripts/sync_vaultwarden_to_nfs.sh
admin 705a2757c1 Major infrastructure migration and Vaultwarden PostgreSQL troubleshooting
COMPREHENSIVE CHANGES:

INFRASTRUCTURE MIGRATION:
- Migrated services to Docker Swarm on OMV800 (192.168.50.229)
- Deployed PostgreSQL database for Vaultwarden migration
- Updated all stack configurations for Docker Swarm compatibility
- Added comprehensive monitoring stack (Prometheus, Grafana, Blackbox)
- Implemented proper secret management for all services

VAULTWARDEN POSTGRESQL MIGRATION:
- Attempted migration from SQLite to PostgreSQL for NFS compatibility
- Created PostgreSQL stack with proper user/password configuration
- Built custom Vaultwarden image with PostgreSQL support
- Troubleshot persistent SQLite fallback issue despite PostgreSQL config
- Identified known issue where Vaultwarden silently falls back to SQLite
- Added ENABLE_DB_WAL=false to prevent filesystem compatibility issues
- Current status: Old Vaultwarden on lenovo410 still working, new one has config issues

PAPERLESS SERVICES:
- Successfully deployed Paperless-NGX and Paperless-AI on OMV800
- Both services running on ports 8000 and 3000 respectively
- Caddy configuration updated for external access
- Services accessible via paperless.pressmess.duckdns.org and paperless-ai.pressmess.duckdns.org

CADDY CONFIGURATION:
- Updated Caddyfile on Surface (192.168.50.254) for new service locations
- Fixed Vaultwarden reverse proxy to point to new Docker Swarm service
- Removed old notification hub reference that was causing conflicts
- All services properly configured for external access via DuckDNS

BACKUP AND DISCOVERY:
- Created comprehensive backup system for all hosts
- Generated detailed discovery reports for infrastructure analysis
- Implemented automated backup validation scripts
- Created migration progress tracking and verification reports

MONITORING STACK:
- Deployed Prometheus, Grafana, and Blackbox monitoring
- Created infrastructure and system overview dashboards
- Added proper service discovery and alerting configuration
- Implemented performance monitoring for all critical services

DOCUMENTATION:
- Reorganized documentation into logical structure
- Created comprehensive migration playbook and troubleshooting guides
- Added hardware specifications and optimization recommendations
- Documented all configuration changes and service dependencies

CURRENT STATUS:
- Paperless services:  Working and accessible externally
- Vaultwarden:  PostgreSQL configuration issues, old instance still working
- Monitoring:  Deployed and operational
- Caddy:  Updated and working for external access
- PostgreSQL:  Database running, connection issues with Vaultwarden

NEXT STEPS:
- Continue troubleshooting Vaultwarden PostgreSQL configuration
- Consider alternative approaches for Vaultwarden migration
- Validate all external service access
- Complete final migration validation

TECHNICAL NOTES:
- Used Docker Swarm for orchestration on OMV800
- Implemented proper secret management for sensitive data
- Added comprehensive logging and monitoring
- Created automated backup and validation scripts
2025-08-30 20:18:44 -04:00

179 lines
5.3 KiB
Bash
Executable File

#!/bin/bash
# Sync Vaultwarden Data to NFS Share
# Safely copies current working data to NFS share for migration
set -euo pipefail
# Configuration
SOURCE_HOST="jonathan@192.168.50.181"
SOURCE_PATH="/home/jonathan/vaultwarden/data"
NFS_PATH="/mnt/vaultwarden"
LOG_FILE="./logs/vaultwarden_sync.log"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Logging function
log() {
echo -e "${BLUE}[$(date +'%Y-%m-%d %H:%M:%S')]${NC} $1" | tee -a "$LOG_FILE"
}
log_success() {
echo -e "${GREEN}[$(date +'%Y-%m-%d %H:%M:%S')] SUCCESS:${NC} $1" | tee -a "$LOG_FILE"
}
log_warning() {
echo -e "${YELLOW}[$(date +'%Y-%m-%d %H:%M:%S')] WARNING:${NC} $1" | tee -a "$LOG_FILE"
}
log_error() {
echo -e "${RED}[$(date +'%Y-%m-%d %H:%M:%S')] ERROR:${NC} $1" | tee -a "$LOG_FILE"
}
# Create log directory
mkdir -p "$(dirname "$LOG_FILE")"
log "Starting Vaultwarden data sync to NFS share"
# Step 1: Verify source Vaultwarden is running
log "Step 1: Verifying source Vaultwarden container status"
if ! ssh "$SOURCE_HOST" "docker ps | grep -q vaultwarden"; then
log_error "Vaultwarden container is not running on $SOURCE_HOST"
exit 1
fi
# Get container ID
CONTAINER_ID=$(ssh "$SOURCE_HOST" "docker ps | grep vaultwarden | awk '{print \$1}'")
log "Found Vaultwarden container: $CONTAINER_ID"
# Step 2: Stop Vaultwarden for consistent sync
log "Step 2: Stopping Vaultwarden container for consistent sync"
ssh "$SOURCE_HOST" "docker stop $CONTAINER_ID"
# Wait a moment for graceful shutdown
sleep 5
# Step 3: Verify NFS mount is accessible
log "Step 3: Verifying NFS mount accessibility"
if ! ssh "$SOURCE_HOST" "[ -d '$NFS_PATH' ]"; then
log_error "NFS path $NFS_PATH does not exist on $SOURCE_HOST"
exit 1
fi
# Test write access
if ! ssh "$SOURCE_HOST" "touch '$NFS_PATH/test_write' && rm -f '$NFS_PATH/test_write'"; then
log_error "Cannot write to NFS path $NFS_PATH"
exit 1
fi
log_success "NFS mount is accessible and writable"
# Step 4: Create backup of current NFS data (just in case)
log "Step 4: Creating backup of current NFS data"
NFS_BACKUP="/tmp/vaultwarden_nfs_backup_$(date +%Y%m%d_%H%M%S).tar.gz"
ssh "$SOURCE_HOST" "cd '$NFS_PATH' && tar czf '$NFS_BACKUP' ."
if ssh "$SOURCE_HOST" "[ -f '$NFS_BACKUP' ]"; then
log_success "NFS backup created: $NFS_BACKUP"
else
log_warning "Failed to create NFS backup"
fi
# Step 5: Clear NFS directory and sync data
log "Step 5: Clearing NFS directory and syncing data"
ssh "$SOURCE_HOST" "rm -rf '$NFS_PATH'/*"
# Sync data from source to NFS
log "Syncing data from source to NFS"
ssh "$SOURCE_HOST" "rsync -av --delete '$SOURCE_PATH/' '$NFS_PATH/'"
# Step 6: Verify sync
log "Step 6: Verifying data sync"
SOURCE_COUNT=$(ssh "$SOURCE_HOST" "find '$SOURCE_PATH' -type f | wc -l")
NFS_COUNT=$(ssh "$SOURCE_HOST" "find '$NFS_PATH' -type f | wc -l")
log "Source files: $SOURCE_COUNT"
log "NFS files: $NFS_COUNT"
if [ "$SOURCE_COUNT" -eq "$NFS_COUNT" ]; then
log_success "File count matches between source and NFS"
else
log_warning "File count mismatch: source=$SOURCE_COUNT, nfs=$NFS_COUNT"
fi
# Check for critical files
if ssh "$SOURCE_HOST" "[ -f '$NFS_PATH/db.sqlite3' ]"; then
log_success "SQLite database synced to NFS"
else
log_error "SQLite database not found in NFS"
exit 1
fi
if ssh "$SOURCE_HOST" "[ -f '$NFS_PATH/rsa_key.pem' ]"; then
log_success "RSA key synced to NFS"
else
log_error "RSA key not found in NFS"
exit 1
fi
# Step 7: Set proper permissions
log "Step 7: Setting proper permissions"
ssh "$SOURCE_HOST" "chmod 644 '$NFS_PATH/db.sqlite3'"
ssh "$SOURCE_HOST" "chmod 644 '$NFS_PATH/rsa_key.pem'"
ssh "$SOURCE_HOST" "chmod -R 755 '$NFS_PATH/attachments'"
ssh "$SOURCE_HOST" "chmod -R 755 '$NFS_PATH/icon_cache'"
ssh "$SOURCE_HOST" "chmod -R 755 '$NFS_PATH/sends'"
ssh "$SOURCE_HOST" "chmod -R 755 '$NFS_PATH/tmp'"
log_success "Permissions set correctly"
# Step 8: Restart Vaultwarden
log "Step 8: Restarting Vaultwarden container"
ssh "$SOURCE_HOST" "docker start $CONTAINER_ID"
# Wait for container to be healthy
log "Waiting for Vaultwarden to be healthy"
for i in {1..30}; do
if ssh "$SOURCE_HOST" "docker ps | grep -q vaultwarden.*healthy"; then
log_success "Vaultwarden container is healthy"
break
fi
if [ $i -eq 30 ]; then
log_error "Vaultwarden container failed to become healthy"
exit 1
fi
sleep 2
done
# Step 9: Final verification
log "Step 9: Final verification"
SOURCE_SIZE=$(ssh "$SOURCE_HOST" "stat -c%s '$SOURCE_PATH/db.sqlite3'")
NFS_SIZE=$(ssh "$SOURCE_HOST" "stat -c%s '$NFS_PATH/db.sqlite3'")
log "Source database size: ${SOURCE_SIZE} bytes"
log "NFS database size: ${NFS_SIZE} bytes"
if [ "$SOURCE_SIZE" -eq "$NFS_SIZE" ]; then
log_success "Database sizes match - sync completed successfully"
else
log_error "Database size mismatch - sync may have failed"
exit 1
fi
log ""
log "=== SYNC COMPLETED SUCCESSFULLY ==="
log "✅ Current Vaultwarden data synced to NFS share"
log "✅ File counts match: $SOURCE_COUNT files"
log "✅ Database sizes match: ${SOURCE_SIZE} bytes"
log "✅ Vaultwarden container restarted and healthy"
log "✅ NFS backup created: $NFS_BACKUP"
log ""
log "Ready to proceed with migration!"
log_success "Vaultwarden data sync completed successfully!"