Major infrastructure migration and Vaultwarden PostgreSQL troubleshooting

COMPREHENSIVE CHANGES:

INFRASTRUCTURE MIGRATION:
- Migrated services to Docker Swarm on OMV800 (192.168.50.229)
- Deployed PostgreSQL database for Vaultwarden migration
- Updated all stack configurations for Docker Swarm compatibility
- Added comprehensive monitoring stack (Prometheus, Grafana, Blackbox)
- Implemented proper secret management for all services

VAULTWARDEN POSTGRESQL MIGRATION:
- Attempted migration from SQLite to PostgreSQL for NFS compatibility
- Created PostgreSQL stack with proper user/password configuration
- Built custom Vaultwarden image with PostgreSQL support
- Troubleshot persistent SQLite fallback issue despite PostgreSQL config
- Identified known issue where Vaultwarden silently falls back to SQLite
- Added ENABLE_DB_WAL=false to prevent filesystem compatibility issues
- Current status: Old Vaultwarden on lenovo410 still working, new one has config issues

PAPERLESS SERVICES:
- Successfully deployed Paperless-NGX and Paperless-AI on OMV800
- Both services running on ports 8000 and 3000 respectively
- Caddy configuration updated for external access
- Services accessible via paperless.pressmess.duckdns.org and paperless-ai.pressmess.duckdns.org

CADDY CONFIGURATION:
- Updated Caddyfile on Surface (192.168.50.254) for new service locations
- Fixed Vaultwarden reverse proxy to point to new Docker Swarm service
- Removed old notification hub reference that was causing conflicts
- All services properly configured for external access via DuckDNS

BACKUP AND DISCOVERY:
- Created comprehensive backup system for all hosts
- Generated detailed discovery reports for infrastructure analysis
- Implemented automated backup validation scripts
- Created migration progress tracking and verification reports

MONITORING STACK:
- Deployed Prometheus, Grafana, and Blackbox monitoring
- Created infrastructure and system overview dashboards
- Added proper service discovery and alerting configuration
- Implemented performance monitoring for all critical services

DOCUMENTATION:
- Reorganized documentation into logical structure
- Created comprehensive migration playbook and troubleshooting guides
- Added hardware specifications and optimization recommendations
- Documented all configuration changes and service dependencies

CURRENT STATUS:
- Paperless services:  Working and accessible externally
- Vaultwarden:  PostgreSQL configuration issues, old instance still working
- Monitoring:  Deployed and operational
- Caddy:  Updated and working for external access
- PostgreSQL:  Database running, connection issues with Vaultwarden

NEXT STEPS:
- Continue troubleshooting Vaultwarden PostgreSQL configuration
- Consider alternative approaches for Vaultwarden migration
- Validate all external service access
- Complete final migration validation

TECHNICAL NOTES:
- Used Docker Swarm for orchestration on OMV800
- Implemented proper secret management for sensitive data
- Added comprehensive logging and monitoring
- Created automated backup and validation scripts
This commit is contained in:
admin
2025-08-30 20:18:44 -04:00
parent 9ea31368f5
commit 705a2757c1
155 changed files with 16781 additions and 1243 deletions

View File

@@ -0,0 +1,18 @@
[2025-08-29 13:08:19] Starting automated backup validation
[2025-08-29 13:08:19] Validating PostgreSQL backups...
[2025-08-29 13:08:19] ❌ No PostgreSQL backup files found
[2025-08-29 13:08:19] Validating MariaDB backups...
[2025-08-29 13:08:19] ❌ No MariaDB backup files found
[2025-08-29 13:08:20] Validating file backups...
[2025-08-29 13:08:20] ⚠️ No backup found for pattern: docker_volumes_*.tar.gz
[2025-08-29 13:08:20] ⚠️ No backup found for pattern: immich_data_*.tar.gz
[2025-08-29 13:08:20] ⚠️ No backup found for pattern: nextcloud_data_*.tar.gz
[2025-08-29 13:08:20] ⚠️ No backup found for pattern: homeassistant_data_*.tar.gz
[2025-08-29 13:08:20] File backup validation summary: 0 passed, 0 failed
[2025-08-29 13:08:20] Validating container configuration backups...
[2025-08-29 13:08:20] ❌ Container configuration backup directory not found
[2025-08-29 13:08:20] Validating Docker Compose file backups...
[2025-08-29 13:08:20] ❌ Docker Compose backup directory not found
[2025-08-29 13:08:20] Generating validation report...
[2025-08-29 13:08:20] ✅ Validation report generated: /home/jonathan/Coding/HomeAudit/logs/backup-validation-results.yaml
[2025-08-29 13:08:20] 🎉 Backup validation completed

View File

@@ -0,0 +1,41 @@
validation_run:
timestamp: "2025-08-29T13:08:19-04:00"
script_version: "1.0"
results:
- backup_type: "postgresql"
status: "FAILED"
details: "No backup files found"
validated_at: "2025-08-29T13:08:19-04:00"
- backup_type: "mariadb"
status: "FAILED"
details: "No backup files found"
validated_at: "2025-08-29T13:08:20-04:00"
- backup_type: "file_backup_docker_volumes_*.tar.gz"
status: "WARNING"
details: "No backup files found"
validated_at: "2025-08-29T13:08:20-04:00"
- backup_type: "file_backup_immich_data_*.tar.gz"
status: "WARNING"
details: "No backup files found"
validated_at: "2025-08-29T13:08:20-04:00"
- backup_type: "file_backup_nextcloud_data_*.tar.gz"
status: "WARNING"
details: "No backup files found"
validated_at: "2025-08-29T13:08:20-04:00"
- backup_type: "file_backup_homeassistant_data_*.tar.gz"
status: "WARNING"
details: "No backup files found"
validated_at: "2025-08-29T13:08:20-04:00"
- backup_type: "container_configs"
status: "FAILED"
details: "Backup directory missing"
validated_at: "2025-08-29T13:08:20-04:00"
- backup_type: "compose_files"
status: "FAILED"
details: "Backup directory missing"
validated_at: "2025-08-29T13:08:20-04:00"
summary:
total_tests: 8
passed_tests: 0
failed_tests: 4
warning_tests: 4

View File

@@ -0,0 +1,16 @@
{
"timestamp": "2025-08-29T15:32:32-04:00",
"backup_dir": "/export/omv800_backup/pre_migration_20250829_133008",
"completed_tasks": [
"4",
"2",
"4",
"0",
"2",
"2"
],
"total_tasks": [
""
],
"task_names": ["databases", "volumes", "configs", "secrets", "user_data", "system_configs"]
}

View File

@@ -0,0 +1,21 @@
{
"timestamp": "2025-08-29T15:15:52-04:00",
"backup_dir": "/export/omv800_backup/pre_migration_20250829_151551",
"completed_tasks": [
"0",
"0",
"0",
"0",
"0",
"0"
],
"total_tasks": [
"5",
"5",
"5",
"0",
"24",
"5"
],
"task_names": ["databases", "volumes", "configs", "secrets", "user_data", "system_configs"]
}

View File

@@ -0,0 +1,21 @@
{
"timestamp": "2025-08-29T15:16:13-04:00",
"backup_dir": "/export/omv800_backup/pre_migration_20250829_151611",
"completed_tasks": [
"0",
"0",
"0",
"0",
"0",
"0"
],
"total_tasks": [
"5",
"5",
"5",
"0",
"24",
"5"
],
"task_names": ["databases", "volumes", "configs", "secrets", "user_data", "system_configs"]
}

View File

@@ -0,0 +1,12 @@
[2025-08-29 13:27:33] === COMPREHENSIVE PRE-MIGRATION BACKUP STARTED ===
[2025-08-29 13:27:33] Timestamp: 20250829_132733
[2025-08-29 13:27:33] Backup directory: /export/omv800_backup/pre_migration_20250829_132733
[2025-08-29 13:27:33] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 13:27:33] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 13:27:33] Creating backup directory on raspberrypi...
[2025-08-29 13:27:35] === BACKING UP ALL DATABASES ===
[2025-08-29 13:27:35] Backing up databases on fedora (user: jonathan)...
[2025-08-29 13:27:35] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 13:27:36] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 13:27:36] Backing up Docker volume: DRIVER VOLUME NAME on fedora
[2025-08-29 13:27:36] Cleaning up temporary files...

View File

@@ -0,0 +1,242 @@
[2025-08-29 13:30:08] === COMPREHENSIVE PRE-MIGRATION BACKUP STARTED ===
[2025-08-29 13:30:08] Timestamp: 20250829_133008
[2025-08-29 13:30:08] Backup directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 13:30:08] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 13:30:08] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 13:30:08] Creating backup directory on raspberrypi...
[2025-08-29 13:30:09] === BACKING UP ALL DATABASES ===
[2025-08-29 13:30:09] Backing up databases on fedora (user: jonathan)...
[2025-08-29 13:30:09] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 13:30:09] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 13:30:09] Backing up Docker volume: 0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f on fedora
[2025-08-29 13:30:12] ✅ Volume backup created: /export/omv800_backup/pre_migration_20250829_133008/docker_volumes/fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz (710550 bytes)
[2025-08-29 13:30:12] === BACKING UP ALL CONFIGURATIONS ===
[2025-08-29 13:30:13] Backing up local configurations...
[2025-08-29 13:30:19] Backing up configurations on fedora (user: jonathan)...
[2025-08-29 13:30:19] Backing up configurations for fedora
[2025-08-29 13:48:20] ✅ Config backup created: /export/omv800_backup/pre_migration_20250829_133008/configurations/fedora_configs_20250829_133008.tar.gz (2558775123 bytes)
[2025-08-29 13:48:20] === BACKING UP ALL SECRETS AND SSL CERTIFICATES ===
[2025-08-29 13:48:20] Backing up local secrets...
[2025-08-29 13:48:21] Backing up SSL certificates on fedora (user: jonathan)...
[2025-08-29 13:48:21] Backing up secrets for fedora
[2025-08-29 13:48:22] ✅ Secrets backup created: /export/omv800_backup/pre_migration_20250829_133008/secrets/fedora_secrets_20250829_133008.tar.gz (881 bytes)
[2025-08-29 13:48:22] === BACKING UP ALL USER DATA AND APPLICATIONS ===
[2025-08-29 13:48:22] Backing up user data on fedora (user: jonathan)...
[2025-08-29 13:48:22] Backing up user data for fedora
[2025-08-29 15:09:28] Cleaning up temporary files...
[2025-08-29 15:16:50] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:16:50] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:16:50] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:16:50] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:16:50] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:16:50] 📊 Initial Progress:
[2025-08-29 15:16:50] === BACKING UP ALL DATABASES ===
[2025-08-29 15:16:50] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:16:50] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:16:50] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:16:51] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:16:51] Cleaning up temporary files...
[2025-08-29 15:18:00] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:18:00] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:18:00] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:18:00] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:18:00] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:18:00] 📊 Initial Progress:
[2025-08-29 15:18:00] === BACKING UP ALL DATABASES ===
[2025-08-29 15:18:00] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:18:00] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:18:00] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:18:01] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:18:01] Cleaning up temporary files...
[2025-08-29 15:18:35] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:18:35] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:18:35] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:18:35] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:18:36] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:18:40] Cleaning up temporary files...
[2025-08-29 15:19:41] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:19:41] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:19:41] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:19:41] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:19:41] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:19:45] 📊 Initial Progress:
[2025-08-29 15:19:46] === BACKING UP ALL DATABASES ===
[2025-08-29 15:19:46] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:19:46] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:19:46] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:19:46] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:19:46] Cleaning up temporary files...
[2025-08-29 15:20:02] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:20:02] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:20:02] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:20:02] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:20:02] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:20:08] 📊 Initial Progress:
[2025-08-29 15:20:08] === BACKING UP ALL DATABASES ===
[2025-08-29 15:20:08] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:20:08] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:20:08] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:20:08] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:20:08] Cleaning up temporary files...
[2025-08-29 15:21:29] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:21:29] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:21:29] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:21:29] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:21:29] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:21:33] 📊 Initial Progress:
[2025-08-29 15:21:34] === BACKING UP ALL DATABASES ===
[2025-08-29 15:21:34] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:21:34] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:21:34] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:21:34] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:21:34] Cleaning up temporary files...
[2025-08-29 15:21:45] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:21:45] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:21:45] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:21:45] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:21:45] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:21:50] 📊 Initial Progress:
[2025-08-29 15:21:50] === BACKING UP ALL DATABASES ===
[2025-08-29 15:21:50] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:21:50] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:21:50] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:21:51] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:21:51] Cleaning up temporary files...
[2025-08-29 15:26:20] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:26:20] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:26:20] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:26:20] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:26:20] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:26:26] 📊 Initial Progress:
[2025-08-29 15:26:26] === BACKING UP ALL DATABASES ===
[2025-08-29 15:26:26] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:26:26] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:26:27] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:26:27] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:26:27] === BACKING UP ALL CONFIGURATIONS ===
[2025-08-29 15:26:28] ⏭️ Skipping existing local config backup
[2025-08-29 15:26:28] === BACKING UP ALL SECRETS AND SSL CERTIFICATES ===
[2025-08-29 15:26:28] ⏭️ Skipping existing local secrets backup
[2025-08-29 15:26:28] === BACKING UP ALL USER DATA AND APPLICATIONS ===
[2025-08-29 15:26:28] === BACKING UP ALL SYSTEM CONFIGURATIONS ===
[2025-08-29 15:26:28] === CREATING BACKUP MANIFEST ===
[2025-08-29 15:26:54] ✅ Backup manifest created: /export/omv800_backup/pre_migration_20250829_133008/backup_manifest_20250829_133008.json
[2025-08-29 15:26:54] === VERIFYING BACKUP COMPLETENESS ===
[2025-08-29 15:26:55] 📊 Backup Summary:
[2025-08-29 15:26:55] Total files: 6
[2025-08-29 15:26:55] Total size: 18G
[2025-08-29 15:26:55] Backup location: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:26:56] ✅ Backup verification passed - all critical components present
[2025-08-29 15:26:56] === BACKUP COMPLETE ===
[2025-08-29 15:26:56] Backup location: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:26:56] Log file: /home/jonathan/Coding/HomeAudit/logs/comprehensive_backup_20250829_133008.log
[2025-08-29 15:26:56] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:26:56] 📊 Final Progress:
[2025-08-29 15:26:57] Cleaning up temporary files...
[2025-08-29 15:27:57] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:27:57] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:27:57] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:27:57] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:27:57] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:28:03] 📊 Initial Progress:
[2025-08-29 15:28:03] === BACKING UP ALL DATABASES ===
[2025-08-29 15:28:03] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:28:03] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:28:03] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:28:04] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:28:04] === BACKING UP ALL CONFIGURATIONS ===
[2025-08-29 15:28:05] ⏭️ Skipping existing local config backup
[2025-08-29 15:28:05] ⏭️ Skipping existing config backup for fedora
[2025-08-29 15:28:05] === BACKING UP ALL SECRETS AND SSL CERTIFICATES ===
[2025-08-29 15:28:06] ⏭️ Skipping existing local secrets backup
[2025-08-29 15:28:06] ⏭️ Skipping existing secrets backup for fedora
[2025-08-29 15:28:06] === BACKING UP ALL USER DATA AND APPLICATIONS ===
[2025-08-29 15:28:07] ⏭️ Skipping existing user data backup for fedora
[2025-08-29 15:28:07] === BACKING UP ALL SYSTEM CONFIGURATIONS ===
[2025-08-29 15:28:07] Backing up system configurations on fedora (user: jonathan)...
[2025-08-29 15:28:16] ✅ System config backup created: /export/omv800_backup/pre_migration_20250829_133008/system_configs/fedora_system_configs_20250829_133008.tar.gz (8243 bytes)
[2025-08-29 15:28:17] === CREATING BACKUP MANIFEST ===
[2025-08-29 15:28:43] ✅ Backup manifest created: /export/omv800_backup/pre_migration_20250829_133008/backup_manifest_20250829_133008.json
[2025-08-29 15:28:43] === VERIFYING BACKUP COMPLETENESS ===
[2025-08-29 15:28:44] 📊 Backup Summary:
[2025-08-29 15:28:44] Total files: 6
[2025-08-29 15:28:44] Total size: 18G
[2025-08-29 15:28:44] Backup location: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:28:45] ✅ Backup verification passed - all critical components present
[2025-08-29 15:28:45] === BACKUP COMPLETE ===
[2025-08-29 15:28:45] Backup location: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:28:45] Log file: /home/jonathan/Coding/HomeAudit/logs/comprehensive_backup_20250829_133008.log
[2025-08-29 15:28:45] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:28:45] 📊 Final Progress:
[2025-08-29 15:28:47] Cleaning up temporary files...
[2025-08-29 15:29:49] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:29:49] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:29:49] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:29:49] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:29:49] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:29:53] 📊 Initial Progress:
[2025-08-29 15:29:53] === BACKING UP ALL DATABASES ===
[2025-08-29 15:29:53] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:29:53] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:29:53] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:29:54] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:29:54] === BACKING UP ALL CONFIGURATIONS ===
[2025-08-29 15:29:55] ⏭️ Skipping existing local config backup
[2025-08-29 15:29:55] ⏭️ Skipping existing config backup for fedora
[2025-08-29 15:29:55] === BACKING UP ALL SECRETS AND SSL CERTIFICATES ===
[2025-08-29 15:29:56] ⏭️ Skipping existing local secrets backup
[2025-08-29 15:29:56] ⏭️ Skipping existing secrets backup for fedora
[2025-08-29 15:29:56] === BACKING UP ALL USER DATA AND APPLICATIONS ===
[2025-08-29 15:29:57] ⏭️ Skipping existing user data backup for fedora
[2025-08-29 15:29:57] === BACKING UP ALL SYSTEM CONFIGURATIONS ===
[2025-08-29 15:29:58] Backing up system configurations on fedora (user: jonathan)...
[2025-08-29 15:30:17] ✅ System config backup created: /export/omv800_backup/pre_migration_20250829_133008/system_configs/fedora_system_configs_20250829_133008.tar.gz (8243 bytes)
[2025-08-29 15:30:17] === CREATING BACKUP MANIFEST ===
[2025-08-29 15:30:41] ✅ Backup manifest created: /export/omv800_backup/pre_migration_20250829_133008/backup_manifest_20250829_133008.json
[2025-08-29 15:30:41] === VERIFYING BACKUP COMPLETENESS ===
[2025-08-29 15:30:41] 📊 Backup Summary:
[2025-08-29 15:30:41] Total files: 7
[2025-08-29 15:30:41] Total size: 18G
[2025-08-29 15:30:42] Backup location: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:30:43] ✅ Backup verification passed - all critical components present
[2025-08-29 15:30:43] === BACKUP COMPLETE ===
[2025-08-29 15:30:43] Backup location: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:30:43] Log file: /home/jonathan/Coding/HomeAudit/logs/comprehensive_backup_20250829_133008.log
[2025-08-29 15:30:43] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:30:43] 📊 Final Progress:
[2025-08-29 15:30:43] Cleaning up temporary files...
[2025-08-29 15:31:55] 🔄 RESUMING COMPREHENSIVE PRE-MIGRATION BACKUP
[2025-08-29 15:31:55] Resume directory: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:31:55] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:31:55] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:31:55] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:32:00] 📊 Initial Progress:
[2025-08-29 15:32:00] === BACKING UP ALL DATABASES ===
[2025-08-29 15:32:00] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:32:00] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:32:00] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:32:00] ⏭️ Skipping existing volume backup: fedora_0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f_20250829_133008.tar.gz
[2025-08-29 15:32:01] === BACKING UP ALL CONFIGURATIONS ===
[2025-08-29 15:32:01] ⏭️ Skipping existing local config backup
[2025-08-29 15:32:02] ⏭️ Skipping existing config backup for fedora
[2025-08-29 15:32:02] === BACKING UP ALL SECRETS AND SSL CERTIFICATES ===
[2025-08-29 15:32:03] ⏭️ Skipping existing local secrets backup
[2025-08-29 15:32:03] ⏭️ Skipping existing secrets backup for fedora
[2025-08-29 15:32:03] === BACKING UP ALL USER DATA AND APPLICATIONS ===
[2025-08-29 15:32:04] ⏭️ Skipping existing user data backup for fedora
[2025-08-29 15:32:04] === BACKING UP ALL SYSTEM CONFIGURATIONS ===
[2025-08-29 15:32:05] ⏭️ Skipping existing system config backup for fedora
[2025-08-29 15:32:05] === CREATING BACKUP MANIFEST ===
[2025-08-29 15:32:30] ✅ Backup manifest created: /export/omv800_backup/pre_migration_20250829_133008/backup_manifest_20250829_133008.json
[2025-08-29 15:32:30] === VERIFYING BACKUP COMPLETENESS ===
[2025-08-29 15:32:31] 📊 Backup Summary:
[2025-08-29 15:32:31] Total files: 7
[2025-08-29 15:32:31] Total size: 18G
[2025-08-29 15:32:31] Backup location: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:32:32] ✅ Backup verification passed - all critical components present
[2025-08-29 15:32:32] === BACKUP COMPLETE ===
[2025-08-29 15:32:32] Backup location: /export/omv800_backup/pre_migration_20250829_133008
[2025-08-29 15:32:32] Log file: /home/jonathan/Coding/HomeAudit/logs/comprehensive_backup_20250829_133008.log
[2025-08-29 15:32:32] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_133008.json
[2025-08-29 15:32:32] 📊 Final Progress:
[2025-08-29 15:32:32] Cleaning up temporary files...

View File

@@ -0,0 +1,9 @@
[2025-08-29 15:15:51] === COMPREHENSIVE PRE-MIGRATION BACKUP STARTED ===
[2025-08-29 15:15:51] Timestamp: 20250829_151551
[2025-08-29 15:15:51] Backup directory: /export/omv800_backup/pre_migration_20250829_151551
[2025-08-29 15:15:51] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:15:51] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:15:51] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_151551.json
[2025-08-29 15:15:51] Creating backup directory on raspberrypi...
[2025-08-29 15:15:53] 📊 Initial Progress:
[2025-08-29 15:15:53] Cleaning up temporary files...

View File

@@ -0,0 +1,14 @@
[2025-08-29 15:16:11] === COMPREHENSIVE PRE-MIGRATION BACKUP STARTED ===
[2025-08-29 15:16:11] Timestamp: 20250829_151611
[2025-08-29 15:16:11] Backup directory: /export/omv800_backup/pre_migration_20250829_151611
[2025-08-29 15:16:11] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 15:16:11] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 15:16:11] Progress file: /home/jonathan/Coding/HomeAudit/logs/backup_progress_20250829_151611.json
[2025-08-29 15:16:11] Creating backup directory on raspberrypi...
[2025-08-29 15:16:12] 📊 Initial Progress:
[2025-08-29 15:16:13] === BACKING UP ALL DATABASES ===
[2025-08-29 15:16:13] Backing up databases on fedora (user: jonathan)...
[2025-08-29 15:16:13] === BACKING UP ALL DOCKER VOLUMES ===
[2025-08-29 15:16:13] Backing up Docker volumes on fedora (user: jonathan)...
[2025-08-29 15:16:13] 🔄 Backing up Docker volume: 0c31f3704137a3c4b0d2acc8070b5c2a432e785536274638736245d94f19ac1f on fedora
[2025-08-29 15:16:15] Cleaning up temporary files...

View File

@@ -0,0 +1,33 @@
[2025-08-29 13:14:07] === COMPREHENSIVE BACKUP TARGET DISCOVERY STARTED ===
[2025-08-29 13:14:07] Timestamp: 20250829_131407
[2025-08-29 13:14:07] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 13:14:07] === DISCOVERING ALL HOSTS ===
[2025-08-29 13:14:07] Reading hosts from inventory.ini...
[2025-08-29 13:14:07] Checking connectivity to fedora_servers...
[2025-08-29 13:14:11] Checking connectivity to offline_devices...
[2025-08-29 13:14:14] Checking connectivity to mobile_devices...
[2025-08-29 13:14:18] Checking connectivity to network_infrastructure...
[2025-08-29 13:14:22] Checking connectivity to all_linux:children...
[2025-08-29 13:14:26] Checking connectivity to all_linux:vars...
[2025-08-29 13:14:30] Checking known hosts from documentation...
[2025-08-29 13:14:30] === DISCOVERING DOCKER ENVIRONMENTS ===
[2025-08-29 13:14:30] Checking Docker on fedora_servers...
[2025-08-29 13:14:34] Checking Docker on offline_devices...
[2025-08-29 13:14:38] Checking Docker on mobile_devices...
[2025-08-29 13:14:42] Checking Docker on network_infrastructure...
[2025-08-29 13:14:46] Checking Docker on all_linux:children...
[2025-08-29 13:14:49] Checking Docker on all_linux:vars...
[2025-08-29 13:14:53] === DISCOVERING ALL DATABASES ===
[2025-08-29 13:14:53] === DISCOVERING ALL VOLUMES AND PERSISTENT DATA ===
[2025-08-29 13:14:53] === DISCOVERING ALL CONFIGURATIONS ===
[2025-08-29 13:14:53] Discovering local configurations...
[2025-08-29 13:14:53] === DISCOVERING ALL SECRETS AND SENSITIVE DATA ===
[2025-08-29 13:14:53] Discovering local secrets...
[2025-08-29 13:14:53] === DISCOVERING ALL NETWORK CONFIGURATIONS ===
[2025-08-29 13:14:53] Discovering local network configuration...
[2025-08-29 13:14:53] === DISCOVERING ALL USER DATA AND APPLICATIONS ===
[2025-08-29 13:14:53] === GENERATING DISCOVERY SUMMARY ===
[2025-08-29 13:14:53] Discovery summary generated: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results/DISCOVERY_SUMMARY.md
[2025-08-29 13:14:53] === DISCOVERY COMPLETE ===
[2025-08-29 13:14:53] Results saved to: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 13:14:53] Cleaning up temporary files...

View File

@@ -0,0 +1,15 @@
[2025-08-29 13:16:14] === COMPREHENSIVE BACKUP TARGET DISCOVERY STARTED ===
[2025-08-29 13:16:14] Timestamp: 20250829_131614
[2025-08-29 13:16:14] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 13:16:14] === DISCOVERING ALL HOSTS ===
[2025-08-29 13:16:14] Checking connectivity to fedora...
[2025-08-29 13:16:14] Checking connectivity to omvbackup...
[2025-08-29 13:16:18] Checking connectivity to lenovo...
[2025-08-29 13:16:18] Checking connectivity to lenovo420...
[2025-08-29 13:16:18] Checking connectivity to omv800...
[2025-08-29 13:16:19] Checking connectivity to surface...
[2025-08-29 13:16:19] Checking connectivity to audrey...
[2025-08-29 13:16:19] Checking connectivity to raspberrypi...
[2025-08-29 13:16:19] === DISCOVERING DOCKER ENVIRONMENTS ===
[2025-08-29 13:16:19] Checking Docker on fedora...
[2025-08-29 13:16:40] Cleaning up temporary files...

View File

@@ -0,0 +1,15 @@
[2025-08-29 13:19:34] === COMPREHENSIVE BACKUP TARGET DISCOVERY STARTED ===
[2025-08-29 13:19:34] Timestamp: 20250829_131934
[2025-08-29 13:19:34] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 13:19:34] === DISCOVERING ALL HOSTS ===
[2025-08-29 13:19:34] Checking connectivity to fedora (user: jonathan)...
[2025-08-29 13:19:34] Checking connectivity to omvbackup (user: jon)...
[2025-08-29 13:19:37] Checking connectivity to lenovo (user: jonathan)...
[2025-08-29 13:19:38] Checking connectivity to lenovo420 (user: jon)...
[2025-08-29 13:19:38] Checking connectivity to omv800 (user: root)...
[2025-08-29 13:19:38] Checking connectivity to surface (user: jon)...
[2025-08-29 13:19:38] Checking connectivity to audrey (user: jon)...
[2025-08-29 13:19:38] Checking connectivity to raspberrypi (user: jon)...
[2025-08-29 13:19:38] === DISCOVERING DOCKER ENVIRONMENTS ===
[2025-08-29 13:19:38] Checking Docker on fedora (user: jonathan)...
[2025-08-29 13:20:08] Cleaning up temporary files...

View File

@@ -0,0 +1,39 @@
[2025-08-29 13:24:33] === AUTOMATED COMPREHENSIVE BACKUP TARGET DISCOVERY STARTED ===
[2025-08-29 13:24:33] Timestamp: 20250829_132433
[2025-08-29 13:24:33] Discovery directory: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 13:24:33] Password file: /home/jonathan/Coding/HomeAudit/secrets/ssh_passwords.env
[2025-08-29 13:24:33] === DISCOVERING ALL HOSTS ===
[2025-08-29 13:24:33] Checking connectivity to fedora (user: jonathan)...
[2025-08-29 13:24:33] Checking connectivity to omvbackup (user: jon)...
[2025-08-29 13:24:37] Checking connectivity to lenovo (user: jonathan)...
[2025-08-29 13:24:37] Checking connectivity to lenovo420 (user: jon)...
[2025-08-29 13:24:37] Checking connectivity to omv800 (user: root)...
[2025-08-29 13:24:38] Checking connectivity to surface (user: jon)...
[2025-08-29 13:24:38] Checking connectivity to audrey (user: jon)...
[2025-08-29 13:24:38] Checking connectivity to raspberrypi (user: jon)...
[2025-08-29 13:24:38] === DISCOVERING DOCKER ENVIRONMENTS ===
[2025-08-29 13:24:38] Checking Docker on fedora (user: jonathan)...
[2025-08-29 13:24:41] === DISCOVERING SYSTEMD SERVICES ===
[2025-08-29 13:24:41] Checking systemd services on fedora (user: jonathan)...
[2025-08-29 13:24:42] === DISCOVERING ALL DATABASES ===
[2025-08-29 13:24:42] Discovering databases on fedora (user: jonathan)...
[2025-08-29 13:24:42] === DISCOVERING ALL VOLUMES AND PERSISTENT DATA ===
[2025-08-29 13:24:42] Discovering volumes on fedora (user: jonathan)...
[2025-08-29 13:24:47] === DISCOVERING ALL CONFIGURATIONS ===
[2025-08-29 13:24:47] Discovering local configurations...
[2025-08-29 13:24:47] Discovering configurations on fedora (user: jonathan)...
[2025-08-29 13:25:07] === DISCOVERING ALL SECRETS AND SENSITIVE DATA ===
[2025-08-29 13:25:07] Discovering local secrets...
[2025-08-29 13:25:07] Discovering secrets on fedora (user: jonathan)...
[2025-08-29 13:25:13] === DISCOVERING ALL NETWORK CONFIGURATIONS ===
[2025-08-29 13:25:13] Discovering local network configuration...
[2025-08-29 13:25:13] Discovering network configuration on fedora (user: jonathan)...
[2025-08-29 13:25:15] === DISCOVERING ALL USER DATA AND APPLICATIONS ===
[2025-08-29 13:25:15] Discovering user data on fedora (user: jonathan)...
[2025-08-29 13:25:22] === DISCOVERING ALL APPLICATION-SPECIFIC DATA ===
[2025-08-29 13:25:22] Discovering application-specific data on fedora (user: jonathan)...
[2025-08-29 13:25:35] === GENERATING DISCOVERY SUMMARY ===
[2025-08-29 13:25:35] Discovery summary generated: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results/DISCOVERY_SUMMARY.md
[2025-08-29 13:25:35] === DISCOVERY COMPLETE ===
[2025-08-29 13:25:35] Results saved to: /home/jonathan/Coding/HomeAudit/comprehensive_discovery_results
[2025-08-29 13:25:35] Cleaning up temporary files...

View File

@@ -0,0 +1,20 @@
[2025-08-30 14:51:16] Starting Vaultwarden SQLite to PostgreSQL migration
[2025-08-30 14:51:16] Step 1: Stopping current Vaultwarden service
[2025-08-30 14:51:27] Step 2: Creating migration container
[2025-08-30 14:51:51] Starting Vaultwarden SQLite to PostgreSQL migration
[2025-08-30 14:51:51] Step 1: Stopping current Vaultwarden service
[2025-08-30 14:52:02] Step 2: Creating migration container
[2025-08-30 14:52:14] Starting Vaultwarden SQLite to PostgreSQL migration
[2025-08-30 14:52:14] Step 1: Stopping current Vaultwarden service
[2025-08-30 14:52:24] Step 2: Creating migration container
[2025-08-30 14:52:26] Step 3: Installing pgloader in migration container
[2025-08-30 14:52:47] Step 4: Creating migration script
[2025-08-30 14:52:48] Step 5: Running database migration
[2025-08-30 14:52:48] ERROR: Database migration failed
[2025-08-30 14:53:06] Starting Vaultwarden SQLite to PostgreSQL migration
[2025-08-30 14:53:06] Step 1: Stopping current Vaultwarden service
[2025-08-30 14:53:17] Step 2: Creating migration container
[2025-08-30 14:53:18] Step 3: Installing pgloader in migration container
[2025-08-30 14:53:34] Step 4: Creating migration script
[2025-08-30 14:53:34] Step 5: Running database migration
[2025-08-30 14:53:35] ERROR: Database migration failed

35
logs/vaultwarden_sync.log Normal file
View File

@@ -0,0 +1,35 @@
[2025-08-30 14:35:05] Starting Vaultwarden data sync to NFS share
[2025-08-30 14:35:05] Step 1: Verifying source Vaultwarden container status
[2025-08-30 14:35:06] Found Vaultwarden container: ef074c4fe727
[2025-08-30 14:35:06] Step 2: Stopping Vaultwarden container for consistent sync
[2025-08-30 14:35:12] Step 3: Verifying NFS mount accessibility
[2025-08-30 14:35:12] SUCCESS: NFS mount is accessible and writable
[2025-08-30 14:35:12] Step 4: Creating backup of current NFS data
[2025-08-30 14:35:13] SUCCESS: NFS backup created: /tmp/vaultwarden_nfs_backup_20250830_143512.tar.gz
[2025-08-30 14:35:13] Step 5: Clearing NFS directory and syncing data
[2025-08-30 14:35:14] Syncing data from source to NFS
[2025-08-30 14:35:19] Step 6: Verifying data sync
[2025-08-30 14:35:20] Source files: 778
[2025-08-30 14:35:20] NFS files: 778
[2025-08-30 14:35:20] SUCCESS: File count matches between source and NFS
[2025-08-30 14:35:20] SUCCESS: SQLite database synced to NFS
[2025-08-30 14:35:21] SUCCESS: RSA key synced to NFS
[2025-08-30 14:35:21] Step 7: Setting proper permissions
[2025-08-30 14:35:23] SUCCESS: Permissions set correctly
[2025-08-30 14:35:23] Step 8: Restarting Vaultwarden container
[2025-08-30 14:35:24] Waiting for Vaultwarden to be healthy
[2025-08-30 14:36:24] SUCCESS: Vaultwarden container is healthy
[2025-08-30 14:36:24] Step 9: Final verification
[2025-08-30 14:36:25] Source database size: 2445312 bytes
[2025-08-30 14:36:25] NFS database size: 2445312 bytes
[2025-08-30 14:36:25] SUCCESS: Database sizes match - sync completed successfully
[2025-08-30 14:36:25]
[2025-08-30 14:36:25] === SYNC COMPLETED SUCCESSFULLY ===
[2025-08-30 14:36:25] ✅ Current Vaultwarden data synced to NFS share
[2025-08-30 14:36:25] ✅ File counts match: 778 files
[2025-08-30 14:36:25] ✅ Database sizes match: 2445312 bytes
[2025-08-30 14:36:25] ✅ Vaultwarden container restarted and healthy
[2025-08-30 14:36:25] ✅ NFS backup created: /tmp/vaultwarden_nfs_backup_20250830_143512.tar.gz
[2025-08-30 14:36:25]
[2025-08-30 14:36:25] Ready to proceed with migration!
[2025-08-30 14:36:25] SUCCESS: Vaultwarden data sync completed successfully!

View File

@@ -0,0 +1,4 @@
[2025-08-30 13:51:56] Starting Vaultwarden migration pre-validation
[2025-08-30 13:51:56] Step 1: Verifying SSH connectivity to source host
[2025-08-30 13:52:24] Starting Vaultwarden migration pre-validation
[2025-08-30 13:52:24] Step 1: Verifying SSH connectivity to source host