Files
HomeAudit/migration_scripts/scripts/check_hardware_requirements.sh
admin 705a2757c1 Major infrastructure migration and Vaultwarden PostgreSQL troubleshooting
COMPREHENSIVE CHANGES:

INFRASTRUCTURE MIGRATION:
- Migrated services to Docker Swarm on OMV800 (192.168.50.229)
- Deployed PostgreSQL database for Vaultwarden migration
- Updated all stack configurations for Docker Swarm compatibility
- Added comprehensive monitoring stack (Prometheus, Grafana, Blackbox)
- Implemented proper secret management for all services

VAULTWARDEN POSTGRESQL MIGRATION:
- Attempted migration from SQLite to PostgreSQL for NFS compatibility
- Created PostgreSQL stack with proper user/password configuration
- Built custom Vaultwarden image with PostgreSQL support
- Troubleshot persistent SQLite fallback issue despite PostgreSQL config
- Identified known issue where Vaultwarden silently falls back to SQLite
- Added ENABLE_DB_WAL=false to prevent filesystem compatibility issues
- Current status: Old Vaultwarden on lenovo410 still working, new one has config issues

PAPERLESS SERVICES:
- Successfully deployed Paperless-NGX and Paperless-AI on OMV800
- Both services running on ports 8000 and 3000 respectively
- Caddy configuration updated for external access
- Services accessible via paperless.pressmess.duckdns.org and paperless-ai.pressmess.duckdns.org

CADDY CONFIGURATION:
- Updated Caddyfile on Surface (192.168.50.254) for new service locations
- Fixed Vaultwarden reverse proxy to point to new Docker Swarm service
- Removed old notification hub reference that was causing conflicts
- All services properly configured for external access via DuckDNS

BACKUP AND DISCOVERY:
- Created comprehensive backup system for all hosts
- Generated detailed discovery reports for infrastructure analysis
- Implemented automated backup validation scripts
- Created migration progress tracking and verification reports

MONITORING STACK:
- Deployed Prometheus, Grafana, and Blackbox monitoring
- Created infrastructure and system overview dashboards
- Added proper service discovery and alerting configuration
- Implemented performance monitoring for all critical services

DOCUMENTATION:
- Reorganized documentation into logical structure
- Created comprehensive migration playbook and troubleshooting guides
- Added hardware specifications and optimization recommendations
- Documented all configuration changes and service dependencies

CURRENT STATUS:
- Paperless services:  Working and accessible externally
- Vaultwarden:  PostgreSQL configuration issues, old instance still working
- Monitoring:  Deployed and operational
- Caddy:  Updated and working for external access
- PostgreSQL:  Database running, connection issues with Vaultwarden

NEXT STEPS:
- Continue troubleshooting Vaultwarden PostgreSQL configuration
- Consider alternative approaches for Vaultwarden migration
- Validate all external service access
- Complete final migration validation

TECHNICAL NOTES:
- Used Docker Swarm for orchestration on OMV800
- Implemented proper secret management for sensitive data
- Added comprehensive logging and monitoring
- Created automated backup and validation scripts
2025-08-30 20:18:44 -04:00

450 lines
16 KiB
Bash
Executable File

#!/bin/bash
# Hardware Requirements Validation Script
# Validates hardware requirements for the infrastructure migration
set -euo pipefail
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Function to print colored output
print_status() {
echo -e "${GREEN}[INFO]${NC} $1"
}
print_warning() {
echo -e "${YELLOW}[WARNING]${NC} $1"
}
print_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
print_header() {
echo -e "${BLUE}[HEADER]${NC} $1"
}
# Configuration
HOSTS=("omv800.local" "jonathan-2518f5u" "surface" "fedora" "audrey")
OUTPUT_FILE="${1:-/tmp/hardware_requirements_report.txt}"
MIN_RAM_GB=4
MIN_STORAGE_GB=10
MIN_CPU_CORES=2
MIN_DOCKER_VERSION="20.10"
MIN_KERNEL_VERSION="4.19"
# Function to check CPU requirements
check_cpu_requirements() {
print_header "Checking CPU Requirements"
local cpu_cores=$(nproc)
local cpu_model=$(grep "model name" /proc/cpuinfo | head -1 | cut -d':' -f2 | xargs)
local cpu_arch=$(uname -m)
echo "CPU Cores: $cpu_cores" >> "$OUTPUT_FILE"
echo "CPU Model: $cpu_model" >> "$OUTPUT_FILE"
echo "CPU Architecture: $cpu_arch" >> "$OUTPUT_FILE"
if [ "$cpu_cores" -ge "$MIN_CPU_CORES" ]; then
echo "CPU Requirements: PASSED" >> "$OUTPUT_FILE"
print_status "CPU Requirements: PASSED ($cpu_cores cores)"
else
echo "CPU Requirements: FAILED (minimum $MIN_CPU_CORES cores required)" >> "$OUTPUT_FILE"
print_error "CPU Requirements: FAILED (minimum $MIN_CPU_CORES cores required)"
return 1
fi
# Check for virtualization support
if grep -q "vmx\|svm" /proc/cpuinfo; then
echo "Virtualization Support: AVAILABLE" >> "$OUTPUT_FILE"
print_status "Virtualization Support: AVAILABLE"
else
echo "Virtualization Support: NOT AVAILABLE" >> "$OUTPUT_FILE"
print_warning "Virtualization Support: NOT AVAILABLE"
fi
}
# Function to check memory requirements
check_memory_requirements() {
print_header "Checking Memory Requirements"
local total_mem_kb=$(grep MemTotal /proc/meminfo | awk '{print $2}')
local total_mem_gb=$((total_mem_kb / 1024 / 1024))
local available_mem_kb=$(grep MemAvailable /proc/meminfo | awk '{print $2}')
local available_mem_gb=$((available_mem_kb / 1024 / 1024))
echo "Total Memory: ${total_mem_gb}GB" >> "$OUTPUT_FILE"
echo "Available Memory: ${available_mem_gb}GB" >> "$OUTPUT_FILE"
if [ "$total_mem_gb" -ge "$MIN_RAM_GB" ]; then
echo "Memory Requirements: PASSED" >> "$OUTPUT_FILE"
print_status "Memory Requirements: PASSED (${total_mem_gb}GB total)"
else
echo "Memory Requirements: FAILED (minimum ${MIN_RAM_GB}GB required)" >> "$OUTPUT_FILE"
print_error "Memory Requirements: FAILED (minimum ${MIN_RAM_GB}GB required)"
return 1
fi
# Check available memory
if [ "$available_mem_gb" -lt 1 ]; then
echo "Available Memory: WARNING (less than 1GB available)" >> "$OUTPUT_FILE"
print_warning "Available Memory: WARNING (less than 1GB available)"
else
echo "Available Memory: SUFFICIENT" >> "$OUTPUT_FILE"
print_status "Available Memory: SUFFICIENT (${available_mem_gb}GB available)"
fi
}
# Function to check storage requirements
check_storage_requirements() {
print_header "Checking Storage Requirements"
local root_partition=$(df / | awk 'NR==2 {print $1}')
local total_storage_kb=$(df / | awk 'NR==2 {print $2}')
local available_storage_kb=$(df / | awk 'NR==2 {print $4}')
local total_storage_gb=$((total_storage_kb / 1024 / 1024))
local available_storage_gb=$((available_storage_kb / 1024 / 1024))
echo "Root Partition: $root_partition" >> "$OUTPUT_FILE"
echo "Total Storage: ${total_storage_gb}GB" >> "$OUTPUT_FILE"
echo "Available Storage: ${available_storage_gb}GB" >> "$OUTPUT_FILE"
if [ "$total_storage_gb" -ge "$MIN_STORAGE_GB" ]; then
echo "Storage Requirements: PASSED" >> "$OUTPUT_FILE"
print_status "Storage Requirements: PASSED (${total_storage_gb}GB total)"
else
echo "Storage Requirements: FAILED (minimum ${MIN_STORAGE_GB}GB required)" >> "$OUTPUT_FILE"
print_error "Storage Requirements: FAILED (minimum ${MIN_STORAGE_GB}GB required)"
return 1
fi
# Check available storage
if [ "$available_storage_gb" -lt 5 ]; then
echo "Available Storage: WARNING (less than 5GB available)" >> "$OUTPUT_FILE"
print_warning "Available Storage: WARNING (less than 5GB available)"
else
echo "Available Storage: SUFFICIENT" >> "$OUTPUT_FILE"
print_status "Available Storage: SUFFICIENT (${available_storage_gb}GB available)"
fi
# Check for SSD
if grep -q "SSD\|nvme" /proc/mounts || lsblk -d -o name,rota | grep -q "0$"; then
echo "SSD Storage: DETECTED" >> "$OUTPUT_FILE"
print_status "SSD Storage: DETECTED"
else
echo "SSD Storage: NOT DETECTED (HDD may impact performance)" >> "$OUTPUT_FILE"
print_warning "SSD Storage: NOT DETECTED (HDD may impact performance)"
fi
}
# Function to check network requirements
check_network_requirements() {
print_header "Checking Network Requirements"
# Check network interfaces
local interfaces=$(ip link show | grep -E "^[0-9]+:" | cut -d: -f2 | tr -d ' ')
echo "Network Interfaces: $interfaces" >> "$OUTPUT_FILE"
# Check for Ethernet interface
if echo "$interfaces" | grep -q "eth\|en"; then
echo "Ethernet Interface: DETECTED" >> "$OUTPUT_FILE"
print_status "Ethernet Interface: DETECTED"
else
echo "Ethernet Interface: NOT DETECTED" >> "$OUTPUT_FILE"
print_warning "Ethernet Interface: NOT DETECTED"
fi
# Check network connectivity
if ping -c 1 8.8.8.8 >/dev/null 2>&1; then
echo "Internet Connectivity: AVAILABLE" >> "$OUTPUT_FILE"
print_status "Internet Connectivity: AVAILABLE"
else
echo "Internet Connectivity: NOT AVAILABLE" >> "$OUTPUT_FILE"
print_warning "Internet Connectivity: NOT AVAILABLE"
fi
# Check local network connectivity
if ping -c 1 192.168.1.1 >/dev/null 2>&1 || ping -c 1 192.168.50.1 >/dev/null 2>&1; then
echo "Local Network Connectivity: AVAILABLE" >> "$OUTPUT_FILE"
print_status "Local Network Connectivity: AVAILABLE"
else
echo "Local Network Connectivity: NOT AVAILABLE" >> "$OUTPUT_FILE"
print_warning "Local Network Connectivity: NOT AVAILABLE"
fi
}
# Function to check Docker requirements
check_docker_requirements() {
print_header "Checking Docker Requirements"
# Check if Docker is installed
if ! command -v docker >/dev/null 2>&1; then
echo "Docker: NOT INSTALLED" >> "$OUTPUT_FILE"
print_error "Docker: NOT INSTALLED"
return 1
fi
# Check Docker version
local docker_version=$(docker --version | cut -d' ' -f3 | cut -d',' -f1)
echo "Docker Version: $docker_version" >> "$OUTPUT_FILE"
# Compare versions
if [ "$(printf '%s\n' "$MIN_DOCKER_VERSION" "$docker_version" | sort -V | head -n1)" = "$MIN_DOCKER_VERSION" ]; then
echo "Docker Version: PASSED" >> "$OUTPUT_FILE"
print_status "Docker Version: PASSED ($docker_version)"
else
echo "Docker Version: FAILED (minimum $MIN_DOCKER_VERSION required)" >> "$OUTPUT_FILE"
print_error "Docker Version: FAILED (minimum $MIN_DOCKER_VERSION required)"
return 1
fi
# Check Docker daemon status
if docker info >/dev/null 2>&1; then
echo "Docker Daemon: RUNNING" >> "$OUTPUT_FILE"
print_status "Docker Daemon: RUNNING"
else
echo "Docker Daemon: NOT RUNNING" >> "$OUTPUT_FILE"
print_error "Docker Daemon: NOT RUNNING"
return 1
fi
# Check Docker storage driver
local storage_driver=$(docker info | grep "Storage Driver" | cut -d: -f2 | xargs)
echo "Docker Storage Driver: $storage_driver" >> "$OUTPUT_FILE"
if [ "$storage_driver" = "overlay2" ]; then
echo "Storage Driver: RECOMMENDED" >> "$OUTPUT_FILE"
print_status "Storage Driver: RECOMMENDED ($storage_driver)"
else
echo "Storage Driver: NOT RECOMMENDED (overlay2 preferred)" >> "$OUTPUT_FILE"
print_warning "Storage Driver: NOT RECOMMENDED (overlay2 preferred)"
fi
}
# Function to check kernel requirements
check_kernel_requirements() {
print_header "Checking Kernel Requirements"
local kernel_version=$(uname -r)
echo "Kernel Version: $kernel_version" >> "$OUTPUT_FILE"
# Extract major.minor version
local kernel_major_minor=$(echo "$kernel_version" | cut -d'-' -f1 | cut -d'.' -f1,2)
# Compare versions
if [ "$(printf '%s\n' "$MIN_KERNEL_VERSION" "$kernel_major_minor" | sort -V | head -n1)" = "$MIN_KERNEL_VERSION" ]; then
echo "Kernel Version: PASSED" >> "$OUTPUT_FILE"
print_status "Kernel Version: PASSED ($kernel_version)"
else
echo "Kernel Version: FAILED (minimum $MIN_KERNEL_VERSION required)" >> "$OUTPUT_FILE"
print_error "Kernel Version: FAILED (minimum $MIN_KERNEL_VERSION required)"
return 1
fi
# Check for required kernel modules
local required_modules=("overlay" "br_netfilter" "iptable_nat")
for module in "${required_modules[@]}"; do
if lsmod | grep -q "^$module"; then
echo "Kernel Module $module: LOADED" >> "$OUTPUT_FILE"
print_status "Kernel Module $module: LOADED"
else
echo "Kernel Module $module: NOT LOADED" >> "$OUTPUT_FILE"
print_warning "Kernel Module $module: NOT LOADED"
fi
done
}
# Function to check GPU requirements
check_gpu_requirements() {
print_header "Checking GPU Requirements"
# Check for NVIDIA GPU
if command -v nvidia-smi >/dev/null 2>&1; then
local nvidia_gpu=$(nvidia-smi --query-gpu=name --format=csv,noheader,nounits | head -1)
local nvidia_memory=$(nvidia-smi --query-gpu=memory.total --format=csv,noheader,nounits | head -1)
echo "NVIDIA GPU: DETECTED ($nvidia_gpu)" >> "$OUTPUT_FILE"
echo "NVIDIA Memory: ${nvidia_memory}MB" >> "$OUTPUT_FILE"
print_status "NVIDIA GPU: DETECTED ($nvidia_gpu)"
# Check Docker GPU support
if docker run --rm --gpus all nvidia/cuda:11.0-base-ubuntu20.04 nvidia-smi >/dev/null 2>&1; then
echo "Docker GPU Support: WORKING" >> "$OUTPUT_FILE"
print_status "Docker GPU Support: WORKING"
else
echo "Docker GPU Support: NOT WORKING" >> "$OUTPUT_FILE"
print_warning "Docker GPU Support: NOT WORKING"
fi
else
echo "NVIDIA GPU: NOT DETECTED" >> "$OUTPUT_FILE"
print_status "NVIDIA GPU: NOT DETECTED"
fi
# Check for Intel GPU
if lsmod | grep -q "i915"; then
echo "Intel GPU: DETECTED" >> "$OUTPUT_FILE"
print_status "Intel GPU: DETECTED"
else
echo "Intel GPU: NOT DETECTED" >> "$OUTPUT_FILE"
print_status "Intel GPU: NOT DETECTED"
fi
# Check for AMD GPU
if lsmod | grep -q "amdgpu"; then
echo "AMD GPU: DETECTED" >> "$OUTPUT_FILE"
print_status "AMD GPU: DETECTED"
else
echo "AMD GPU: NOT DETECTED" >> "$OUTPUT_FILE"
print_status "AMD GPU: NOT DETECTED"
fi
}
# Function to check system requirements
check_system_requirements() {
print_header "Checking System Requirements"
local os_name=$(grep "PRETTY_NAME" /etc/os-release | cut -d'"' -f2)
local os_version=$(grep "VERSION_ID" /etc/os-release | cut -d'"' -f2)
echo "Operating System: $os_name" >> "$OUTPUT_FILE"
echo "OS Version: $os_version" >> "$OUTPUT_FILE"
# Check for supported distributions
if echo "$os_name" | grep -q "Ubuntu\|Debian\|CentOS\|Fedora\|RHEL"; then
echo "OS Compatibility: SUPPORTED" >> "$OUTPUT_FILE"
print_status "OS Compatibility: SUPPORTED ($os_name)"
else
echo "OS Compatibility: UNKNOWN" >> "$OUTPUT_FILE"
print_warning "OS Compatibility: UNKNOWN ($os_name)"
fi
# Check systemd
if command -v systemctl >/dev/null 2>&1; then
echo "Systemd: AVAILABLE" >> "$OUTPUT_FILE"
print_status "Systemd: AVAILABLE"
else
echo "Systemd: NOT AVAILABLE" >> "$OUTPUT_FILE"
print_warning "Systemd: NOT AVAILABLE"
fi
# Check for required packages
local required_packages=("curl" "wget" "git" "ssh" "rsync")
for package in "${required_packages[@]}"; do
if command -v "$package" >/dev/null 2>&1; then
echo "Package $package: INSTALLED" >> "$OUTPUT_FILE"
print_status "Package $package: INSTALLED"
else
echo "Package $package: NOT INSTALLED" >> "$OUTPUT_FILE"
print_warning "Package $package: NOT INSTALLED"
fi
done
}
# Function to check security requirements
check_security_requirements() {
print_header "Checking Security Requirements"
# Check for firewall
if command -v ufw >/dev/null 2>&1 && ufw status | grep -q "active"; then
echo "Firewall (UFW): ACTIVE" >> "$OUTPUT_FILE"
print_status "Firewall (UFW): ACTIVE"
elif command -v firewall-cmd >/dev/null 2>&1 && firewall-cmd --state | grep -q "running"; then
echo "Firewall (firewalld): ACTIVE" >> "$OUTPUT_FILE"
print_status "Firewall (firewalld): ACTIVE"
else
echo "Firewall: NOT ACTIVE" >> "$OUTPUT_FILE"
print_warning "Firewall: NOT ACTIVE"
fi
# Check for SELinux
if command -v getenforce >/dev/null 2>&1; then
local selinux_status=$(getenforce)
echo "SELinux Status: $selinux_status" >> "$OUTPUT_FILE"
print_status "SELinux Status: $selinux_status"
else
echo "SELinux: NOT INSTALLED" >> "$OUTPUT_FILE"
print_status "SELinux: NOT INSTALLED"
fi
# Check for AppArmor
if command -v aa-status >/dev/null 2>&1; then
echo "AppArmor: INSTALLED" >> "$OUTPUT_FILE"
print_status "AppArmor: INSTALLED"
else
echo "AppArmor: NOT INSTALLED" >> "$OUTPUT_FILE"
print_status "AppArmor: NOT INSTALLED"
fi
}
# Function to generate requirements report
generate_report() {
print_header "Generating Hardware Requirements Report"
echo "=== Hardware Requirements Validation Report ===" > "$OUTPUT_FILE"
echo "Date: $(date)" >> "$OUTPUT_FILE"
echo "Hostname: $(hostname)" >> "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"
print_status "Requirements report initialized at $OUTPUT_FILE"
}
# Function to display usage
usage() {
echo "Usage: $0 [output_file]"
echo " output_file: Path to save requirements report (default: /tmp/hardware_requirements_report.txt)"
echo ""
echo "This script validates hardware requirements for infrastructure migration."
echo "It checks CPU, memory, storage, network, Docker, kernel, and security requirements."
}
# Main execution
main() {
print_header "Starting Hardware Requirements Validation"
# Initialize report
generate_report
# Run all checks
local failed_checks=0
check_cpu_requirements || ((failed_checks++))
check_memory_requirements || ((failed_checks++))
check_storage_requirements || ((failed_checks++))
check_network_requirements || ((failed_checks++))
check_docker_requirements || ((failed_checks++))
check_kernel_requirements || ((failed_checks++))
check_gpu_requirements
check_system_requirements
check_security_requirements
# Summary
echo "" >> "$OUTPUT_FILE"
echo "=== SUMMARY ===" >> "$OUTPUT_FILE"
if [ "$failed_checks" -eq 0 ]; then
echo "Overall Status: PASSED" >> "$OUTPUT_FILE"
print_status "Hardware requirements validation PASSED"
print_status "Report saved to: $OUTPUT_FILE"
else
echo "Overall Status: FAILED ($failed_checks critical checks failed)" >> "$OUTPUT_FILE"
print_error "Hardware requirements validation FAILED ($failed_checks critical checks failed)"
print_status "Report saved to: $OUTPUT_FILE"
exit 1
fi
}
# Parse command line arguments
if [ "$1" = "-h" ] || [ "$1" = "--help" ]; then
usage
exit 0
fi
# Run main function
main "$@"