diff --git a/docs/en/solutions/How_to_Migrate_MySQL_57_to_80.md b/docs/en/solutions/How_to_Migrate_MySQL_57_to_80.md new file mode 100644 index 0000000..6d94fa2 --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_MySQL_57_to_80.md @@ -0,0 +1,2414 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.x +--- + +# MySQL 5.7 to 8.0 Migration Guide + +## Background + +### The Challenge + +MySQL 5.7 reached End of Life (EOL) in October 2023, and organizations must upgrade to MySQL 8.0 to continue receiving security updates and to leverage new features. Migrating production databases involves complex considerations, including schema compatibility, character set changes, authentication plugin updates, and ensuring data integrity during the migration process. + +### The Solution + +This guide provides comprehensive, verified instructions for migrating MySQL 5.7 to 8.0 on Alauda Container Platform (ACP). The solution employs a mysqldump-based migration strategy with comprehensive validation: + +- **Proven Approach**: Validated in ACP 4.2.0 using Alauda Database Service for MySQL v4.2.0. +- **Complete Object Coverage**: Migrates all standard MySQL objects (tables, views, routines, triggers, events, users, grants). +- **Schema Compatibility**: Automated checks and fixes for MySQL 8.0 compatibility issues. +- **Comprehensive Verification**: Verification across 9 object categories, including view execution testing. +- **Minimal Risk**: Detailed rollback procedures and validation at each step. + +## Environment Information + +Applicable Versions: ACP >= 4.2.0, MySQL Operator >= 4.2.0 +Source: Percona XtraDB Cluster (PXC) 5.7.44 +Target: MySQL Group Replication (MGR) 8.0.44 + +## Tested and Verified + +This migration solution has been **verified** in Kubernetes environments using PXC 5.7.44 and MGR 8.0.44 clusters. + +### What Has Been Verified + +| Category | Verified Items | +|----------|----------------| +| **Basic Migration** | Tables, data rows, foreign keys, indexes | +| **Schema Compatibility** | Reserved keyword detection, ZEROFILL handling, invalid date defaults, TEXT column defaults | +| **Database Objects** | Stored procedures, functions, triggers, events, views (with execution testing) | +| **Users & Privileges** | User account creation, privilege migration, authentication plugin compatibility | +| **Character Sets** | utf8mb4 conversion, multi-language support (Chinese, Japanese, Latin accents), emoji preservation | +| **GTID Handling** | GTID_PURGED filtering for MGR targets, data integrity maintained | + +## Quick Reference + +### Key Concepts +- **Source Cluster**: Existing MySQL 5.7.44 PXC cluster. +- **Target Cluster**: New MySQL 8.0.44 MGR cluster. +- **GTID**: Global Transaction Identifiers for transaction tracking. +- **Schema Compatibility**: MySQL 8.0 reserved keywords and syntax changes +- **Character Set Migration**: Converting to utf8mb4 for full Unicode support +- **DEFINER Privileges**: Security context for stored routines/views/events/triggers + +### PXC vs MGR: Key Differences + +| Aspect | PXC 5.7 (Source) | MGR 8.0 (Target) | +|--------|-----------------|------------------| +| **Pod Name Pattern** | `${NAME}-pxc-0` | `${NAME}-0` | +| **Container Specifier** | Not required (defaults to mysql) | Required: `-c mysql` | +| **Primary Endpoint** | `${NAME}-proxysql.${NS}.svc.cluster.local:3306` | `${NAME}-read-write.${NS}.svc.cluster.local:3306` | +| **Replica Endpoint** | Same as primary (ProxySQL handles routing) | `${NAME}-read-only.${NS}.svc.cluster.local:3306` | +| **Replication Type** | Galera (synchronous multi-master) | Group Replication (single-primary with async replicas) | +| **Secret Name Pattern** | `${NAME}` | `mgr-${NAME}-password` | + +**Important:** Always check your actual pod names with `kubectl get pod -n ` before running migration commands. + +### Common Use Cases + +| Scenario | Database Size | Estimated Downtime | Section Reference | +|----------|---------------|-------------------|------------------| +| **Small Database** | < 10GB | 15-30 minutes | [Migration Procedure](#step-4-migrate-data-users-and-privileges) | +| **Medium Database** | 10-50GB | 30-60 minutes | [Migration Procedure](#step-4-migrate-data-users-and-privileges) | +| **Large Database** | 50-200GB | 1-2 hours | [Migration Procedure](#step-4-migrate-data-users-and-privileges) | +| **Schema Issues** | Any size | +1-2 hours for fixes | [Schema Compatibility](#step-1-schema-compatibility-analysis) | +| **Character Set Migration** | Any size | +30-60 minutes | [Character Set Migration](#step-2-character-set-and-collation-analysis) | + +## Prerequisites + +Before performing MySQL migration, ensure you have: + +- ACP v4.2.0 or later with MySQL Operator v4.2.0 or later +- MySQL plugin deployed following the [installation guide](https://docs.alauda.io/mysql-mgr/4.2/installation.html) +- Review the [Alauda MySQL MGR Documentation](https://docs.alauda.io/mysql-mgr/4.2/functions/01-create.html) to understand instance creation +- **Source Cluster Requirements**: + - A healthy MySQL 5.7.44 PXC cluster + - GTID mode enabled (`@@gtid_mode = ON`, `@@enforce_gtid_consistency = ON`) + - Root or administrative access credentials +- **Target Cluster Requirements**: + - A new MySQL 8.0.44 MGR cluster created *before* the migration + - Storage capacity of 2-3x the source database size + - Same or higher resource allocation (CPU/Memory) as source + - Network connectivity from your local machine to both clusters +- **Pre-Migration Tasks**: + - Complete [Schema Compatibility Analysis](#step-1-schema-compatibility-analysis) and fix issues + - Complete [Character Set Migration](#step-2-character-set-and-collation-analysis) if using legacy charsets + - Identify user databases to migrate (DO NOT include: `information_schema`, `mysql`, `performance_schema`, `sys`) + - Schedule maintenance window with application team + - Notify stakeholders about planned downtime + - Prepare rollback plan documented in [Disaster Recovery](#disaster-recovery) + +### Important Limitations + +- Application downtime is required during export and import to ensure consistency. +- Recommended maximum database size: 200GB (larger databases may require alternative approaches). +- GTID must be enabled on the source cluster. +- The target cluster must be created before the migration begins. +- Storage performance (IOPS/Throughput) on the target should match or exceed the source. +- Some MySQL 8.0 features (Roles, Caching SHA2 passwords) require post-migration configuration. + +## Getting Started + +Before executing migration commands, gather the following information: + +### 1. Get MySQL Root Password + +```bash +# For PXC 5.7 source +kubectl get secret -n -o jsonpath='{.data.root}' | base64 -d + +# For MGR 8.0 target +kubectl get secret mgr--password -n -o jsonpath='{.data.root}' | base64 -d +``` + +**Example:** +```bash +# Get source password +kubectl get secret source -n jpsu2-midautons -o jsonpath='{.data.root}' | base64 -d +# Output: root123@ + +# Get target password +kubectl get secret mgr-target-password -n jpsu2-midautons -o jsonpath='{.data.root}' | base64 -d +# Output: root123@ +``` + +### 2. Identify Pod Names + +```bash +# Check source PXC pods +kubectl get pod -n | grep +# Example output: source-pxc-0, source-pxc-1, source-pxc-2 + +# Check target MGR pods +kubectl get pod -n | grep +# Example output: target-0, target-1, target-2 + +# Verify MGR container name +kubectl describe pod -0 -n | grep "Container:" +# MGR pods have multiple containers - always use `-c mysql` for MySQL commands +``` + +### 3. Verify Cluster Status + +```bash +# Check PXC source status +kubectl get mysql -n +# Expected: STATE = ready, PXCSTATE = ready + +# Check MGR target status +kubectl get mysql -n +# Expected: All 3 members ready, STATUS = Running +``` + +### 4. kubectl Exec Best Practices + +When running MySQL commands via `kubectl exec`, follow these patterns: + +**For PXC 5.7 (source):** +```bash +# No container specifier needed for PXC +kubectl exec -pxc-0 -n -- \ + mysql -uroot -p -e "SQL_HERE" +``` + +**For MGR 8.0 (target):** +```bash +# Always use -c mysql for MGR +kubectl exec -0 -n -c mysql -- \ + mysql -uroot -p -e "SQL_HERE" +``` + +**Important Notes:** +- Always use the parameter order: `kubectl exec -n -- ` +- Use `--` (double dash) before the command to separate kubectl options from the command +- Use `\` (backslash) for multi-line commands +- Avoid heredocs (`<` before the pod name to avoid parsing issues + +## Execution Guide + +This guide uses the automated migration scripts provided in the [Appendix](#appendix-migration-scripts-reference) to simplify the migration process. + +### Step 1: Schema Compatibility Analysis + +Perform this analysis **one week before** the planned migration. + +Run the `00-pre-migration-check.sh` script to automatically detect schema compatibility issues and identify databases to migrate. + +```bash +# Edit configuration +vi 00-pre-migration-check.sh + +# Run check +chmod +x 00-pre-migration-check.sh +./00-pre-migration-check.sh +``` + +The script will output: +1. List of user databases to migrate (copy the `DATABASES="..."` line for later) +2. Schema compatibility issues (Reserved keywords, Invalid dates, ZEROFILL, etc.) +3. Character set analysis + +If the script reports issues, use the commands below to fix them. + +#### Fix Schema Issues + +```bash +# Fix reserved keyword columns (example) +kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + USE db1; + ALTER TABLE users CHANGE COLUMN rank user_rank INT; + " + +# Fix invalid date defaults (example) +kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + USE db1; + ALTER TABLE events MODIFY COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP; + " + +# Fix ZEROFILL columns (remove ZEROFILL) +kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + USE db1; + ALTER TABLE products MODIFY COLUMN price DECIMAL(10,2); + " +``` + +### Step 2: Character Set and Collation Analysis + +The `00-pre-migration-check.sh` script (run in Step 1) already checks for non-utf8mb4 tables. If any "tables not using utf8mb4" are reported, convert them **3-5 days before** the planned migration. + +#### Convert to utf8mb4 + +```bash +# Convert databases to utf8mb4 +for db in ${DATABASES}; do + kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + ALTER DATABASE ${db} CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci; + " +done + +# Convert tables to utf8mb4 +for db in ${DATABASES}; do + TABLES=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e " + SELECT TABLE_NAME FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_TYPE = 'BASE TABLE'; + ") + + for table in ${TABLES}; do + echo "Converting ${db}.${table}..." + kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} ${db} -e " + ALTER TABLE ${table} CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci; + " + done +done +``` + +**Important Note**: For tables with long VARCHAR/TEXT indexes (>191 characters), you may need to adjust index lengths: + +```sql +-- Example: Fix index length for utf8mb4 +ALTER TABLE users DROP INDEX idx_email; +ALTER TABLE users ADD UNIQUE INDEX idx_email (email(191)); +``` + +### Step 3: Create Target MySQL 8.0 Instance + +Create the target MySQL 8.0 instance **shortly before** the data migration phase to conserve resources. + +**Important**: Create the target MySQL 8.0 instance before starting the migration script. + +**Using Web Console:** + +Refer to the [Create MySQL Instance documentation](https://docs.alauda.io/mysql-mgr/4.2/functions/01-create.html) for detailed instructions. Key configuration points: + +1. Select version **8.0** +2. Configure resources (recommend **+10-20% memory** over source cluster due to MySQL 8.0 overhead) +3. Set storage size to **2-3x** source database size + +**Using Command Line:** + +```bash +TARGET_NAME="mysql-8-target" +NAMESPACE="your-namespace" +STORAGE_SIZE="500Gi" # Adjust based on your source DB size + +cat << EOF | kubectl -n $NAMESPACE apply -f - +apiVersion: middleware.alauda.io/v1 +kind: Mysql +metadata: + name: $TARGET_NAME + namespace: $NAMESPACE + labels: + mysql/arch: mgr +spec: + mgr: + enableStorage: true + image: {} + members: 1 + monitor: + enable: true + exporter: {} + resources: + server: + limits: + cpu: "2" + memory: 4Gi + requests: + cpu: "2" + memory: 4Gi + router: + replicas: 1 + resources: + limits: + cpu: 500m + memory: 512Mi + requests: + cpu: 500m + memory: 512Mi + svcRO: + type: ClusterIP + svcRW: + type: ClusterIP + strictSecurityModeEnabled: true + upgradeOption: {} + volumeClaimTemplate: + metadata: {} + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: ${STORAGE_SIZE} + storageClassName: dataservice-topolvmsc + status: {} + params: + mysql: {} + router: + DEFAULT: + max_total_connections: "200" + logger: + level: info + upgradeOption: + autoUpgrade: false + crVersion: 4.2.0 + version: "8.0" +EOF +``` + +**Note:** The YAML above uses the Alauda MySQL CRD format. Key differences from standard Kubernetes: +- Use `spec.mgr` instead of `spec.type` +- `members: 1` for single-node (increase to 3 for HA) +- `storageClassName` must match your cluster's available StorageClass +- `strictSecurityModeEnabled: true` is required for most ACP environments + +**Verify target cluster:** + +```bash +# Wait for cluster to be ready +kubectl -n $NAMESPACE get mysql $TARGET_NAME -w + +# Expected output: +# NAME VERSION STATE PXCSTATE MGRSTATE +# mysql-8-target 8.0 Ready ready +``` + +### Step 4: Migrate Data, Users, and Privileges + +Use the `01-migrate-all.sh` script to execute the migration. This script: +1. Verifies prerequisites (GTID, versions, connectivity) +2. Streams data for all specified databases directly from source to target +3. Migrates user accounts and privileges (using `mysql_native_password` for compatibility) + +**Procedure:** + +1. **Stop Application Writes**: Scale the application to zero replicas to ensure data consistency. + + **Critical**: The application must remain stopped (or strictly read-only) from this point until the Cutover Phase is complete. Any data written to the source database after this step will be lost. + + ```bash + kubectl scale deployment --replicas=0 -n + ``` + +2. **Configure Script**: + Edit `01-migrate-all.sh` and set your cluster names, namespaces, and the `DATABASES` variable (using the list from Step 1). + +3. **Run Migration**: + ```bash + chmod +x 01-migrate-all.sh + ./01-migrate-all.sh + ``` + +**Important Notes:** +- The script uses **streaming migration**, so no disk space is consumed for dump files. +- It automatically handles `GTID_PURGED` filtering for MGR compatibility. +- User accounts are migrated with `mysql_native_password` to maximize compatibility with existing applications. + + +### Step 5: Verify Migration + +Run the `02-verify-migration.sh` script to confirm that all database objects have been migrated successfully. + +```bash +chmod +x 02-verify-migration.sh +./02-verify-migration.sh +``` + +The script performs the following checks for each database: +1. **Tables**: Compares count on source vs target +2. **Views**: Compares count and tests execution of every view +3. **Stored Procedures/Functions**: Compares counts +4. **Triggers/Events**: Compares counts +5. **Row Counts**: Performs sample row count checks +6. **Users**: Verifies user accounts were migrated + +**Note**: If any check fails, the script will output a red failure message. Do not proceed to cutover until verification passes. + +### Step 6: Post-Migration Optimization + +Optimize the target MySQL 8.0 instance following a successful migration. + +#### 1. Update Table Statistics + +```bash +TARGET_NAME="mysql-8-target" +TARGET_NAMESPACE="your-namespace" +MYSQL_PASSWORD="your-password" +DATABASES="db1 db2 db3" # ← YOUR databases only (NOT: information_schema, mysql, performance_schema, sys) + +for db in ${DATABASES}; do + echo "Analyzing tables in ${db}..." + TABLES=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e " + SELECT TABLE_NAME FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_TYPE = 'BASE TABLE'; + ") + + for table in ${TABLES}; do + kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} ${db} -e "ANALYZE TABLE ${table};" 2>&1 | grep -v "Table" + done + + echo " ✓ Analyzed $(echo ${TABLES} | wc -w) tables" +done +``` + +#### 2. Create Histograms (MySQL 8.0 Feature) + +Histograms improve query performance for non-indexed columns: + +```bash +# Example: Create histogram on frequently filtered column +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + ANALYZE TABLE db1.orders UPDATE HISTOGRAM ON customer_id, status WITH 100 BUCKETS; + " +``` + +#### 3. Check Fragmentation + +```bash +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + SELECT TABLE_SCHEMA, TABLE_NAME, + ROUND(DATA_FREE / 1024 / 1024, 2) AS 'Fragmentation (MB)' + FROM information_schema.TABLES + WHERE TABLE_SCHEMA IN ('${DATABASES// /,\'','\'}') + AND DATA_FREE > 0 + ORDER BY DATA_FREE DESC; + " +``` + +If significant fragmentation found (>100MB), rebuild tables: + +```sql +-- Rebuild fragmented table +OPTIMIZE TABLE db1.orders; +``` + +#### 4. Create Performance Baseline + +```bash +# Record current performance metrics (table count, row count, size) to /tmp/mysql-8-baseline.txt for later comparison +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + SELECT NOW() AS baseline_date, + COUNT(*) AS total_tables, + SUM(TABLE_ROWS) AS total_rows, + ROUND(SUM(DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024 / 1024, 2) AS total_size_gb + FROM information_schema.TABLES + WHERE TABLE_SCHEMA IN ('${DATABASES// /,\'','\'}'); + " > /tmp/mysql-8-baseline.txt +``` + +## Cutover Phase + +### Step 7: Application Cutover + +After migration verification is complete, switch over application traffic: + +#### 1. Verify Application is Stopped + +Ensure the application is still stopped (as performed in Step 4). + +```bash +# Ensure application is scaled down +kubectl scale deployment --replicas=0 -n + +# Verify no active connections +kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e "SHOW PROCESSLIST;" | grep -v "Sleep" +``` + +#### 2. Update Application Connection String + +```bash +# Update ConfigMap or environment variables +kubectl patch configmap -n --type=json \ + -p='[{"op": "replace", "path": "/data/database-host", "value":"mysql-8-target-read-write.'${TARGET_NAMESPACE}'.svc.cluster.local"}]' + +kubectl patch configmap -n --type=json \ + -p='[{"op": "replace", "path": "/data/database-port", "value":"3306"}]' +``` + +#### 3. Restart Application + +```bash +# Scale up application +kubectl scale deployment --replicas= -n + +# Wait for pods to be ready +kubectl -n rollout status deployment +``` + +#### 4. Verify Application Functionality + +```bash +# Test database connectivity from application pod +kubectl exec -it -n -- \ + mysql -h mysql-8-target-read-write.${TARGET_NAMESPACE}.svc.cluster.local \ + -uroot -p${MYSQL_PASSWORD} -e "SELECT 1 AS test;" + +# Check application logs for errors +kubectl logs -n --tail=100 | grep -i error +``` + +### Monitoring + +Monitor the migrated instance for 24-48 hours: + +```bash +# Check MySQL 8.0 instance health +kubectl -n ${TARGET_NAMESPACE} get mysql ${TARGET_NAME} -w + +# Monitor error logs +kubectl logs -n ${TARGET_NAMESPACE} ${TARGET_NAME}-0 -c mysql --tail=100 -f + +# Check replication status (if applicable) +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e "SHOW SLAVE STATUS\G" +``` + +## Disaster Recovery + +### Rollback Plan + +If critical issues are discovered after the cutover: + +```bash +# 1. Stop application +kubectl scale deployment --replicas=0 -n + +# 2. Update connection string back to source +kubectl patch configmap -n --type=json \ + -p='[{"op": "replace", "path": "/data/database-host", "value":"'${SOURCE_NAME}'-proxysql.'${SOURCE_NAMESPACE}'.svc.cluster.local"}]' + +# 3. Restart application +kubectl scale deployment --replicas= -n + +# 4. Verify connectivity +kubectl exec -it -n -- \ + mysql -h ${SOURCE_NAME}-proxysql.${SOURCE_NAMESPACE}.svc.cluster.local \ + -uroot -p${MYSQL_PASSWORD} -e "SELECT 1 AS test;" + +# 5. Monitor application logs +kubectl logs -n --tail=100 -f +``` + +### Common Issues and Solutions + +#### Issue: GTID_PURGED Error + +**Symptoms:** +```text +ERROR 3546 (HY000) at line XX: Cannot update GTID_PURGED with the Group Replication plugin running +``` + +**Solution:** Already handled in migration procedure by filtering with `grep -v "SET @@GLOBAL.GTID_PURGED"` + +#### Issue: Character Set Conversion Errors + +**Symptoms:** +```text +ERROR 1366 (HY000): Incorrect string value +``` + +**Solution:** +```bash +# Check current character set +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + SELECT TABLE_SCHEMA, TABLE_NAME, TABLE_COLLATION + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_COLLATION NOT LIKE 'utf8mb4%'; + " + +# Convert to utf8mb4 +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + ALTER DATABASE ${db} CHARACTER SET = utf8mb4 COLLATE = utf8mb4_unicode_ci; + " +``` + +#### Issue: DEFINER Privilege Errors + +**Symptoms:** +```text +ERROR 1449 (HY000): The user specified as a definer ('user'@'host') does not exist +``` + +**Solution:** +```bash +# Find all objects with missing definers +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + SELECT DISTINCT DEFINER + FROM information_schema.VIEWS + WHERE TABLE_SCHEMA = '${db}' + AND DEFINER NOT IN (SELECT CONCAT(user, '@', host) FROM mysql.user); + " + +# Recreate missing users or update DEFINER +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + ALTER VIEW db1.my_view SQL SECURITY INVOKER AS SELECT ...; + " +``` + +#### Issue: Authentication Plugin Errors + +**Symptoms:** +```text +ERROR 2059 (HY000): Authentication plugin 'caching_sha2_password' cannot be loaded +``` + +**Solution:** +```bash +# Update user to use mysql_native_password for compatibility +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e " + ALTER USER 'app_user'@'%' IDENTIFIED WITH mysql_native_password BY 'password'; + FLUSH PRIVILEGES; + " +``` + +## Troubleshooting + +### Diagnostic Commands + +#### Check Migration Progress + +```bash +# Monitor migration progress (streaming mode) +kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e "SHOW PROCESSLIST;" + +# Monitor network traffic (if migration is slow) +kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -e "SHOW PROCESSLIST;" +``` + +#### Verify Data Integrity + +```bash +# Compare row counts for all tables +for db in ${DATABASES}; do + echo "=== Database: ${db} ===" + kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e " + SELECT TABLE_NAME, TABLE_ROWS + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_TYPE = 'BASE TABLE' + ORDER BY TABLE_NAME; + " > /tmp/source_counts.txt + + kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e " + SELECT TABLE_NAME, TABLE_ROWS + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_TYPE = 'BASE TABLE' + ORDER BY TABLE_NAME; + " > /tmp/target_counts.txt + + diff /tmp/source_counts.txt /tmp/target_counts.txt || echo "Row count differences detected!" +done +``` + +#### Check MySQL 8.0 Error Logs + +```bash +# Real-time error monitoring +kubectl logs -n ${TARGET_NAMESPACE} ${TARGET_NAME}-0 -c mysql --tail=100 -f | grep -i error + +# Search for specific errors +kubectl logs -n ${TARGET_NAMESPACE} ${TARGET_NAME}-0 -c mysql --tail=1000 | grep -i "definer" +``` + +## Best Practices + +### Pre-Migration Planning + +- **Test in Staging**: Always perform a test migration in non-production environment first +- **Schema Cleanup**: Fix all schema compatibility issues before production migration +- **Character Set Migration**: Convert to utf8mb4 well in advance (at least 3-5 days before) +- **Backup Strategy**: Ensure recent backups are available before migration +- **Maintenance Window**: Schedule adequate downtime based on database size +- **Communication**: Notify all stakeholders including application teams and DBAs + +### During Migration + +- **Stop Application Writes**: Ensure no writes during export/import for consistency +- **Monitor Progress**: Track export/import progress at regular intervals +- **Verify Incrementally**: Run verification scripts after each major step +- **Document Issues**: Record any issues encountered for future reference +- **Keep Source Running**: Don't delete source until migration is verified + +### Post-Migration + +- **Comprehensive Testing**: Thoroughly test application functionality +- **Performance Monitoring**: Monitor query performance and resource utilization for 24-48 hours +- **Optimization**: Run post-migration optimization procedures +- **Keep Source for Rollback**: Maintain source cluster for 24-48 hours for rollback window +- **Update Documentation**: Update connection strings, runbooks, and monitoring dashboards + +## Reference + +### Size vs Time Estimates + +| Database Size | Export Time | Import Time | Total Downtime | +|---------------|-------------|-------------|----------------| +| < 10GB | 1-5 min | 2-10 min | 15-30 min | +| 10-50GB | 5-20 min | 10-30 min | 30-60 min | +| 50-100GB | 20-40 min | 30-60 min | 1-2 hours | +| 100-200GB | 40-80 min | 1-2 hours | 2-4 hours | + +### mysqldump Flags Reference + +| Flag | Purpose | +|------|---------| +| `--single-transaction` | Consistent snapshot using MVCC (InnoDB) | +| `--quick` | Retrieve rows one at a time (memory efficient) | +| `--lock-tables=false` | Don't lock tables (relies on single-transaction) | +| `--set-gtid-purged=ON` | Include GTID information | +| `--routines` | Export stored procedures and functions | +| `--events` | Export events | +| `--triggers` | Export triggers | +| `--databases` | Specify databases to export | + +### Verification Checklist + +After migration, verify: +- [ ] Same number of tables +- [ ] Same row counts per table +- [ ] Same number of views +- [ ] All views execute successfully +- [ ] Same number of procedures +- [ ] Same number of functions +- [ ] Same number of triggers +- [ ] Same number of events +- [ ] All DEFINER accounts exist +- [ ] All users migrated +- [ ] All grants migrated +- [ ] Application can connect +- [ ] Application functionality works + +### Useful Links + +- [Alauda MySQL MGR Documentation](https://docs.alauda.io/mysql-mgr/4.2/functions/01-create.html) +- [MySQL 8.0 Release Notes](https://dev.mysql.com/doc/refman/8.0/en/mysql-nutshell.html) +- [MySQL 8.0 Upgrade Guide](https://dev.mysql.com/doc/refman/8.0/en/upgrade-prerequisites.html) + +## Appendix: Migration Scripts Reference + +This section provides detailed documentation for the automated migration scripts designed to simplify the MySQL 5.7 to 8.0 migration process. + +### Overview + +The migration scripts provide a three-step automated approach: + +| Script | Purpose | When to Run | Duration | +|--------|---------|-------------|----------| +| **00-pre-migration-check.sh** | Pre-migration compatibility analysis | 1 week before migration | 2-5 minutes | +| **01-migrate-all.sh** | Complete migration (data + users) | During maintenance window | 15-60 minutes | +| **02-verify-migration.sh** | Comprehensive verification | After migration | 5-10 minutes | + +### Script 1: Pre-Migration Check + +**Purpose:** Detects schema compatibility issues and validates environment setup. + +**What It Checks:** +- Kubernetes cluster connectivity +- Source cluster health and status +- GTID mode enabled on source +- Auto-detects user databases +- Reserved keyword usage (RANK, GROUPS, FUNCTION, etc.) +- Invalid date defaults (`0000-00-00`) +- ZEROFILL column usage +- TEXT columns with DEFAULT values +- Character set compatibility (utf8mb4) + +**Configuration:** +```bash +SOURCE_NAME="source" # Source cluster name +SOURCE_NAMESPACE="your-namespace" # Source namespace +MYSQL_PASSWORD="your-password" # Source root password +DATABASES="ALL" # "ALL" to auto-detect +``` + +**Usage:** +```bash +vi 00-pre-migration-check.sh # Edit configuration +chmod +x 00-pre-migration-check.sh +./00-pre-migration-check.sh +``` + +**Expected Output:** +```text +======================================== +MySQL 5.7 to 8.0 Pre-Migration Check +======================================== + +>>> Checking kubectl context +✓ Connected to Kubernetes cluster + +>>> Checking source cluster +✓ Source cluster source found +✓ Source cluster status: ready + +>>> Checking GTID mode on source +✓ GTID mode is enabled + +>>> Detecting user databases +✓ Databases to migrate: + app_db customer_db reporting_db + +⚠ Copy this line for your migration script: +DATABASES="app_db customer_db reporting_db" + +>>> Checking for reserved keywords (MySQL 8.0) +✓ No reserved keyword issues found + +[... more checks ...] + +======================================== +Pre-Migration Check Summary +======================================== + +✓ Configuration verified: + Source cluster: source.your-namespace + Databases to migrate: app_db customer_db reporting_db + +Next steps: + 1. Fix any schema compatibility issues found above + 2. Convert character sets if needed + 3. Run script 01-migrate-all.sh to perform migration +``` + +### Script 2: Complete Migration + +**Purpose:** Migrates all databases, users, and privileges from source to target. + +**What It Does:** +- Validates prerequisites (both clusters, GTID, versions) +- Migrates databases using streaming (no disk storage required) +- Migrates user accounts with `mysql_native_password` +- Migrates all privileges and grants +- Performs basic verification + +**Configuration:** +```bash +SOURCE_NAME="source" +SOURCE_NAMESPACE="your-namespace" +SOURCE_MYSQL_PASSWORD="source-root-password" + +TARGET_NAME="mysql-8-target" +TARGET_NAMESPACE="your-namespace" +TARGET_MYSQL_PASSWORD="target-root-password" + +DATABASES="app_db customer_db reporting_db" # From pre-migration check +``` + +**Usage:** +```bash +# Before running: Stop application writes! +kubectl scale deployment --replicas=0 -n + +# Edit and run +vi 01-migrate-all.sh +chmod +x 01-migrate-all.sh +./01-migrate-all.sh +``` + +**Expected Output:** +```text +======================================== +MySQL 5.7 to 8.0 Migration +======================================== + +⚠ IMPORTANT: Ensure application writes are stopped during migration + +>>> Checking prerequisites +✓ Connected to Kubernetes cluster +✓ Source cluster found: source +✓ Target cluster found: mysql-8-target +✓ Target cluster version: 8.0.44 +✓ GTID mode enabled on source +ℹ Will migrate 3 database(s): app_db customer_db reporting_db + +======================================== +Migrating Databases +======================================== + +ℹ Migrating database [1/3]: app_db +✓ Migrated app_db + +ℹ Migrating database [2/3]: customer_db +✓ Migrated customer_db + +ℹ Migrating database [3/3]: reporting_db +✓ Migrated reporting_db + +✓ All databases migrated successfully (3/3) + +======================================== +Migrating Users and Privileges +======================================== + +>>> Creating user accounts +ℹ Found 5 user(s) to migrate +✓ User accounts created + +>>> Granting privileges +✓ Privileges granted + +>>> Verifying migrated users +✓ Migrated 5 user(s) + +[... verification ...] + +======================================== +Migration Summary +======================================== + +Source: source.your-namespace +Target: mysql-8-target.your-namespace +Databases migrated: 3/3 +Users migrated: 5 +Duration: 15m 32s + +✓ Migration completed successfully! + +Next steps: + 1. Run script 02-verify-migration.sh for comprehensive verification + 2. Update application connection strings + 3. Perform application testing + 4. Monitor for 24-48 hours before decommissioning source +``` + +### Script 3: Comprehensive Verification + +**Purpose:** Validates that all database objects migrated correctly. + +**What It Verifies:** +- Tables (count comparison) +- Views (count + execution test for each view) +- Stored Procedures (count) +- Stored Functions (count) +- Triggers (count) +- Events (count) +- Row counts (sample check on first 5 tables per database) +- User accounts (count + list) + +**Configuration:** +```bash +SOURCE_NAME="source" +SOURCE_NAMESPACE="your-namespace" +SOURCE_MYSQL_PASSWORD="source-root-password" + +TARGET_NAME="mysql-8-target" +TARGET_NAMESPACE="your-namespace" +TARGET_MYSQL_PASSWORD="target-root-password" + +DATABASES="app_db customer_db reporting_db" # Same as migration +``` + +**Usage:** +```bash +vi 02-verify-migration.sh +chmod +x 02-verify-migration.sh +./02-verify-migration.sh +``` + +**Expected Output:** +```text +======================================== +MySQL 5.7 to 8.0 Migration Verification +======================================== + +>>> Verifying Tables + +Database: app_db +✓ Tables: 15 (match) + +Database: customer_db +✓ Tables: 8 (match) + +[... more verifications ...] + +======================================== +Verification Summary +======================================== + +Total checks: 42 +Passed: 42 +Failed: 0 + +✓ ALL CHECKS PASSED! + +Migration verification successful. Next steps: + 1. Update application connection strings to point to target + 2. Perform application testing + 3. Monitor target cluster for 24-48 hours + 4. Keep source cluster available for rollback during this period +``` + +### Getting Passwords + +**Source Cluster (PXC 5.7):** +```bash +kubectl get secret -n -o jsonpath='{.data.root}' | base64 -d +``` + +**Target Cluster (MGR 8.0):** +```bash +kubectl get secret mgr--password -n -o jsonpath='{.data.root}' | base64 -d +``` + +### Troubleshooting Scripts + +#### Script fails with "Cannot connect to Kubernetes cluster" +```bash +kubectl config current-context +kubectl cluster-info +``` + +#### Script fails with "Source cluster not found" +```bash +kubectl get mysql -n +``` + +#### Migration fails for specific database +```bash +# Check target logs +kubectl logs -n -0 -c mysql --tail=100 + +# Manually test single database migration +kubectl exec -pxc-0 -n -- \ + mysqldump -uroot -p --single-transaction --quick \ + --lock-tables=false --set-gtid-purged=ON --routines --events --triggers \ + --databases 2>/dev/null | \ + grep -v "SET @@GLOBAL.GTID_PURGED" | \ + kubectl exec -i -0 -n -c mysql -- \ + mysql -uroot -p +``` + +### Full Workflow Example + +```bash +# ===== 1 WEEK BEFORE MIGRATION ===== +./00-pre-migration-check.sh +# → Output shows: DATABASES="app_db customer_db reporting_db" +# → Fix any schema issues found +# → Convert to utf8mb4 if needed + +# ===== DAY OF MIGRATION (Maintenance Window) ===== + +# Stop application writes +kubectl scale deployment --replicas=0 -n + +# Update migration script with DATABASES from pre-check +vi 01-migrate-all.sh +# DATABASES="app_db customer_db reporting_db" + +# Run migration +./01-migrate-all.sh + +# Run verification +./02-verify-migration.sh + +# Update application connection string to target +kubectl patch configmap -n --type=json \ + -p='[{"op": "replace", "path": "/data/db-host", "value":"mysql-8-target-read-write.namespace.svc.cluster.local"}]' + +# Restart application +kubectl scale deployment --replicas=3 -n + +# Wait for pods ready +kubectl -n rollout status deployment + +# Test application +curl http:///health + +# Monitor for 24-48 hours +kubectl logs -n mysql-8-target-0 -c mysql --tail=100 -f + +# ===== AFTER SUCCESSFUL TESTING (24-48 hours later) ===== +# Decommission source cluster +kubectl delete mysql -n +``` + +### Script Features + +All scripts include: + +- ✅ **Color-coded output**: Green (success), Red (error), Yellow (warning), Blue (info) +- ✅ **Progress indicators**: Shows current step and overall progress +- ✅ **Error handling**: Exits on critical errors with clear messages +- ✅ **Automatic detection**: Auto-discovers databases when `DATABASES="ALL"` +- ✅ **Comprehensive checks**: Validates all prerequisites before proceeding +- ✅ **Detailed output**: Shows exactly what was migrated and verified +- ✅ **Minimal configuration**: Only 4-6 variables to configure per script + +### Important Notes + +1. **DO NOT Include System Databases**: The `DATABASES` variable must contain ONLY user/application databases. Do NOT include: `information_schema`, `mysql`, `performance_schema`, `sys`. + +2. **Stop Application Writes**: Ensure no application writes during migration to maintain data consistency. + +3. **Keep Source Cluster**: Do not delete source cluster until after application testing and 24-48 hours of successful operation. + +4. **Test in Staging**: Always perform a test migration in non-production environment first. + +5. **Monitor Post-Migration**: Monitor target cluster for 24-48 hours before decommissioning source. + +### Script Compatibility + +- **MySQL Guide Version**: v2.5+ +- **Source**: PXC 5.7.44 +- **Target**: MGR 8.0.44 +- **Kubernetes**: Tested on Alauda Container Platform 4.2+ +- **Shell**: Bash 4.0+ + +### Script Source Code + +The following scripts can be copied directly from this document. Save each script to a file, make it executable, and run it. + +#### Script 1: 00-pre-migration-check.sh + +Save this script as `00-pre-migration-check.sh`: + +```bash +#!/bin/bash +#============================================================================= +# MySQL 5.7 to 8.0 Migration - Pre-Migration Check Script +#============================================================================= +# +# This script performs all pre-migration checks and fixes: +# 1. Schema compatibility analysis +# 2. Character set analysis +# 3. Database listing for migration +# +# Usage: +# 1. Edit the configuration section below +# 2. Run: chmod +x 00-pre-migration-check.sh +# 3. Run: ./00-pre-migration-check.sh +# +# Expected output: +# - List of any schema compatibility issues that need fixing +# - List of any character set conversions needed +# - List of databases to migrate (copy this for migration script) +# +#============================================================================= + +set -e # Exit on error + +#============================================================================= +# CONFIGURATION - EDIT THESE VALUES +#============================================================================= + +SOURCE_NAME="source" +SOURCE_NAMESPACE="your-namespace" +MYSQL_PASSWORD="your-password" + +# Set to "ALL" to auto-detect databases, or specify space-separated list +# DATABASES="ALL" # Auto-detect all user databases +# DATABASES="db1 db2 db3" # Or specify manually +DATABASES="ALL" + +# Color output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +#============================================================================= +# FUNCTIONS +#============================================================================= + +print_header() { + echo "" + echo "========================================" + echo "$1" + echo "========================================" +} + +print_section() { + echo "" + echo -e "${BLUE}>>> $1${NC}" +} + +print_success() { + echo -e "${GREEN}✓ $1${NC}" +} + +print_warning() { + echo -e "${YELLOW}⚠ $1${NC}" +} + +print_error() { + echo -e "${RED}✗ $1${NC}" +} + +check_kubectl_context() { + print_section "Checking kubectl context" + + if ! kubectl cluster-info &>/dev/null; then + print_error "Cannot connect to Kubernetes cluster" + exit 1 + fi + print_success "Connected to Kubernetes cluster" +} + +check_source_cluster() { + print_section "Checking source cluster" + + if ! kubectl get mysql ${SOURCE_NAME} -n ${SOURCE_NAMESPACE} &>/dev/null; then + print_error "Source cluster ${SOURCE_NAME} not found in namespace ${SOURCE_NAMESPACE}" + exit 1 + fi + print_success "Source cluster ${SOURCE_NAME} found" + + # Check cluster status + STATUS=$(kubectl get mysql ${SOURCE_NAME} -n ${SOURCE_NAMESPACE} -o jsonpath='{.status.state}') + if [ "${STATUS}" != "ready" ]; then + print_warning "Source cluster status: ${STATUS} (expected: ready)" + else + print_success "Source cluster status: ready" + fi +} + +check_gtid_enabled() { + print_section "Checking GTID mode on source" + + GTID_MODE=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e "SELECT @@gtid_mode" 2>/dev/null | grep -v "Warning") + + if [ "${GTID_MODE}" = "ON" ]; then + print_success "GTID mode is enabled" + else + print_error "GTID mode is NOT enabled (required for migration)" + exit 1 + fi +} + +detect_databases() { + print_section "Detecting user databases" + + if [ "${DATABASES}" = "ALL" ]; then + DATABASES=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e "SHOW DATABASES" 2>/dev/null | \ + grep -v -E "^(information_schema|mysql|performance_schema|sys)$" | \ + tr '\n' ' ' | sed 's/ $//') + + if [ -z "${DATABASES}" ]; then + print_error "No user databases found" + exit 1 + fi + fi + + print_success "Databases to migrate:" + echo " ${DATABASES}" + echo "" + print_warning "Copy this line for your migration script:" + echo -e "${GREEN}DATABASES=\"${DATABASES}\"${NC}" +} + +check_reserved_keywords() { + print_section "Checking for reserved keywords (MySQL 8.0)" + + ISSUES=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e " + SELECT CONCAT(TABLE_SCHEMA, '.', TABLE_NAME, '.', COLUMN_NAME) + FROM information_schema.COLUMNS + WHERE COLUMN_NAME IN ('RANK', 'GROUPS', 'FUNCTION', 'SYSTEM', 'RELOAD', + 'ARRAY', 'OFFSET', 'CUBE', 'ROLE', 'VALUES') + AND TABLE_SCHEMA NOT IN ('mysql', 'information_schema', 'performance_schema', 'sys') + AND TABLE_SCHEMA IN ('${DATABASES// /,\'','\'}') + ORDER BY TABLE_SCHEMA, TABLE_NAME; + " 2>/dev/null | grep -v "Warning") + + if [ -z "${ISSUES}" ]; then + print_success "No reserved keyword issues found" + else + print_error "Found columns using MySQL 8.0 reserved keywords:" + echo "${ISSUES}" | while read line; do + echo " - ${line}" + done + echo "" + print_warning "These columns must be renamed before migration" + echo "Example fix:" + echo " ALTER TABLE employees CHANGE COLUMN rank employee_rank INT;" + fi +} + +check_invalid_dates() { + print_section "Checking for invalid date defaults" + + ISSUES=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e " + SELECT CONCAT(TABLE_SCHEMA, '.', TABLE_NAME, '.', COLUMN_NAME) + FROM information_schema.COLUMNS + WHERE DATA_TYPE IN ('date', 'datetime', 'timestamp') + AND COLUMN_DEFAULT LIKE '0000-00-00%' + AND TABLE_SCHEMA IN ('${DATABASES// /,\'','\'}') + ORDER BY TABLE_SCHEMA, TABLE_NAME; + " 2>/dev/null | grep -v "Warning") + + if [ -z "${ISSUES}" ]; then + print_success "No invalid date defaults found" + else + print_error "Found columns with invalid date defaults:" + echo "${ISSUES}" | while read line; do + echo " - ${line}" + done + echo "" + print_warning "These columns must be fixed before migration" + echo "Example fix:" + echo " ALTER TABLE events MODIFY COLUMN created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP;" + fi +} + +check_zerofill() { + print_section "Checking for ZEROFILL usage" + + ISSUES=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e " + SELECT CONCAT(TABLE_SCHEMA, '.', TABLE_NAME, '.', COLUMN_NAME, ' ', COLUMN_TYPE) + FROM information_schema.COLUMNS + WHERE COLUMN_TYPE LIKE '%ZEROFILL%' + AND TABLE_SCHEMA IN ('${DATABASES// /,\'','\'}') + ORDER BY TABLE_SCHEMA, TABLE_NAME; + " 2>/dev/null | grep -v "Warning") + + if [ -z "${ISSUES}" ]; then + print_success "No ZEROFILL usage found" + else + print_warning "Found ZEROFILL columns (deprecated in MySQL 8.0):" + echo "${ISSUES}" | while read line; do + echo " - ${line}" + done + echo "" + print_warning "ZEROFILL will be removed during migration" + echo "To fix manually:" + echo " ALTER TABLE products MODIFY COLUMN price DECIMAL(10,2);" + fi +} + +check_text_defaults() { + print_section "Checking for TEXT columns with DEFAULT values" + + ISSUES=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e " + SELECT CONCAT(TABLE_SCHEMA, '.', TABLE_NAME, '.', COLUMN_NAME) + FROM information_schema.COLUMNS + WHERE DATA_TYPE IN ('text', 'tinytext', 'mediumtext', 'longtext') + AND COLUMN_DEFAULT IS NOT NULL + AND TABLE_SCHEMA IN ('${DATABASES// /,\'','\'}') + ORDER BY TABLE_SCHEMA, TABLE_NAME; + " 2>/dev/null | grep -v "Warning") + + if [ -z "${ISSUES}" ]; then + print_success "No TEXT columns with DEFAULT values found" + else + print_error "Found TEXT columns with DEFAULT values (not allowed in MySQL 8.0):" + echo "${ISSUES}" | while read line; do + echo " - ${line}" + done + echo "" + print_warning "These DEFAULT values must be removed before migration" + fi +} + +check_character_sets() { + print_section "Checking character sets" + + NON_UTF8=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e " + SELECT CONCAT(TABLE_SCHEMA, '.', TABLE_NAME, ' - ', TABLE_COLLATION) + FROM information_schema.TABLES + WHERE TABLE_SCHEMA IN ('${DATABASES// /,\'','\'}') + AND TABLE_COLLATION NOT LIKE 'utf8mb4%' + AND TABLE_TYPE = 'BASE TABLE' + ORDER BY TABLE_SCHEMA, TABLE_NAME; + " 2>/dev/null | grep -v "Warning") + + if [ -z "${NON_UTF8}" ]; then + print_success "All tables are using utf8mb4" + else + print_warning "Found tables not using utf8mb4:" + echo "${NON_UTF8}" | while read line; do + echo " - ${line}" + done + echo "" + print_warning "Consider converting to utf8mb4 before migration" + echo "See section 'Character Set and Collation Analysis' in the documentation" + fi +} + +check_lower_case_table_names() { + print_section "Checking lower_case_table_names" + + LCTN=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${MYSQL_PASSWORD} -N -e "SELECT @@lower_case_table_names" 2>/dev/null | grep -v "Warning") + + if [ "${LCTN}" = "1" ]; then + print_warning "Source cluster has lower_case_table_names=1" + echo " Ensure target MySQL 8.0 cluster is also configured with lower_case_table_names=1" + echo " This setting cannot be changed after initialization in MySQL 8.0." + else + print_success "Source cluster has lower_case_table_names=${LCTN}" + fi +} + +print_summary() { + print_header "Pre-Migration Check Summary" + + echo "" + print_success "Configuration verified:" + echo " Source cluster: ${SOURCE_NAME}.${SOURCE_NAMESPACE}" + echo " Databases to migrate: ${DATABASES}" + echo "" + + echo "Next steps:" + echo " 1. Fix any schema compatibility issues found above" + echo " 2. Convert character sets if needed" + echo " 3. Run script 01-migrate-all.sh to perform migration" + echo "" +} + +#============================================================================= +# MAIN EXECUTION +#============================================================================= + +main() { + print_header "MySQL 5.7 to 8.0 Pre-Migration Check" + + check_kubectl_context + check_source_cluster + check_gtid_enabled + detect_databases + check_reserved_keywords + check_invalid_dates + check_zerofill + check_text_defaults + check_character_sets + check_lower_case_table_names + print_summary + + print_success "Pre-migration check completed" +} + +main +``` + +#### Script 2: 01-migrate-all.sh + +Save this script as `01-migrate-all.sh`: + +```bash +#!/bin/bash +#============================================================================= +# MySQL 5.7 to 8.0 Migration - Complete Migration Script +#============================================================================= +# +# This script performs the complete migration from MySQL 5.7 to 8.0: +# 1. Migrates all databases (streaming, no intermediate storage) +# 2. Migrates users and privileges +# 3. Performs basic verification +# +# Prerequisites: +# - Target MySQL 8.0 cluster must be created and ready +# - Pre-migration check should have been completed +# - Application writes should be stopped during migration +# +# Usage: +# 1. Edit the configuration section below +# 2. Run: chmod +x 01-migrate-all.sh +# 3. Run: ./01-migrate-all.sh +# +# Estimated downtime: 15-60 minutes depending on database size +# +#============================================================================= + +set -e # Exit on error + +#============================================================================= +# CONFIGURATION - EDIT THESE VALUES +#============================================================================= + +SOURCE_NAME="source" +SOURCE_NAMESPACE="your-namespace" +SOURCE_MYSQL_PASSWORD="source-root-password" + +TARGET_NAME="mysql-8-target" +TARGET_NAMESPACE="your-namespace" +TARGET_MYSQL_PASSWORD="target-root-password" + +# IMPORTANT: databases to migrate (DO NOT include: information_schema, mysql, performance_schema, sys) +DATABASES="db1 db2 db3" # ← Copy from pre-migration check output + +# Users to exclude from migration (system users) +EXCLUDE_USERS="'mysql.sys', 'mysql.session', 'mysql.infoschema', 'root', 'clustercheck', 'monitor', 'operator', 'xtrabackup', 'repl'" + +# Color output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +CYAN='\033[0;36m' +NC='\033[0m' # No Color + +# Statistics +TOTAL_DATABASES=0 +MIGRATED_DATABASES=0 +FAILED_DATABASES=0 +START_TIME=$(date +%s) + +#============================================================================= +# FUNCTIONS +#============================================================================= + +print_header() { + echo "" + echo "========================================" + echo "$1" + echo "========================================" +} + +print_section() { + echo "" + echo -e "${BLUE}>>> $1${NC}" +} + +print_success() { + echo -e "${GREEN}✓ $1${NC}" +} + +print_error() { + echo -e "${RED}✗ $1${NC}" +} + +print_warning() { + echo -e "${YELLOW}⚠ $1${NC}" +} + +print_info() { + echo -e "${CYAN}ℹ $1${NC}" +} + +check_prerequisites() { + print_section "Checking prerequisites" + + # Check kubectl + if ! kubectl cluster-info &>/dev/null; then + print_error "Cannot connect to Kubernetes cluster" + exit 1 + fi + print_success "Connected to Kubernetes cluster" + + # Check source cluster + if ! kubectl get mysql ${SOURCE_NAME} -n ${SOURCE_NAMESPACE} &>/dev/null; then + print_error "Source cluster ${SOURCE_NAME} not found in namespace ${SOURCE_NAMESPACE}" + exit 1 + fi + print_success "Source cluster found: ${SOURCE_NAME}" + + # Check target cluster + if ! kubectl get mysql ${TARGET_NAME} -n ${TARGET_NAMESPACE} &>/dev/null; then + print_error "Target cluster ${TARGET_NAME} not found in namespace ${TARGET_NAMESPACE}" + print_error "Please create the target cluster before running migration" + exit 1 + fi + print_success "Target cluster found: ${TARGET_NAME}" + + # Check target is MySQL 8.0 + TARGET_VERSION=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e "SELECT VERSION();" 2>/dev/null | grep -v "Warning") + + if [[ ! "${TARGET_VERSION}" =~ ^8\.0\. ]]; then + print_error "Target cluster is not MySQL 8.0 (version: ${TARGET_VERSION})" + exit 1 + fi + print_success "Target cluster version: ${TARGET_VERSION}" + + # Check GTID on source + GTID_MODE=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e "SELECT @@gtid_mode" 2>/dev/null | grep -v "Warning") + + if [ "${GTID_MODE}" != "ON" ]; then + print_error "GTID mode is not enabled on source (required for migration)" + exit 1 + fi + print_success "GTID mode enabled on source" + + # Count databases + TOTAL_DATABASES=$(echo ${DATABASES} | wc -w) + print_info "Will migrate ${TOTAL_DATABASES} database(s): ${DATABASES}" +} + +migrate_databases() { + print_header "Migrating Databases" + + local db_num=0 + + for db in ${DATABASES}; do + db_num=$((db_num + 1)) + echo "" + print_info "Migrating database [${db_num}/${TOTAL_DATABASES}]: ${db}" + + # Migrate using streaming (no intermediate storage) + kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysqldump -uroot -p${SOURCE_MYSQL_PASSWORD} \ + --single-transaction \ + --quick \ + --lock-tables=false \ + --set-gtid-purged=ON \ + --routines \ + --events \ + --triggers \ + --databases ${db} \ + 2>/dev/null | + grep -v "SET @@GLOBAL.GTID_PURGED" | + kubectl exec -i ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} --init-command="SET FOREIGN_KEY_CHECKS=0;" 2>&1 | grep -v "Using a password" || true + + # Verify migration succeeded by checking if database exists on target + DB_EXISTS=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.SCHEMATA + WHERE SCHEMA_NAME = '${db}'; + " 2>/dev/null | grep -v "Warning") + + if [ "${DB_EXISTS}" = "1" ]; then + + print_success "Migrated ${db}" + MIGRATED_DATABASES=$((MIGRATED_DATABASES + 1)) + else + print_error "Failed to migrate ${db}" + FAILED_DATABASES=$((FAILED_DATABASES + 1)) + fi + done + + echo "" + if [ ${MIGRATED_DATABASES} -eq ${TOTAL_DATABASES} ]; then + print_success "All databases migrated successfully (${MIGRATED_DATABASES}/${TOTAL_DATABASES})" + else + print_error "Some databases failed to migrate (${MIGRATED_DATABASES}/${TOTAL_DATABASES} succeeded, ${FAILED_DATABASES} failed)" + fi +} + +migrate_users() { + print_header "Migrating Users and Privileges" + + print_section "Creating user accounts" + + # Stream CREATE USER statements + USER_COUNT=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM mysql.user + WHERE user NOT IN (${EXCLUDE_USERS}); + " 2>/dev/null | grep -v "Warning") + + print_info "Found ${USER_COUNT} user(s) to migrate" + + # Create users (ignore grep exit code) + kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT CONCAT('CREATE USER IF NOT EXISTS ''', user, '''@''', host, ''' IDENTIFIED WITH mysql_native_password AS ''', replace(authentication_string, '\'', '\'\''), ''';') + FROM mysql.user + WHERE user NOT IN (${EXCLUDE_USERS}); + " 2>/dev/null | grep -v "^Warning" | + kubectl exec -i ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} 2>&1 | grep -v "Using a password" || true + + # Verify user creation + USER_COUNT_AFTER=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM mysql.user + WHERE user NOT IN (${EXCLUDE_USERS}); + " 2>/dev/null | grep -v "Warning") + + if [ "${USER_COUNT_AFTER}" -ge "${USER_COUNT}" ]; then + print_success "User accounts created" + else + print_error "Failed to create user accounts" + fi + + print_section "Granting privileges" + + # Stream GRANT statements (ignore grep exit code) + kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT CONCAT('SHOW GRANTS FOR ''', user, '''@''', host, ''';') + FROM mysql.user + WHERE user NOT IN (${EXCLUDE_USERS}); + " 2>/dev/null | grep -v "^Warning" | while read query; do + kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -e "${query}" 2>/dev/null | grep "^GRANT" | sed 's/$/;/' + done | + kubectl exec -i ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} 2>&1 | grep -v "Using a password" || true + + print_success "Privileges granted" + + # Flush privileges + kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -e "FLUSH PRIVILEGES;" 2>&1 | grep -v "Using a password" >/dev/null || true + + print_section "Verifying migrated users" + + MIGRATED_USERS=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM mysql.user + WHERE user NOT IN (${EXCLUDE_USERS}); + " 2>/dev/null | grep -v "Warning") + + print_success "Migrated ${MIGRATED_USERS} user(s)" +} + +verify_migration() { + print_header "Migration Verification" + + print_section "Verifying databases" + + for db in ${DATABASES}; do + # Check if database exists + DB_EXISTS=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.SCHEMATA + WHERE SCHEMA_NAME = '${db}'; + " 2>/dev/null | grep -v "Warning") + + if [ "${DB_EXISTS}" = "1" ]; then + # Count tables + TABLE_COUNT=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_TYPE = 'BASE TABLE'; + " 2>/dev/null | grep -v "Warning") + + print_success "${db}: ${TABLE_COUNT} table(s) migrated" + else + print_error "${db}: Database not found on target" + fi + done + + print_section "Verifying users" + + MIGRATED_USERS=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT CONCAT(user, '@', host) + FROM mysql.user + WHERE user NOT IN (${EXCLUDE_USERS}) + ORDER BY user; + " 2>/dev/null | grep -v "Warning") + + if [ -n "${MIGRATED_USERS}" ]; then + print_success "Migrated users:" + echo "${MIGRATED_USERS}" | while read user; do + echo " - ${user}" + done + else + print_warning "No users migrated (or all were excluded)" + fi +} + +print_summary() { + local END_TIME=$(date +%s) + local DURATION=$((END_TIME - START_TIME)) + local MINUTES=$((DURATION / 60)) + local SECONDS=$((DURATION % 60)) + + print_header "Migration Summary" + + echo "" + echo "Source: ${SOURCE_NAME}.${SOURCE_NAMESPACE}" + echo "Target: ${TARGET_NAME}.${TARGET_NAMESPACE}" + echo "Databases migrated: ${MIGRATED_DATABASES}/${TOTAL_DATABASES}" + echo "Users migrated: ${MIGRATED_USERS}" + echo "Duration: ${MINUTES}m ${SECONDS}s" + echo "" + + if [ ${FAILED_DATABASES} -eq 0 ] && [ ${MIGRATED_DATABASES} -eq ${TOTAL_DATABASES} ]; then + print_success "Migration completed successfully!" + echo "" + echo "Next steps:" + echo " 1. Run script 02-verify-migration.sh for comprehensive verification" + echo " 2. Update application connection strings" + echo " 3. Perform application testing" + echo " 4. Monitor for 24-48 hours before decommissioning source" + echo "" + else + print_error "Migration completed with errors" + echo "" + echo "Please review the errors above and:" + echo " 1. Check target cluster logs: kubectl logs -n ${TARGET_NAMESPACE} ${TARGET_NAME}-0 -c mysql --tail=100" + echo " 2. Verify failed databases manually" + echo " 3. Re-run migration for failed databases if needed" + echo "" + exit 1 + fi +} + +#============================================================================= +# MAIN EXECUTION +#============================================================================= + +main() { + print_header "MySQL 5.7 to 8.0 Migration" + + print_warning "IMPORTANT: Ensure application writes are stopped during migration" + echo "" + sleep 2 + + check_prerequisites + migrate_databases + migrate_users + verify_migration + print_summary + + print_success "Migration script completed" +} + +main +``` + +#### Script 3: 02-verify-migration.sh + +Save this script as `02-verify-migration.sh`: + +```bash +#!/bin/bash +#============================================================================= +# MySQL 5.7 to 8.0 Migration - Comprehensive Verification Script +#============================================================================= +# +# This script performs comprehensive verification of the migration: +# 1. Verifies all database objects (tables, views, routines, triggers, events) +# 2. Tests view execution +# 3. Compares row counts +# 4. Verifies user accounts +# +# Usage: +# 1. Edit the configuration section below +# 2. Run: chmod +x 02-verify-migration.sh +# 3. Run: ./02-verify-migration.sh +# +#============================================================================= + +set -e # Exit on error + +#============================================================================= +# CONFIGURATION - EDIT THESE VALUES +#============================================================================= + +SOURCE_NAME="source" +SOURCE_NAMESPACE="your-namespace" +SOURCE_MYSQL_PASSWORD="source-root-password" + +TARGET_NAME="mysql-8-target" +TARGET_NAMESPACE="your-namespace" +TARGET_MYSQL_PASSWORD="target-root-password" + +# IMPORTANT: databases that were migrated (DO NOT include: information_schema, mysql, performance_schema, sys) +DATABASES="db1 db2 db3" # ← Same as used in migration script + +# Users to exclude from verification (system users and MySQL MGR users) +EXCLUDE_USERS="'mysql.sys', 'mysql.session', 'mysql.infoschema', 'root', 'clustercheck', 'monitor', 'operator', 'xtrabackup', 'repl', 'exporter', 'healthchecker', 'clusterchecker', 'mysql', 'percona.telemetry', 'manage'" +# Note: MySQL MGR system users (mysql_innodb_cluster_%, mysql_router%) are filtered in verify_users() + +# Color output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +CYAN='\033[0;36m' +NC='\033[0m' # No Color + +# Verification counters +TOTAL_CHECKS=0 +PASSED_CHECKS=0 +FAILED_CHECKS=0 + +# Temp directory +WORK_DIR="/tmp/mysql-migration-verify" +mkdir -p ${WORK_DIR} + +#============================================================================= +# FUNCTIONS +#============================================================================= + +print_header() { + echo "" + echo "========================================" + echo "$1" + echo "========================================" +} + +print_section() { + echo "" + echo -e "${BLUE}>>> $1${NC}" +} + +print_success() { + echo -e "${GREEN}✓ $1${NC}" +} + +print_error() { + echo -e "${RED}✗ $1${NC}" +} + +print_warning() { + echo -e "${YELLOW}⚠ $1${NC}" +} + +print_info() { + echo -e "${CYAN}ℹ $1${NC}" +} + +check_count() { + local source_count=$1 + local target_count=$2 + local object_name=$3 + + TOTAL_CHECKS=$((TOTAL_CHECKS + 1)) + + if [ "${source_count}" = "${target_count}" ]; then + print_success "${object_name}: ${target_count} (match)" + PASSED_CHECKS=$((PASSED_CHECKS + 1)) + return 0 + else + print_error "${object_name}: Source=${source_count}, Target=${target_count} (mismatch)" + FAILED_CHECKS=$((FAILED_CHECKS + 1)) + return 1 + fi +} + +verify_tables() { + print_section "Verifying Tables" + + for db in ${DATABASES}; do + echo "" + echo "Database: ${db}" + + SOURCE_COUNT=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_TYPE = 'BASE TABLE'; + " 2>/dev/null | grep -v "Warning") + + TARGET_COUNT=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_TYPE = 'BASE TABLE'; + " 2>/dev/null | grep -v "Warning") + + check_count "${SOURCE_COUNT}" "${TARGET_COUNT}" "Tables" + done +} + +verify_views() { + print_section "Verifying Views" + + for db in ${DATABASES}; do + echo "" + echo "Database: ${db}" + + SOURCE_COUNT=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.VIEWS + WHERE TABLE_SCHEMA = '${db}'; + " 2>/dev/null | grep -v "Warning") + + TARGET_COUNT=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.VIEWS + WHERE TABLE_SCHEMA = '${db}'; + " 2>/dev/null | grep -v "Warning") + + if check_count "${SOURCE_COUNT}" "${TARGET_COUNT}" "Views"; then + # Test view execution if counts match + if [ "${TARGET_COUNT}" -gt 0 ]; then + VIEW_FAILED=0 + VERIFY_TMP="${WORK_DIR}/view_verify.txt" + echo "0" >${VERIFY_TMP} + + kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT TABLE_NAME + FROM information_schema.VIEWS + WHERE TABLE_SCHEMA = '${db}'; + " 2>/dev/null | grep -v "Warning" | while read view_name; do + if ! kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} ${db} -e "SELECT COUNT(*) FROM \`${view_name}\`;" 2>&1 | grep -q "ERROR"; then + : # view works + else + echo "1" >>${VERIFY_TMP} + fi + done + + if [ "$(cat ${VERIFY_TMP} | wc -l)" -eq 1 ] && [ "$(cat ${VERIFY_TMP})" = "0" ]; then + print_success "All views execute successfully" + else + print_error "Some views failed execution" + fi + + rm -f ${VERIFY_TMP} + fi + fi + done +} + +verify_routines() { + print_section "Verifying Stored Procedures" + + for db in ${DATABASES}; do + echo "" + echo "Database: ${db}" + + SOURCE_COUNT=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.ROUTINES + WHERE ROUTINE_SCHEMA = '${db}' AND ROUTINE_TYPE = 'PROCEDURE'; + " 2>/dev/null | grep -v "Warning") + + TARGET_COUNT=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.ROUTINES + WHERE ROUTINE_SCHEMA = '${db}' AND ROUTINE_TYPE = 'PROCEDURE'; + " 2>/dev/null | grep -v "Warning") + + check_count "${SOURCE_COUNT}" "${TARGET_COUNT}" "Stored Procedures" + done + + echo "" + print_section "Verifying Stored Functions" + + for db in ${DATABASES}; do + echo "" + echo "Database: ${db}" + + SOURCE_COUNT=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.ROUTINES + WHERE ROUTINE_SCHEMA = '${db}' AND ROUTINE_TYPE = 'FUNCTION'; + " 2>/dev/null | grep -v "Warning") + + TARGET_COUNT=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.ROUTINES + WHERE ROUTINE_SCHEMA = '${db}' AND ROUTINE_TYPE = 'FUNCTION'; + " 2>/dev/null | grep -v "Warning") + + check_count "${SOURCE_COUNT}" "${TARGET_COUNT}" "Stored Functions" + done +} + +verify_triggers() { + print_section "Verifying Triggers" + + for db in ${DATABASES}; do + echo "" + echo "Database: ${db}" + + SOURCE_COUNT=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.TRIGGERS + WHERE TRIGGER_SCHEMA = '${db}'; + " 2>/dev/null | grep -v "Warning") + + TARGET_COUNT=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.TRIGGERS + WHERE TRIGGER_SCHEMA = '${db}'; + " 2>/dev/null | grep -v "Warning") + + check_count "${SOURCE_COUNT}" "${TARGET_COUNT}" "Triggers" + done +} + +verify_events() { + print_section "Verifying Events" + + for db in ${DATABASES}; do + echo "" + echo "Database: ${db}" + + SOURCE_COUNT=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.EVENTS + WHERE EVENT_SCHEMA = '${db}'; + " 2>/dev/null | grep -v "Warning") + + TARGET_COUNT=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.EVENTS + WHERE EVENT_SCHEMA = '${db}'; + " 2>/dev/null | grep -v "Warning") + + check_count "${SOURCE_COUNT}" "${TARGET_COUNT}" "Events" + done +} + +verify_row_counts() { + print_section "Verifying Row Counts (Sample)" + + for db in ${DATABASES}; do + echo "" + echo "Database: ${db}" + + # Get first 5 tables for sampling + TABLES=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT TABLE_NAME + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_TYPE = 'BASE TABLE' + LIMIT 5; + " 2>/dev/null | grep -v "Warning") + + if [ -z "${TABLES}" ]; then + print_warning "No tables found in ${db}" + continue + fi + + ROW_MISMATCH=0 + for table in ${TABLES}; do + SOURCE_ROWS=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT TABLE_ROWS + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_NAME = '${table}'; + " 2>/dev/null | grep -v "Warning") + + TARGET_ROWS=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT TABLE_ROWS + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = '${db}' AND TABLE_NAME = '${table}'; + " 2>/dev/null | grep -v "Warning") + + # Allow small variance due to statistics + if [ "${SOURCE_ROWS}" != "${TARGET_ROWS}" ]; then + print_warning "Row count variance for ${table}: Source=${SOURCE_ROWS}, Target=${TARGET_ROWS}" + ROW_MISMATCH=1 + fi + done + + if [ ${ROW_MISMATCH} -eq 0 ]; then + print_success "Row counts: Sample check passed" + fi + done +} + +verify_users() { + print_section "Verifying User Accounts" + + SOURCE_USERS=$(kubectl exec ${SOURCE_NAME}-pxc-0 -n ${SOURCE_NAMESPACE} -- \ + mysql -uroot -p${SOURCE_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM mysql.user + WHERE user NOT IN (${EXCLUDE_USERS}) + AND user NOT LIKE 'mysql_innodb_cluster_%' + AND user NOT LIKE 'mysql_router%'; + " 2>/dev/null | grep -v "Warning") + + TARGET_USERS=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM mysql.user + WHERE user NOT IN (${EXCLUDE_USERS}) + AND user NOT LIKE 'mysql_innodb_cluster_%' + AND user NOT LIKE 'mysql_router%'; + " 2>/dev/null | grep -v "Warning") + + check_count "${SOURCE_USERS}" "${TARGET_USERS}" "User accounts" || true + + # Show migrated users + if [ "${TARGET_USERS}" -gt 0 ]; then + echo "" + print_info "Migrated users:" + kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT CONCAT(user, '@', host) + FROM mysql.user + WHERE user NOT IN (${EXCLUDE_USERS}) + AND user NOT LIKE 'mysql_innodb_cluster_%' + AND user NOT LIKE 'mysql_router%' + ORDER BY user; + " 2>/dev/null | grep -v "Warning" | while read user; do + echo " - ${user}" + done + fi +} + +test_data_integrity() { + print_section "Testing Data Integrity" + + print_info "Performing sample data integrity checks..." + + for db in ${DATABASES}; do + # Check if database exists on target + DB_EXISTS=$(kubectl exec ${TARGET_NAME}-0 -n ${TARGET_NAMESPACE} -c mysql -- \ + mysql -uroot -p${TARGET_MYSQL_PASSWORD} -N -e " + SELECT COUNT(*) + FROM information_schema.SCHEMATA + WHERE SCHEMA_NAME = '${db}'; + " 2>/dev/null | grep -v "Warning") + + if [ "${DB_EXISTS}" = "1" ]; then + print_success "${db}: Database exists on target" + else + print_error "${db}: Database NOT found on target" + FAILED_CHECKS=$((FAILED_CHECKS + 1)) + fi + done +} + +print_summary() { + print_header "Verification Summary" + + echo "" + echo "Total checks: ${TOTAL_CHECKS}" + echo -e "${GREEN}Passed: ${PASSED_CHECKS}${NC}" + echo -e "${RED}Failed: ${FAILED_CHECKS}${NC}" + echo "" + + if [ ${FAILED_CHECKS} -eq 0 ] && [ ${PASSED_CHECKS} -eq ${TOTAL_CHECKS} ]; then + print_success "ALL CHECKS PASSED!" + echo "" + echo "Migration verification successful. Next steps:" + echo " 1. Update application connection strings to point to target" + echo " 2. Perform application testing" + echo " 3. Monitor target cluster for 24-48 hours" + echo " 4. Keep source cluster available for rollback during this period" + echo "" + return 0 + else + print_error "SOME CHECKS FAILED" + echo "" + echo "Please review the failed checks above and:" + echo " 1. Check target cluster logs: kubectl logs -n ${TARGET_NAMESPACE} ${TARGET_NAME}-0 -c mysql --tail=100" + echo " 2. Verify failed objects manually" + echo " 3. Re-run migration for specific databases if needed" + echo "" + return 1 + fi +} + +cleanup() { + rm -rf ${WORK_DIR} +} + +#============================================================================= +# MAIN EXECUTION +#============================================================================= + +main() { + # Trap to cleanup on exit + trap cleanup EXIT + + print_header "MySQL 5.7 to 8.0 Migration Verification" + + verify_tables + verify_views + verify_routines + verify_triggers + verify_events + verify_row_counts + verify_users + test_data_integrity + print_summary +} + +main +``` + +--- + +## Summary + +This guide provides comprehensive, test-verified instructions for migrating MySQL 5.7 to 8.0 on Alauda Container Platform. The solution has been verified in Kubernetes test environments with PXC 5.7.44 and MGR 8.0.44 clusters. + +### What This Guide Covers + +| Test Category | Test Cases | What It Validates | +|--------------|------------|-------------------| +| Basic Migration | Core functionality | Tables, data, foreign keys, indexes | +| Schema Compatibility | MySQL 8.0 issues | Reserved keywords, ZEROFILL, date defaults, TEXT columns | +| Database Objects | All object types | Procedures, functions, triggers, events, views | +| User & Privilege Migration | Security & access | User accounts, grants, authentication plugins | +| Character Set Migration | Data integrity | utf8mb4 conversion, multi-language support | +| GTID Handling | Replication | GTID_PURGED filtering for MGR targets | + +### Key Benefits + +- ✅ **Proven Approach**: Tested in Kubernetes test environments +- ✅ **Complete Coverage**: Migrates ALL standard MySQL objects with comprehensive verification +- ✅ **Schema Compatibility**: Automated checks and fixes for MySQL 8.0 compatibility issues +- ✅ **Character Set Support**: Complete utf8mb4 migration strategy +- ✅ **Security**: User and privilege migration with MySQL 8.0 authentication guidance +- ✅ **Performance**: Post-migration optimization for MySQL 8.0 features +- ✅ **Risk Mitigation**: Detailed rollback procedures and validation at each step + +### Production Readiness Checklist + +Before using this guide in production, ensure you have: + +- [ ] Reviewed the [Getting Started](#getting-started) section to understand your environment +- [ ] Tested the migration procedure in a non-production environment +- [ ] Completed [Schema Compatibility Analysis](#schema-compatibility-analysis) and fixed all issues +- [ ] Completed [Character Set Migration](#character-set-and-collation-analysis) if using legacy charsets +- [ ] Scheduled adequate maintenance window based on database size +- [ ] Communicated with all stakeholders (application teams, DBAs, SREs) +- [ ] Prepared rollback plan (see [Disaster Recovery](#disaster-recovery)) +- [ ] Verified application compatibility with MySQL 8.0 authentication plugins + +### What This Guide Delivers + +By following these practices, organizations can successfully migrate their MySQL databases to version 8.0, ensuring: + +- ✅ **Continued security support** (MySQL 5.7 EOL was October 2023) +- ✅ **Access to new features** (CTEs, Window Functions, Histograms, etc.) +- ✅ **Data integrity maintained** through comprehensive verification +- ✅ **Minimal downtime** with tested procedures +- ✅ **Rollback capability** if issues arise + +### Support and Troubleshooting + +If you encounter issues not covered in this guide: + +1. Check the [Troubleshooting](#troubleshooting) section for common issues +2. Review the [Important Limitations](#important-limitations) section +3. Verify your environment matches the [Prerequisites](#prerequisites) +4. Follow the [kubectl Exec Best Practices](#getting-started) to avoid common command errors +5. Check MySQL error logs: `kubectl logs -n -c mysql --tail=100`