Beyond the Switchover: Using RDS Snapshots to Safely Test MySQL 8.4 Compatibility Before Your Blue/Green Cutover

AWS RDS Blue/Green deployment is a powerful zero-downtime tool, but its Green environment is read-only by design. Here is how to layer a snapshot-based test instance on top of it to validate application compatibility before committing to the full switchover. There is a moment every DevOps engineer, DBA, and SRE dreads: a database engine reaches end-of-life, AWS begins billing you for Extended Support, and the upgrade clock is ticking. For MySQL 8.0, that moment arrived in 2026. AWS began charging Extended Support fees for any RDS instance still running 8.0, and teams across the industry started planning migrations to MySQL 8.4, the current Long Term Support (LTS) release. AWS provides an excellent managed feature for this exact scenario: RDS Blue/Green Deployments. It is powerful, well-documented, and purpose-built for zero-downtime upgrades. But there is a constraint baked into how it works that trips up a lot of engineers the first time they reach for it, and it becomes a real blocker when you are managing a shared RDS instance across multiple projects with different readiness timelines. This post documents exactly what that constraint is, why it exists, what error you will see in your application logs, and how to layer a snapshot-based testing pattern on top of Blue/Green to get the best of both worlds: safe compatibility validation per project, followed by a single zero-downtime cutover for everything. The Context: A Shared RDS Instance Serving Multiple Projects Picture a common real-world setup. You have one RDS MySQL 8.0 instance - call it proddb - running in us-east-1. It hosts several logical databases across seven different projects/environments: a SaaS platform in Node.js, several Laravel applications, a portfolio site. Some of these applications have been running for years. The code was written before MySQL 8.4 existed. You have no guarantee that everything will work after the engine upgrade. You cannot afford to do a big bang upgrade. If one application breaks on MySQL 8.4, you need the others to continue running on the working engine while you fix it. You need a way to validate each project individually before committing everything. This is the scenario where Blue/Green deployments look like the perfect answer - and they are, but not in the way you might initially expect. What AWS RDS Blue/Green Deployment Actually Does Before getting into the limitation, it is worth being precise about what Blue/Green deployment is and why it exists. When you create a Blue/Green deployment in RDS, AWS provisions a Green environment that is an exact replica of your Blue environment (your current production instance). RDS manages the replication between them automatically, using physical replication for MySQL. The Green instance stays continuously synchronised with Blue. Your schema, your data, your parameter groups - everything is mirrored. The workflow is designed like this: - Create the Blue/Green deployment - Upgrade the Green instance to MySQL 8.4 (you do this manually after creation, or configure it during setup) - Test the Green environment - Trigger the switchover During switchover, AWS stops writes to Blue, waits for replication lag to reach zero, renames the instances (Green takes the original Blue endpoint, Blue gets an -old1 suffix and is set to read-only), and brings write traffic live on the new MySQL 8.4 instance. Typical downtime for single-region configurations is under five seconds. The old Blue instance is retained so you can roll back if needed. This is genuinely excellent engineering. For a planned, coordinated upgrade where all your applications are ready to move at the same time, it is the right tool. The Constraint You Will Hit: Green Is Read-Only by Design Here is where the trouble starts if your goal is gradual, per-project migration. The AWS documentation is explicit about this: "During testing, we recommend that you keep your databases in the green environment read only. Enable write operations on the green environment with caution because they can result in replication conflicts. They can also result in unintended data in the production databases after switchover." The Green environment is read-only by default. RDS sets the read_only parameter to 1 on the Green instance to protect replication integrity. If writes reach the Green during the testing phase, they could cause replication conflicts and potentially corrupt data that flows back after switchover. There is a technical reason for this. MySQL physical replication works by replaying binary log events from the Blue primary on the Green replica. If you write directly to the Green instance, those writes are not in the Blue binlog. When the switchover happens and the old Blue becomes the replica, those writes have no origin on the Blue side. The replication relationship breaks, or worse, data that was never in production gets silently promoted into what is now your production database. AWS therefore makes the protection the default. You can override it with a parameter group change, but the official guidance is to leave it alone. What This Looks Like in Your Application Logs If you follow the logical path - "I have a Green instance running MySQL 8.4, let me point my Dev application at it and test" - this is the error you will see immediately: Error: The MySQL server is running with the --read-only option so it cannot execute this statement at PromisePool.query (/app/node_modules/express-mysql-session/ node_modules/mysql2/promise.js:356:22) at /app/node_modules/express-mysql-session/index.js:384:30 { code: 'ER_OPTION_PREVENTS_STATEMENT', errno: 1290, sql: 'INSERT INTO sessions (session_id, expires, data) VALUES ('KYzPpAzZpm9vHHp7WI_6W7H0UHkBHnaq', 1786734753, '{"cookie":...}') ON DUPLICATE KEY UPDATE expires = VALUES(expires), data = VALUES(data)', sqlState: 'HY000', sqlMessage: 'The MySQL server is running with the --read-only option so it cannot execute this statement' } This surfaces immediately on the first write operation - in this case, the session store trying to persist a session record during an ELB health check. The application cannot even initialise properly. The cascade is: - ELB health checker hits /admin/login - Application attempts to create or update a session record in MySQL - Green instance rejects the write with ER_OPTION_PREVENTS_STATEMENT (errno 1290) - Application returns HTTP 500 - A secondary error ERR_HTTP_HEADERS_SENT follows because the error handler tries to write a response after the response has already been partially sent - Health check fails, ELB marks the target unhealthy [API][/admin/login][500][user_id:-] - { "request_id":"msrwaseu-3r7f2m", "method":"GET", "duration_ms":29, "headers":{ "host":"172.31.3.155:4000", "user-agent":"ELB-HealthChecker/2.0" } } Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client at ServerResponse.setHeader (node:_http_outgoing:655:11) at ServerResponse.header (/app/node_modules/express/lib/response.js:684:10) The ERR_HTTP_HEADERS_SENT error is a cascade failure, not the root cause. Fix the MySQL read-only problem and this disappears with it. The important thing to understand is that this is not a misconfiguration on your part. The Green instance is behaving exactly as designed. You pointed an application with write operations at a managed replica. The replica correctly refused the writes. Why the Naive Fixes Do Not Work "Can I just set read_only=0 on the Green instance via a parameter group?" Technically yes. The read_only parameter is dynamic in MySQL, meaning you can change it without a reboot. Some community members on AWS rePost have suggested this as a workaround. However, AWS's own guidance explicitly warns against it because it breaks the replication integrity guarantee. If your application writes data to the Green during testing and that data does not exist on Blue, the switchover will either fail guardrail checks or promote inconsistent data. This is not a theoretical risk - it is the exact failure mode the read-only default exists to prevent. "Can I just do the switchover now and test on the new production instance?" This defeats the purpose of testing before the switchover. If your application breaks on MySQL 8.4 and you have already cut over, you are now dealing with a production incident. The rollback path (replicating from the new MySQL 8.4 back to the old -old1 MySQL 8.0 instance using binlog position) is complex and time-sensitive. "Can I selectively switchover just one project's database?" No. Blue/Green deployment is an instance-level operation, not a database-level operation. When you trigger the switchover, every logical database on the instance moves simultaneously. You cannot switch one logical database while leaving others on Blue. The Solution: Snapshot the Green, Test in Isolation, Switch Everything at Once This is the pattern that gives you everything: - Per-project compatibility testing against a writable MySQL 8.4 instance - Replication on the Blue/Green deployment continues uninterrupted throughout - A single managed switchover when all projects are validated - No manual dump and restore for any project at cutover time Here is how it works. Step 1: Keep your Blue/Green deployment intact and revert your application immediately If you pointed an application at the Green endpoint and it is throwing 500s, revert DB_HOST in your secrets manager or SSM Parameter Store back to the Blue endpoint and redeploy. Get the application healthy again first. aws ssm put-parameter \ --name "/your-project/dev/DB_HOST" \ --value "proddb.xxxxxxxx.us-east-1.rds.amazonaws.com" \ --type "String" \ --overwrite \ --region us-east-1 Do not delete the Blue/Green deployment. It is doing its job. Replication is running, the Green is staying synchronised with Blue, and you will use it for the final switchover. Step 2: Take a snapshot of the Green instance In the AWS Console, navigate to Databases, select your Green instance (pr