For a while, Spotivate was running on a single EC2 instance. Everything was in it — MySQL, Django, static files, etc. Yes, we know this is a terrible setup. Single point of failure, bad performance, etc. Here comes the excuses. We had better things to do, like customer development, sales, design, product development, etc. We had no time for ops! Plus, our traffic wasn’t really that high especially in the beginning. Our CPU / IO load was low. And we knew we can fix things fairly easily. Then one day, our EC2 instance went down for half an hour. Ooops! Called AWS support. They had a disk failure. Our last snapshot was a day old. So our site was down this whole time.
We figured we had to do it right. And EC2 makes it super easy. Our goal:
- Remove all single points of failure, thus making the system fully fault tolerant.
- As a result, the response time should go up, especially when under load.
Here’s the plan:
- Move MySQL to RDS.
- Move Django static files to S3.
- Have multiple EC2 instances via Elastic Load Balancer.
In this article, I’ll talk about the steps we took to move our MySQL to RDS.
If you don’t know what RDS is, read more about it here. Basically it’s AWS’s version of database server. RDS comes loaded with features. Here’s summary of what’s relevant:
- Easy to deploy via the Management Console or command line.
- Automatic backup (you get to choose how many days and when).
- Multi-availability zone deployment means AWS automatically creates a primary DB instance and synchronously replicates the data to a standby instance in a different Availability Zone, thereby removing this as a single point of failure.
- Replication that allows you to create read-only replicas. This is especially valuable for Spotivate, since our personalized email server put a heavy load on the DB. By having this, the performance of our website won’t be affected while we send out our weekly emails.
Well, let’s get on with it.
Step 1: Goto your management console and select RDS
Step 2: Find a database server that fits your bill. In our case, MySQL.
Step 3: Here’s where you pick the MySQL version and the instance size.
|Multi-AZ Deployment||Select “Yes” which creates a standby instance in a different AZ. That’s the whole point of this article, right?|
|Allocated Storage||Choose a storage size that’s appropriate. Go small, as you can easily upgrade later with minimal down time. Generally, estimate enough for 3 months down the road.|
|DB Instance Identifier||This is just the prefix to the public DNS.|
|Master Username||Your database user name, typically “root”|
|Master Password||Your database root user password|
Step 4: Here you specify the database name, port, etc.
You also get to create (or assign) a database security group for this database. This is a little different from the EC2 security group. For database security group, you assign which EC2 security group to use. And any EC2 instance that belongs to that EC2 security group has access to the database. By default, everything else is turned off including ping. For more info, visit here.
Step 5: Backup Settings
Here, you specify the backup retention period, and when to backup. Make sure your backup window and maintenance window don’t overlap.
Step 6: That’s it. Review and Launch.
Step 7: Test it out.
After the DB has been launched (takes several minutes – enough time for coffee), you can find the public DNS from the detail page. This machine is accessible externally and within EC2. However, the security group by default prohibits any external access to the database server. Only EC2 instances that belong to the security group have access. From my web server, I can use my typical “mysql” command to connect to the new RDS instance.
Step 8: Import.
Our database is fairly small, so we can just dump the database and pipe it to the new instance. Here’s a fun command that you can use (make sure you stop your web server first to avoid consistency issue).
mysqldump [your current db] | mysql --host=[rds host name] --user=root --password [root password]
That’s it! All you need to do now is change your Django settings to use the new database instance. Bring down your local MySQL and restart your Django server to see if everything is running properly. If so, change
chkconfig to keep the local MySQL from restarting.
Next time, I’ll talk about the migration of our static files to S3.