My last week’s been comprised of a lot of reading, a lot of brain-storming, and a lot of planning.
Here’s a scenerio:
- Step 1: One of the sites on my VPS gets slashdotted, dugg, or picked up on by a major news site. It has actually happened a few times already although fortunately it’s been a side-story rather than a front-page news item (I say “fortunately” because I was on a shared host the times it happened. Being a side-story, it didn’t kill the server or get me canned).
- Step 2: VPS starts to choke – either Apache swells up and starts swapping / dies when it hits the memory limit, or MySQL starts suffering from the same effect. In any case, the site doesn’t handle the traffic.
If the VPS isn’t responsive enough to log in, the possible rush-to-fix’s become:
- See if the host can bump the resources on the VPS. If the physical server has already had all it’s resources provisioned out, I could be out of luck. If they try moving the node to another machine that does have room, there’s the possible chance for something to break severely resulting in a dead VPS and requiring a restore from backups. Not ideal since it could involve downtime.
- Grab a larger VPS elsewhere, restore backups to it, and point the DNS servers there. Of course, this takes some time. At the very least, I’m looking at the time to get a new server provisioned, plus the time to install the control panel, plus the time to upload the backups. Then, it takes a while for the DNS to propogate. Slowly, the new server would start getting the traffic and the old one would become responsive again. Still not an ideal solution.
So 2 less-than-ideal solutions if the VPS is hammered to the point where it isn’t responding. Both require me to stop whatever I’m doing and be around to deal with the issues.
How about options if the server is responsive? May get by with 1 change, but may have to do a few things:
- Force all the CMS’s to use a file cache to reduce the load on MySQL.
- Tweak (and restart) Apache to get every last ounce out of it.
- Install LiteSpeed (only drop-in replacement for Apache), and hope to heck that it handles the load and that new issues aren’t introduced. This of course assumes that the demo or paid license can be obtained relatively quickly – it may not.
- Throw the smaller sites into maintenance mode (or temporarily kill them off completely until traffic dies down).
- Upload all the images/etc to a file-hosting service (or a shared service with instant set-up, using the user directory as the file path), and then edit the site to pull images and static files from the other service, hoping that offloading the static files to another server is enough to keep the VPS going.
Again, not the greatest solutions. They all require immediate work (“sorry I can’t make it to your wedding today…”), and making frantic changes to a live site in the midst of being hammered generally increases the chance that something may go wrong. Some necessitate a little down-time.
Seeing that there aren’t really good solutions to deal with the madness when it strikes, let’s see what can be done to put the VPS in a position where there are good solutions when it does:
- Switch to another webserver. Nginx and Lighttpd are both quite capable of taking a good beating.
Not a great solution for me unfortunately (although for others this may be one of the best solutions). I strongly considered it, but I’ll go into a post later about why I decided against it at this time.
- Switch to a dedicated server or another VPS that’s well beyond what’s needed.
Very possible, very costly. It’s tough to justify paying more than you need to in advance just because it’ll make it easier when you do need it in the future.
- Switch to a “cloud” VPS that can be upgraded.
Now we’re getting somewhere. Pay for what I need, upgrade as I need it.
- Move common static content (images, etc) elsewhere.
Completely worth considering, I’ll mention why, and what I’ll be doing, in two steps.
Switching to a “cloud” VPS that can be upgraded.
What makes this a good idea?
- The biggest advantage is that you pay for what you need, and it’s easy to upgrade/downgrade as needed.
- The VPS image isn’t “tied” to a specific server – the back-end is able to move it around at will, and all without downtime.
- Because the VPS isn’t tied to a specific server, if you upgrade your VPS, and there isn’t room on the server it’s sitting on, it just gets moved behind the scenes to one that does have room.
- Reliability’s a plus. If the current server kicks the can, your VPS is moved to another and keeps functioning. Compare this to a standard VPS where if the server dies, you have downtime while they replace the machine and restore backups.
What makes this not-so-good?
- From what I’ve read, some cloud systems have less-than-stellar SQL performance. I’d suspect this would apply more to cloud shared hosting, since they’d probably be running the database on separate machines which adds a little latency.
- Depending on the cloud set-up, reliability may not be improved all that much. Really, if your image is stored on a single server within the cloud, and that server instantly croaks or bursts into flames, your VPS could be gone just the same as with a standard non-cloud VPS, resulting in the need to restore backups. That said, there are ways a cloud could (and probably should) be set up to actually have your VPS image stored on multiple servers, not only adding redundancy but allowing it to be served up similar to a CDN.
- Finally, if something absolutely terrible happens to the software running the cloud behind the scenes, well… let’s just say I doubt anybody on the cloud will be impressed.
Really, what this boils down to is that nothing’s bulletproof. There’s still the chance for downtime, and you should still have your own backups. The big plus is being able to scale your VPS on a whim.
I looked specifically at VPS.net . You buy nodes, add them to your VPS’s, and if you want to increase them (whether for a day or permanently), you can. In short, you pay for what you need now, and you have the ability to add more resources both temporarily and in the long-term. It’s a good way to keep things both simple and cost-effective, meeting demands as you grow.
Moving static content (images/etc) elsewhere
There are 2 good places to move static content.
For images and other small files that are called by the page, while you could use a separate slimmed-down VPS running only nginx/lighttpd/etc to serve static files, a CDN may be an even better idea. It gets a little pricey (you’re often charged per GB of transfer), but in turn your site will load much faster for people on the other side of the world (assuming of course the CDN provider has part of their network there). The more images/css/etc that make up your page, the larger the improvement. It also takes the burden of serving these files off your server. Fewer requests to Apache is a good thing.
However, larger files (typically downloads) generally don’t require a CDN, because unless you’re selling a product, people tend not to care much if the download is slow, especially for relatively small files. For downloads that are offered from your site (programs, files, etc), a cheap shared host that offers “unlimited” bandwidth can be a good place to move those static files to. Besides moving these requests elsewhere (slightly reducing the load on your VPS), this can also make your VPS backups take significantly less space, particularly if you’ve got a couple hundred MB worth of files that you offer up for download. A big caveat though… many shared hosts shut down client sites that are using too many resources. Generally, they shut down sites that are using too much processor power or memory. Serving static files fortunately doesn’t use much of either (unless you’re serving a lot). However, they have to pay for their bandwidth too. If your bandwidth usage is sky-high, you may be shut down. At the very least, you’ll want to read the terms and see if they’ve given any hard limits for bandwidth usage, prohibit serving files off the site, or charge for usage above a certain amount. You’ll also want to have the files backed up in case you do get shut down. Be reasonable with your expectations here… you’re not going to start the next Download.com on a shared host. I’ll put it this way – if you’ve needed to upgrade your VPS because of bandwidth issues, an “unlimited” shared host for $10 probably isn’t going to work for you. If it does, great (and I’m happy for ya), but I’d still be careful.
For me, the choices were as follows:
VPS.net’s CDN for static files. You can buy a chunk of non-time-limited bandwidth in 1TB intervals (starting at $75), or pay monthly for various amounts of bandwidth in 1TB intervals (starting at $50/month). I opted for non-time-limited bandwidth since most of my images are a part of the basic non-fancy site templates I use and are relatively small.
HostGator’s “unlimited” shared hosting. Brent from HostGator stated in a blog post (Web Archive) that “At HostGator, we pretty much get an unlimited amount of bandwidth from our provider thanks to having thousands of servers.” For under $10/month, I’ll be moving the downloads I offer to them. It decreases the size of my VPS backups, takes that load off my VPS, and from the sounds of it they have so much bandwidth that it’s not going to matter to them anyway – if anything they’ll probably be happy that a paying customer is barely touching the CPU/memory usage by simply serving static files.
Breaking it all down, moving the static stuff off to the CDN and shared host will mean that I can delay an upgrade a bit longer. To be frank it’s enough to “buy some time” on my current VPS. That said, it still won’t guarantee the server will live if hit hard enough, and that’s where the cloud hosting comes in. If the VPS is zerged by traffic, I can simply add nodes to the VPS instantly. Even if I don’t get zerged, the sites started on a $30/year shared package (that’s under $3/month!) and grew over time to larger packages and then finally to the needs of a VPS. It’s inevitable that I’ll have to upgrade again, and when the time comes it’ll be nice to sit back and do it at the click of a button.
This change will be happening within the next couple of weeks. I’ll be trying it out and post an update with the results.