Welcome back. This series is going through the process of setting up a scalable production VPS for a ruby on rails app, from a non-sysadmin perspective. The first post is here.

Since my last post, I’ve been doing a number of different things with my server in an attempt to find the best configuration and share it with you. Before I move forward with the specifics of that, I wanted to take a brief moment to share some of the things I’ve learnt while experimenting with this.

Load Balancing is Fun

I haven’t talked about it much yet, but you probably already know that for serious rails deployment, we’re talking about multiple instances of the Mongrel web-server, and some sort of software load balancer to pass the HTTP requests through to them.

I’ve tried out two different load balancers on my ‘experimentation server’. The first was Pound, which was nice because it’s configuration file really is just so damn simple (mainly because it does nothing but load balance). On the downside, the Ubuntu packages for it were too old to actually work nicelly, so I had to compile it. Then, I had to ensure that it restarted on reboot (I wrote my own init.d script). This isn’t a bad solution, but Pound doesn’t serve static content, and Mongrel isn’t well suited to it either, so it typically requires the installation of another web server as well, such as Apache or Lighttpd.

I also tried out using Apache2, with mod_load_balancer. At this point, I have a confession to make:

I was scared of apache configuration files.

There, I’ve said it. I’ve used apache for years, for static sites as well as PHP, and Rails, and it’s always worked fine. However, I always felt that I needed to stay ‘on the beaten path’ of the config that my linux distro provided for me, and look closely at other people’s recipes before considering anything but the simples change to my apache configuration.

Now you may well think that I’m a bit silly, but I’m well pleased to now be free of this notion. Apache configuration is very straightforward. I think it’s made a bit complicated by distros including config files fill of default setting which I don’t really need. Standard configuration of apache is simple, virtual hosts are simple, and mod load balancer is also straight forward to configure and seems to be rock solid to use.

So, I’m going to stick with the apache option, and in my next post on this subject, I’ll be talking about that setup in more detail.

RAM is Your Biggest Cost for Small Sites

VPS prices are heavily dependent on how much RAM you get. This is because servers can only have so much ram, and so the amount of available RAM tends to determine how many VPSs can fit on a single physical server.

I’m told, by people in the know, that this year we’re starting to see physical servers with a lot more available RAM than previously (ie; 32 gig instead of 4 gig). I’ll be interested to see if this means that we’ll see a lot more cheap VPSs which have insufficient CPU power. For now, however, I generally find that although I can max out the CPU on a small VPS by writting crappy code (for experimental purposes only, I assure you), it is RAM which is the most limiting resource.

One of the cheapes VPS options that I know of, is that US$20 plan from slicehost.com. That gives you 256 MB of RAM, which I’m quickly realising is not enough for a rails app with a bit of load. That’s right, 256 MB is not enough.

Why is this so? Well, Mongrel seems to be the main culprit. Mongrel keeps the whole rails stack in RAM, as it should to perform fast enough, which means that it ends up taking a pretty constant slice of the RAM. Unlike running a PHP site with Apache, RAM doesn’t really go down so much during light load, unless you dynamically reduce the number of Mongrels that you are running. So, this means that you get predictability, but it also means that you’re running pretty close to a permanent worse case scenario. Once you understand this, it’s a pretty good solution, proving you plan for it.

Zed Shaw has written some great stuff on how to find out how many mongrels you should run for your given situation. There is obviously a limit, because eventually you run out of CPU cycles so adding more mongrels doesn’t help. However, my rough guess at the moment is that initially you need a few, and with a 256 MB VPS, I only feel comfortable running TWO mongrel processes.

According to my quick and dirty figures, a mongrel process with a decent sized rails app requires about 60 MB. MySQL can gobble up RAM under load, growing from maybe 22 MB when it’s idle, to several times that when processing your nasty un-optimised active record queries. I’m a bit unsure about how much RAM apache needs under real load, but I’m currently estimating that at around 50 MB.

Problems with running out of RAM are problems of the nastiest sort, so you want to leave yourself a buffer. If you try and sneak a third Mongrel process into a 256 MB VPS, then you are running too close to the wire, and a single large database query might bring your site grinding to a halt. So don’t do it. Leave a good chunk free, and stick with two, or move to a 512 MB VPS. For a proper web application, even one just starting out, I’m currently recommending the latter.

Coming up Next

While it’s kind of fun investigating this stuff, my mate Mike Bailey (who knows far more about this than I do), has just gone and automated a lot of it with a new gem containing capistrano tasks. He’ll be announcing that in the next couple of days, and I’m testing it for him over the weekend. So, if things go well, I’m hoping that my next post includes some words about installing using that process. Anything that we do repeatedly, and particularly anything that can go wrong, should be automated.