With rails, we build up our application using the chunks of functionality that the framework and it’s libraries provide. This is different to web platforms like Drupal, where we might reasonable expect to get a basically working application (for the default use cases) out of the box, and be able to configure it to change it slightly.

The problem with configuring a monolithic platform is of course that we’ll never get it to work exactly the way we want. That approach works well if our requirements are close to it’s area of core functionality, but for custom software, we want to build up, rather than configure downward.

Building our apps up has been a key philosophy in rails from the beginning. In fact, it’s been so important that the core team has strongly resisted any changes to rails which looked too much like they encouraged monolithic, configurable software.

So, if we think of a rails app as a structure built up from lots of little blocks, which all do different things and are often interchangeable, then we have a good working metaphor for the ideal rails development process. Rails 3 (due to the Merb influence) has done a lot of work to ensure that some of the fundamental building blocks of rails are easily swappable. This blocks, like which ORM to use, form the foundations of the awesome block fortress that we want to build.

With each layer of blocks that we build up, the blocks tend to be somewhat dependent on the interface of the blocks below. For example, we might build a login system that requires a user model that basically behaves like an ActiveModel object, although we don’t care if it’s implemented using ActiveRecord, DataMapper, or something else.

Once the tower of blocks rises above the foundation of the Rails framework itself, then adding more blocks becomes dependent on some sort of agreement on those interfaces. This tends to happen when gems or plugins become so commonly used that supporting them, or something that looks a bit like them, is a natural approach for further libraries.

For example, there are a number of libraries supporting user authentication, and most of them expose a current_user method which returns some sort of user object. Although this is far from being a clearly defined interface, it’s almost all you need for other libraries to do things conditional on being logged in.

As we reach a consensus on certain libraries being good ones, even the ones that follow them tend to implement these core bits of interface. For example, you’ll find that changing an app from using Authlogic to using Devise is fairly simple.

This process made it feel a bit like the blocks were slowly getting bigger, and we could build bigger and bigger towers out of them without needing to write our own code from scratch. Certain parts of our app are good candidates for giving a lot of thought and custom code, but other parts are simply the bolt-ons that we need to get there, and it’s great to be able to use a generic implementation without having to re-invent the wheel. We don’t rewrite paperclip when we want to store images, and most people tend to use an authentication system these days.

So, why did the progression of these bigger and bigger blocks stop? Where are the rest?

If we want to be able to build up, rather than configure down, from any level of detail, then we need blocks at each level of detail. We need something to store files (paperclip), something to upload and store files (which uses the previous), something to provide an image gallery (which uses the previous), etc. It’s up to us which areas of our app use large blocks, and which ones use small ones.

When we start to talk about blocks that are large chunks of our application, they start needing to include user interface. This tends to be where we get scared and run away. The tricky bit here is the lack of defined interface, and that’s why you tend to see entire open source rails apps that do useful things like run a blog, forum, image gallery, social profiles, but not so many rails plugins that do these, and even less rails plugins that do it by building the functionality up from even smaller blocks which you can swap out.

In my next post, I’ll lay out some examples of what these “slightly larger that presently available” blocks of code might be, and take a stab at figuring out how we might start to define interfaces for them, while keeping a decentralized community process.

Ruby on Rails’ opinionated attitude towards testing has introduced a lot of people to test driven development. Whether it’s because rails development has encouraged good testing practices, or because it has attracted pro-testing developers, rails (and ruby in general) now has a large ecosystem of testing and quality tools.

If you’re a rails developer, you’re probably already writing tests. You’re probably using RSpec or Shoulda (and also Cucumber, but I’m just talking about unit tests here). You’ve probably run your code through rcov, to see how well much code your tests cover, and you’ve probably also looked at other code quality metrics, like the ones generated by metric_fu (or a hosted service like codeyak).

Quality metrics for your code force you to write good code. They are an essential part of test-driven development, because in TDD our design is supposed to be driven by spotting code smells. Unless we think we can get the design (both in the macro and micro) correct first time (and we can’t), then we always have to be on the lookout for these code smells that will drive us to refactor and evolve the design into something better. Since I’m using words like “drive” and “force”, obviously I intend these metrics to be enforced. If you aren’t already doing it, start now.

The following example presumes that you’re testing with rspec, but it only requires slight modifications to make the rcov threshold work with other testing libraries.

First up, ensure that your app has the following gems installed (bundled):

metric_fu, flay, flog, reek, roodi, rspec, rcov

Then, grab my continuous integration rake tasks from here:

http://gist.github.com/364034

Finally, tell your continuous integration server to run “rake ci”. I use cruise_control.rb, but there are plenty to choose from.

You can modify the hash values in the THRESHOLDS constant to set the quality thresholds for your project. Set them so that they just pass at the moment, and then try to improve the code and slowly crank them tighter. For rcov that means raising it (towards 100.0), and for the others, that means lowering them.

Since you’re using metric_fu, you’ll also get graphs and pretty reports of the output. If you’re using cruise_control.rb, look for the “output” link in your build artifacts on the web interface. I’m not executing rcov via metric_fu in my code, because the metric_fu rcov task hides errors in the specs, and we also want the build to fail if a spec fails.

There’s a bunch of information around on how to handle incoming emails with your rails application, in particular the wiki page, but I have some concerns with the methods that are being suggested, and in this article I present an alternative which I’ve been trying out and I really like.

Handling incoming email is, in essence, very simple. All you need to do is get the email, which is a big chunk of text, parse it with a ruby email class, such as TMail (which is used by ActionMailer), and perform some action. If you’re only handling a few specific addresses, it might be best to fetch the email via POP3, and I’ve done that before using a daemon to regularly poll the pop account.

POP3 is not a viable solution if you want to handle all email for a certain domain. At this point, we probably want to talk about SMTP.

A Very Short Guide to SMTP

Simple Mail Transport Protocol is pretty damn cool if you ask me. It’s dead simple, basically the client can only say “hello, here’s an email from X to Y”. Just like HTTP, it’s fully push based. There’s no polling, emails get pushed across the internet. Just like HTTP, it has a hole stack of response codes which are of course appropriate to trying to send an email, rather than talk to a web resource.

Using Postfix Mail Filters to Call Ruby

Postfix is a common open source SMTP server. Before I looked at it, it was big and scary. After a few hours of expert help, I wonder what seemed so complicated. One of the basic ways that we can use postfix to push mail to our rails app is by specifying a command like script which gets executed whenever postfix gets an email. This is the first option presented on the rails wiki, and they suggest using a script which calls the receive method of one of your ActionMailer classes.

My Concern

If we’re going to use ActionMailer to parse an email, and then presumably fire off a bunch of ActiveRecord code to make changes to your database as a result, clearly we’re loading the entire rails stack. Every time we get an email we’re loading the entire rails stack. This seems like how we handled web requests back in the day when there was only mod_cgi. No shared resources between requests, a big performance hit for loading all of rails and then getting rid of it each time, and the concern that we can only handle as many incoming emails as we have RAM on our server as the rails code takes up a bunch of memory.

What I Want

I don’t want to have to worry about the resources I need to scale my email server, I already do that with my application servers. I want to handle emails in a way that re-uses an in-memory copy of the rails classes and called be scaled in a predictable way.

That sounds a lot like a mongrel cluster.

We all have one of those already right. So why not handle incoming mail over HTTP? It’s dead easy, it scales well, and the result is really Rails-ish.

respond_to.email

I was hoping to get a plugin out of this. It’d be so handy that people would queue for miles to download it. The trouble is, it’s actually not even enough code to bother, it’s only about three lines of ruby and the same of postfix config. So, lets call this a pattern. I’ll describe how to do it, and you can all run off and do it yourself.

Step One: Install Postfix

Install postfix on one of your servers. For any sizable rails site, I like to have a little VPS just for daemons, cron jobs, scripts, and the mail server, to keep it separate from all the web stuff. On ubuntu, this was as easy as “sudo apt-get install postfix”. For the default configuration type, I chose “internet site”.

Step Two: Setup Your MX Record

For mail to start arriving at your mail server, you need to add a MX record to your DNS which points at the url of your server. Depending on your host, you probably have a web interface to do this, and it’s probably dead easy.

The Magic Script!

Create a file called mail_handler.rb, and pop it somewhere in your rails project. I created a /bin directory for it. Don’t use a rake task, the goal here is not to load in any unecessary stuff. Here’s the contents.

If ruby is somewhere else on your machine, change the line at the top to be correct (try “which ruby” on that machine to see where it is). I’ve chosen to hardcode in the url that I want to post the email to so that I don’t have to load any other files. If you have more deployment environments to worry about, you might want to put the target url in a yml file and parse it here. Just don’t load your rails environment file, that’s the whole point of this.

Configuring Postfix to Call the Script

In this example, the domain that I want to handle email for is “craigambrose.com”. Everywhere you see this, replace it with your own domain name. Most of the commands below need root access.

In /etc/postfix/main.cf

In /etc/postfix/virtual (this is a file, you may need to create this)

The above says to redirect any address at craigambrose.com to the alias “rails_mailer”, which I’ll create next. You could run multiple rails apps of the same server by giving them all unique aliases. On the left, you can use a regular expression to match addresses if you only want to match some of them.

To apply this change to virtuals, run:

In /etc/aliases

That’s the alias we created on the left. On the right is the path to my script, change as necessary. The pipe character before the script path means “the following is a shell command, not an email address”.

To apply this change to aliases, run:

To apply the main configuration changes to postfix, run:

Testing the Setup

I should be able to send an email now to “someaddress@craigambrose.com”. To see it get process by postfix, we might want to watch the postfix info log:

When the mail is process, you should see a line like:

Then, go peek at your rails app logs. You should see that the mail has been passed through by the script. Even if you haven’t written an action to handle it yet, the log entry should be there.

Troubleshooting

If you didn’t see the correct line in your postfix logs, then perhaps there’s a problem with your DNS Set. You could try talking to postfix directly. Mail servers listen on port 25, and you can telnet into them and speak directly. Try “telnet YOUR_SERVER_IP 25” And the try typing in what the client says in the sample SMTP communication on wikipedia with the example address changed to the domain that you want to test. If that works, but sending email didn’t, you’ll need to investigate your DNS setup.

Handling the Rails Action

The target url I put in my mail script was http://www.craigambrose.com/emails, so the mail is going to get POSTed to that resource. With normal rails resource routes, that means that we’re expecting to handle the email in the create action of the EmailsController. That seems very sensible to me. My script puts the unparsed email into params[:email].

To parse it with TMail, all you need to do is:

Alternatively you could pass it to the “receive” method of any ActionMailer derived class, which does the above automatically.

I’ve had some reports that TMail is both a little slow, and also not quite up to parsing all the possible ways that an email might be encoded in the big bad world. That’s a subject for another blog post.

Final Performance Note

When postfix is calling your script, it makes so that only a certain number of calls are occurring concurrently, the default is 20, which seems pretty good to me. If you’d like to tweak this, use the following setting in main.cf (and don’t forget to reload postfix afterwards).

Acknowledgments

Setting up servers is not my area of expertise. Many thanks to Andrew Snow of Octopus for the postfix help and Pete Yandell for sharing some of the lessons learned on his great mailing list site 9cays