Conference keynotes often introduce plenty of new ideas and challenges to a development community, but they are a bit of a one way dialogue if we don’t have a think about them and share our views. In particular Dave Thomas, of the Pragmatic Programers, is not shy about trying to inspire programmers to take new directions, and I always look forward to his keynotes in particular.

For RailsConf 07, Dave’s talk centered around the idea of “cargo cults”. Cargo cults were popular on Pacific island nations, who emulated the activities of the Americans during the war, to the point of building runways, control towers, and trying to summon planes with coconut headphones, in order to make the cargo come. The metaphor here is that we, as rails developers, need to stop doing things just because they have worked for others in the past, and start being truly creative.

While it’s hard to disagree with a call to be creative, I’m going to go out on a limb and say that I don’t think this is a particularly good direction for the Rails community to take at present.

Ruby on Rails has several strengths that make it quite unique. Some of these relate to the software itself, but these are minor compared to the strengths of the Rails community. There have been plenty of other open source ecosystems with a strong community, so what makes the Rails one different to, for example, the python community?

My belief, if you’ll excuse me for stretching David’s metaphor perhaps a little too far, is that it’s our cargo cults. The rails community demonstrates a particular unity of purpose. While we are all out building our own crazy web applications, the opportunities for re-use of code and design ideas is particularly high. If you ask three python programmers how to build web applications, you’ll get three different answers. If you ask three ruby programmers, statistically speaking, you’re liable just to hear about Rails three times.

This goes further than just the dominance of Rails within the Ruby community. Even within Rails, we have our cargo cults. REST is a good example, and one mentioned during David’s talk. Sure, there’s life beyond REST, but the benefit of picking such an approach as a community, and sticking to it (at least for the next year or two), is staggering. If I build a RESTful application, it’s no big deal, but if Rails developers everywhere build them, then suddenly there’s an ecosystem of useful applications that can talk to each other in such a practical and simple way that it makes Soap developers weep.

Not only do we have our cargo cults, but we also have our high priests. The whole REST thing happened because at RailsConf 06, DHH said that we should do it. This year, he was a little more reserved, but was still quite happy to go out on a limb and pick some particular technologies which were more right than others. One slide of his talk listed the friends and allies of the rails community. Our friends, we are told, include OpenID and Atom. There’s no reason why the Ruby on Rails framework need to be particularly associated with either of these technologies, and from a technical perspective, it probably wont be. But DHH says that they are our friends. Suddenly, OpenID support is appearing in Rails apps all over the place. This is an amazing result which you can normally only see happen within a hierarchical organisation. Here we are seeing unity of purpose within a very anarchistic open source community.

So I say, investigating new ideas is fine, but be aware that there is great benefit to be gained by following the path that we’re all traveling on together. People with different ideas from the Rails core team tend to write plugins, and if people think that the ideas are good enough, they eventually gain traction. Just take a look at the adoption of rspec within the rails community, even though it is quite different from the default testing system.

You can think of this as a cargo cult if you like, or laugh and say that rails developers have swallowed too much of their own hype. However, taking a common path that represents the combination of a range of views without simply all going our separate ways is a technique that works very well in nature.

Amongst humans, we call this technique consensus.

I’ve moved this blog from Typo, to Simplelog, both rails based blogging systems. Normally I don’t do this sort of meta-blogging, but I though I’d share a few tips about the migration, and some comments on Simplelog.

Installation

Like the other rails blogs, simplelog has some installation instructions for how to get the simplelog files on your sever, and like all rails developers, I promptly ignored them, and setup a copy of simplelog in a subversion repository of my own and deployed to my server using capistrano. This makes deployment and upgrading much easier, as I can test new simplelog versions on my local machine first, and then roll out updates using cap deploy as normal.

Migrating your Data

There don’t seem to be any nice scripts for migrating data from Typo to Simplelog. There are some scripts for exporting typo in data into Moveable Type or word press format, and you could probably hook together two or three different scripts to eventually get from Typo to Simplelog, but I couldn’t figure out a simple way to do that without also installing a PHP app, and that just seemed way too complex, so I thought I’d roll my own.

The bad news I’m afraid, is that I haven’t bothered to sufficiently generalise it to make it work for everyone, but if you want to do this too, then you can use my code as a starting point and refine it for your needs.

When you want to script some little task in rails, the tool for the job is of course rake. I started working on a little rake task inside my typo installation, to export the data in a format I could use for simplelog. I started by comparing both database schemas, and seeing how I could generate SQL to use with simplelog.

It turns out that both these databases are non-trivial. They use single-table inheritance, and the also both have cache columns full of keywords and search text and so forth. The active record classes in both apps have the code to do this, so the best way to convert your data is to leverage that code. You could write a ruby app that loads both model layers and connects to both database, but that didn’t seem too trivial, so what I ended up doing was creating a typo rake export task that generated a simplelog rake import task.

The Code

Here’s the code, it only copies the data I was interested in (articles and comments), and it does so with a few specific values (such as a fixed author id). It also, on large databases, will hit some id conflicts as it does so. To fix this, run it multiple times until the auto-numbered ids become bigger than the ids in your old database (or solve that problem properly). Also, it’s butt ugly, remember, I just wanted to do this in an hour and never see it again.

Before running this script, make sure that you have setup your simplelog database (with the migrations), and created a user (my had id 2, which is hardcoded in the script). Also, set the default markup format in simplelog to be whatever you used in typo (I was using textile).

def q(value)
  value = '' if value.nil?
  if value.is_a? String
    escaped = value.gsub(/\\/, '\\\\').gsub(/\r\n/, "\n").gsub(/'/, "\\\\'").gsub(/\n/, '\' + "\\\\n" + \'')
    return "'#{escaped}'" 
  end
  value
end

task :typo_export => [:environment] do
  file = File.new("/path_to_craigs_simplelog/lib/tasks/import_from_typo.rake", 'w')  file << "task :import_from_typo => [:environment] do \n" 
  file << "  require 'find'\n" 
  file << "  require 'post'\n" 
  file << "  require 'preference'\n" 
  file << "  require 'application'\n"   file << "  Post.delete_all\n" 
  file << "  update_posts = \"\"\n" 
  for article in Article.find(:all)
    file << "  new_record = Post.create(:author_id => 2, :created_at => '#{article.created_at.to_s(:db)}'.to_time, :modified_at => '#{article.updated_at.to_s(:db)}'.to_time, :permalink => #{q article.permalink}, :title => #{q article.title}, :body_raw => #{q article.body}, :comment_status => 1, :text_filter => #{q 'textile'})\n" 
    file << "  update_posts += \"UPDATE posts SET id = #{article.id} WHERE id = \#{new_record.id}; \"\n\n" 
  end
  file << "  Post.connection.execute update_posts\n\n"   file "\n\n" 
  file << "  Comment.delete_all\n" 
  file << "  update_comments = \"\"\n"   for comment in Comment.find(:all)
    file << "  rew_record = Comment.create(:post_id => #{comment.article.id}, :name => #{q comment.author}, :email => #{q (comment.email.blank? ? 'anon@noemail.com' : comment.email)}, :subject => '', :body_raw => #{q comment.body}, :ip => #{q comment.ip}, :is_approved => 1, :url => #{q comment.url})\n" 
    file << "  update_comments += \"UPDATE comments SET id = #{comment.id}, is_approved = 1 WHERE id = \#{new_record.id}\"\n\n" 
  end
  file << "  Post.connection.execute update_comments\n\n"   file << "end\n" 
end

Once this is run, you can go to the simplelog installation and run rake import_from_typo. If it doesn’t work, figure out what went wrong, modify the script, and repeat. Please notice that this is a destructive import (note the calls to “delete_all”).

This is a long post already, so I’ll post comments on simplelog next time.

The first time a Ruby on Rails release introduced a few backwards incompatible changes, it caused a bit of an uproar (the one I’m thinking of was 1.0.1, which broke Typo, amongst other things). Suddenly, everyone realised that we were writing web applications that weren’t just dependent on Ruby on Rails, they were dependent on a particular version of Rails.

So, these days we all know that we need to lock our Rails applications into using a set version. There are two ways of doing this, and each has its pros and cons.

[1] Freezing Rails in the Vendor Directory

If your application finds a copy of Rails in the ./vendor/rails directory, then it will use that instead of whatever rails gems are installed on the system. This is really handy, and it’s the method that I initially adopted for all my sites when the rails 1.0.1 problem occurred. At first, I was copying the Rails files into my project subversion repositories, but I quickly learned that the easier way to do it was with subversion externals.

From the root of your rails application, execute:

svn propedit svn:externals vendor/

The subversion externals properties from that directory will now be editable in your default editor. Each line in this folder represents a link to an external repository, with the name of the local folder to export to first, then the repository URL. So, for example, to use rails version 1.2.3, we would use the following.

rails http://svn.rubyonrails.org/rails/tags/rel_1-2-3/

You can find the URL of that tag, or any other rails tag or branch (or the trunk itself) by browsing the rails repository.

The upside of this method, is that your application is safe to deploy on almost any machine with the correct ruby stack installed, even if the rails gems are not present.

The downside, is that rails is pretty large, and every time you do a svn checkout, it has to grab it all. In particular this slows down your svn deploy command. If your using deprec, it can really slow down your svn setup command too, because the file permissions have to be set on all those folders.

[2] Specifying the Rails Version in environment.rb

This is the accepted method now, and is done by default in all new rails applications. To lock in that same version number, all you need to do is add the following to your environment.rb, if it isn’t already present:

RAILS_GEM_VERSION = '1.2.3' unless defined? 

This means that rails will load the gem for rails 1.2.3, if it exists. If it doesn’t exist, it will throw an error, rather than run your application. Remember that gem doesn’t remove old versions when it installs new ones, so even if a newer version of rails is installed on the server, if the correct version was there once, it should stay better.

These days, I’m moving my sites over to using this method, for the reasons of hard disk space and speed that I mentioned above.

The only downside is, I need to make sure that the right version of the gem is installed. However, most applications have other gem dependencies too, apart from just rails.

How do you ensure that the required gems are installed on a new server? If you’re logging in to the server manually and installing the gems, have a think about automation. Surely this is the province of the cap setup task. It might seem easy to do it now, but don’t forget that you might have to do it again when you move servers in six months time. Also, when that happens, you might be in a hurry. You might also be migrating to a multi-server cluster.

Don’t wait, automate now. Wack all your setup needs into cap tasks. Consider using before or after filters on the cap setup action. Please note that if you’re using deprec, it already adds these filters, so you’ll need to use an action called after_after_setup. This problem is fixed in capistrano 2, which should be out as a stable release very soon, but until then, you can do it the hard way.

For a long time, on many of my Rails apps, I’ve been letting my Mongrel processes run as the root user. Any sysadmin worth her salt will tell you that this is a bad idea, because any exploits in you web application will give the malicious user root access to your machine, and also any bugs in your application have the potential to destroy all sorts of important data. For example, try adding <% rm_rf ’/’ %> to one of your rails templates and you’ll see what I mean (that’s sarcasm by the way, don’t do it).

So, we need to run those mongrel processes with a user with less privileges. Our mongrel user should be able to read our entire rails application, and only be able to write to those relevant directories, such as logs, cache, sessions, pids and so on.

This security problem was a significant issue for my current client, and so we hired Mike Bailey (the guy who writes the deprec gem), to ensure that deprec has
a system for handling this properly. It’s in the latest deprec release, and here’s how it works.

Deprec 1.7

Deprec has had a whole swathe of small releases in the last couple of weeks. Mike is back from his adventures in india, and busy fixing bugs and adding new features. By the time you read this, the release number might be even bigger, but these instructions on how to apply to old sites will probably still be valid. For new sites, you don’t really have to worry about this stuff.

Before Upgrading

Make sure that your gems are up to date on the server.

Why do you need to do this? Well it turns out that mongrel 1.0.1 came out recently, and at the moment, deprec can’t correctly answer the questions that gem will ask about which version to use when upgrading. If you hit this problem, or want to avoid it, log into your server(s) and run sudo gem update now. You can decide which version of mongrel you want (probably 2 – ruby) and then deprec wont have any troubles.

On an app that’s been around for a while you might also want to run cap cleanup, if you don’t already do this every now and then to remove old releases on the server. This will make the following tasks run much faster.

Applying the New Groups and Permissions

Remember the cap setup task that sets up your initial directory structure for an application? Deprec also adds to that task to ensure that file permissions are correct, and the mongrel cluster set up. This task is re-runnable. It doesn’t hurt to run it as often as you want, although it is pretty slow. By running it with the new deprec release, you can setup the right file permissions to use mongrel as a non-root user.

Please note that the last step performed by the setup task is to create the database. This currently fails, as the database already exists. This bug will get fixed, but it doesn’t matter, just ignore the error.

Roll Out a New Version

To get your app rolled out with the new permissions, run cap deploy, even if it hasn’t changed since the last release.

Consider Updating Your Rails Stack

If you setup your machine with deprec some time ago, as I did, then there will be a few bugs. In particular there was an old bug where correct initialisation scripts for mongrel and apache where not added to the default runlevel, and they didn’t start on launch. To make sure that your system is configured as per the latest deprec, run: cap install_rails_stack. This is also always re-runnable to bring your system up to date.

Summary

So, for upgrading an old server to the latest deprec, my recommendations are:

# first, login to the server and run "sudo gem update" 
cap cleanup
cap setup
cap deploy
cap install_rails_stack
cap restart_apache

For setting up a new server, instructions vary slightly according to your host, but for an Ubuntu 6.0.6 server at slicehost.com, I use the following process:

export HOSTS=your.hostname.com
cap setup_admin_account_as_root
cap setup_ssh_keys
cap install_rails_stack
cap setup
cap deploy_with_migrations
cap restart_apache

Post continues, click to read more...