delayed_job (and other daemons) in a production environment

Posted by Michael Reinsch on 2010-08-16

The delayed_job plugin for Rails does a good job for pushing tasks that take some time to process into the background, so that your users (and your Rails processes) can do other things than to wait. It uses daemons to process the backgrounded tasks, so for your system to work correctly it is essential that those daemons are running. Thus you want to make sure that those daemons are getting started when the server boots and are restarted in case they die. Those points are not addressed by delayed_job.

The usual way to get processes started at boot time under Linux is to use an init.d script. But init.d scripts only address the boot process - if the daemon dies, it won't get restarted.

D.J. Bernstein's daemontools make it very simple to create system services which achieve both of the above points: starting your daemons at system boot time and restarting them in case they die. And the best feature: to create a new service, you don't even need to include any of the typical daemon features (such as backgrounding the process) into your program. So while delayed_job uses the daemons library to provide those features, we won't be needing those.

The following steps show how to set up a new service. This assumes that you already installed daemontools (available for Ubuntu/Debian for instance: apt-get install daemontools-run).

1. Create a new directory. This is going to be the service directory.
2. Create a shell script "run" in the service directory which runs your program. This can be as simple as exec /path/to/my/program.
3. Create a symlink from the system service directory (for Ubuntu that'd be /etc/service), pointing to your new service directory.

That's it. To control your service, use the svc tool. See the manpage for more information.

In case of delayed_job, we are using the following run script:

#!/bin/sh
export RAILS_ENV=production
exec 2>/dev/null
exec setuidgid railsuser /srv/railsuser/project/current/script/delayed_job run

The script changes the user to "railsuser" (you don't want to run your delayed job processing under root; change it to match your setup), and then starts the usual delayed_job script, telling it to not put itself into the background.

One specialty to note is the handling of stderr. We redirect it to /dev/null to avoid potential "Broken pipe" exceptions in case something writes to stderr, which isn't available. Redirecting sdterr to stdout did not work.

Now, when updating, you will want to restart the delayed_job service. With capistrano we use the following task definitions:

namespace :delayed_job do
  desc "Start delayed job (if not running)"
  task :start, :roles => :app do
    sudo "svc -u /etc/service/#{application}_#{rails_env}_delayed_job"
  end
  desc "Stop delayed job"
  task :stop, :roles => :app do
    sudo "svc -d /etc/service/#{application}_#{rails_env}_delayed_job"
  end
  desc "Restart delayed job"
  task :restart, :roles => :app do
    sudo "svc -t /etc/service/#{application}_#{rails_env}_delayed_job"
  end
end

after "deploy:start", "delayed_job:start" 
after "deploy:stop", "delayed_job:stop" 
after "deploy:restart", "delayed_job:restart"

This requires that your deployment user will be able to run svc using sudo, so make sure to add this to your sudoers.

Also note that for the service names in the system service directory we use the pattern #{application}_#{rails_env}_delayed_job. Those are links to the service's directory, which are located under /srv/railsuser/project/services for our setup.

With this setup we have a pretty reliable delayed_job, and can use the same framework to run most (if not all) other services we might need with very little effort.