reevoolabs : open source technology

Whilst there are already a number of options for collecting exceptions from your production Rails apps (such as exception_notifier), none of them are designed for collecting exceptions asynchronously. This means you are introducing a potential bottleneck in your request loop should any part of your exception collection process (e.g. sending the exception to a mail server) slow down for any reason.

At Reevoo, we already use beanstalkd for a number of applications so it seemed like a perfect fit for collecting our exceptions. The end result of this was our exception_messaging plugin which builds on top of our beanstalk_messaging plugin allowing you to configure your Rails app to send exceptions (along with supporting request, environment and session data) to a beanstalkd queue.

We’ve had this running in production for a while now but we weren’t really doing anything useful with the data. We started work on a basic Rails application that consumed the messages from the queue and stored them in a database with the idea being that we could build some basic reporting tools on top of this data, but what with work being work, we never found the time to do this properly.

Enter Hoptoad

Hoptoad is a third-party service that you can use to post your exception data to. It is exactly the kind of app we would have liked to have built. Fortunately for us, somebody has already done the hard work. It collects and aggregates your exception data and presents it in a easy to use interface. Oh, and currently, it’s free.

Hoptoad provide a simple Rails plugin that you can drop in to your Rails project for synchronous integration with their service. They also publish a RESTful API for retrieving and posting data.

Out of the box, the plugin is probably fine for most users. For high-traffic websites where performance is vital and/or you want to avoid creating a dependency on a third-party service, you want a more robust solution. This is where Beanstalkd comes in.

Rather than re-inventing the wheel, I wanted to reuse the existing plugin code to send the exceptions collected from our queue; however, the parts of the plugin code that I wanted to use (the code that wrapped around their API and submitted exception data) was tightly coupled to the Rails integration and configuration process. Fortunately, the code was hosted on GitHub so I created a fork and began work on refactoring their plugin into loosely coupled components. At the time of writing, much of the code has been refactored and can be used in a standalone manner although there is still some work left to do. I’m hoping it will be merged into the master repository by the guys at Hoptoad (who have been very receptive to my enquiries and suggestions). You can grab the refactored plugin from a branch in my GitHub fork.

Consuming queued exceptions and sending them to Hoptoad

Once the refactoring work was done, sending our collected exception data was fairly trivial. I had to make a few changes to our exception_messaging plugin to make sure we were collecting all of the data we needed (simplifying a lot of the code in the process) but once we were collecting exception data in the correct format, all that was left to do was to write a simple Ruby script that used the updated plugin and a Beanstalk::Poller (courtesy of our beanstalk_messaging plugin) to process the messages:

require 'beanstalk_configuration' # sets up $queue_manager
require 'hoptoad/standalone'

Hoptoad.configure do |config|
  config.api_key = 'your_projects_hoptoad_api_key'
end

notifier = Hoptoad::Notifier.from_config(Hoptoad.config)

Beanstalk::QueuePoller.new($queue_manager).poll(:the_exceptions_queue) do |message|
  notifier.notify(message.ybody)
end

For more information on how to poll Beanstalk queues, see the instructions for the beanstalk_messaging plugin. To configure your app to use beanstalkd for collecting exceptions, see the instructions for the exception_messaging plugin.

Any problems or questions? Leave a comment.