Starling
- written by Blaine Cook at Twitter
- Starling is a Message Queue Server based on MemCached
- written in Ruby
- stores jobs in memory (message queue)
- Ruby client: for instance Workling
- documentation: some good tutorials, for example the railscast about starling and workling or this blog post about starling
- Starling is a Message Queue Server based on MemCached
- written in Ruby
- stores jobs in memory (message queue)
- Ruby client: for instance Workling
- documentation: some good tutorials, for example the railscast about starling and workling or this blog post about starling
Web applications often need to do work which does not belong in the request / response cycle. Photo resizing or API
calls to other services block the request, slowing things down
uncessarily. Long running tasks will make your user application
unresponsive and your users impatient, or even destabilize your
environment. The solution is to take these tasks out of the request
cycle.
Getting started with backgrounding should be really simple. Traditionally, backgrounding was done in Rails applications through use of BackgroundRB. But BackgroundRB has gained a bad reputation over time, because of Drb issues and its general weightiness. Although BackgroundRB was recently completely re-written, it is still only one of many options available at the moment. Unfortunately, it’s not really clear which option should be used.
In this talk I will survey what’s currently available. I will look at the queue based systems Starling and BeanstalkD, and also at systems that achieve a similar effect by using the database and threading or forking, BackgroundJob and spawn. Finally I will look at Workling, wich can combine all these approaches under a very simple roof.
The main focus will be on Starling, Twitters inhouse message queue, released by Blaine Cook and team a few months ago. Starling is a lightweight persistent message queue which speaks memcache. It is used at Twitter to distribute work such as sending SMS. I will look at how to use Starling, as well as differences in the evented version of Starling.
Next, I will discuss Workling, which seamlessly integrates Starling into your Rails app. I will show how Workling and Starling can be used to build a backgrounded addressbook crawler with an ajax progress indicator wich runs the worker code on a remote machine.
Finally I will demonstrate how the same code can be configured so that it runs locally with Spawn or also with BackgroundJob.
Getting started with backgrounding should be really simple. Traditionally, backgrounding was done in Rails applications through use of BackgroundRB. But BackgroundRB has gained a bad reputation over time, because of Drb issues and its general weightiness. Although BackgroundRB was recently completely re-written, it is still only one of many options available at the moment. Unfortunately, it’s not really clear which option should be used.
In this talk I will survey what’s currently available. I will look at the queue based systems Starling and BeanstalkD, and also at systems that achieve a similar effect by using the database and threading or forking, BackgroundJob and spawn. Finally I will look at Workling, wich can combine all these approaches under a very simple roof.
The main focus will be on Starling, Twitters inhouse message queue, released by Blaine Cook and team a few months ago. Starling is a lightweight persistent message queue which speaks memcache. It is used at Twitter to distribute work such as sending SMS. I will look at how to use Starling, as well as differences in the evented version of Starling.
Next, I will discuss Workling, which seamlessly integrates Starling into your Rails app. I will show how Workling and Starling can be used to build a backgrounded addressbook crawler with an ajax progress indicator wich runs the worker code on a remote machine.
Finally I will demonstrate how the same code can be configured so that it runs locally with Spawn or also with BackgroundJob.
