Message Queue Starling

Starling

- written by Blaine Cook at Twitter
- Starling is a Message Queue Server based on MemCached
- written in Ruby
- stores jobs in memory (message queue)
- Ruby client: for instance Workling
- documentation: some good tutorials, for example the railscast about starling and workling or this blog post about starling

Web applications often need to do work which does not belong in the request / response cycle. Photo resizing or API calls to other services block the request, slowing things down uncessarily. Long running tasks will make your user application unresponsive and your users impatient, or even destabilize your environment. The solution is to take these tasks out of the request cycle.
Getting started with backgrounding should be really simple. Traditionally, backgrounding was done in Rails applications through use of BackgroundRB. But BackgroundRB has gained a bad reputation over time, because of Drb issues and its general weightiness. Although BackgroundRB was recently completely re-written, it is still only one of many options available at the moment. Unfortunately, it’s not really clear which option should be used.
In this talk I will survey what’s currently available. I will look at the queue based systems Starling and BeanstalkD, and also at systems that achieve a similar effect by using the database and threading or forking, BackgroundJob and spawn. Finally I will look at Workling, wich can combine all these approaches under a very simple roof.
The main focus will be on Starling, Twitters inhouse message queue, released by Blaine Cook and team a few months ago. Starling is a lightweight persistent message queue which speaks memcache. It is used at Twitter to distribute work such as sending SMS. I will look at how to use Starling, as well as differences in the evented version of Starling.
Next, I will discuss Workling, which seamlessly integrates Starling into your Rails app. I will show how Workling and Starling can be used to build a backgrounded addressbook crawler with an ajax progress indicator wich runs the worker code on a remote machine.
Finally I will demonstrate how the same code can be configured so that it runs locally with Spawn or also with BackgroundJob.

What is Rails?

== Welcome to Rails

Rails is a web-application and persistence framework that includes everything
needed to create database-backed web-applications according to the
Model-View-Control pattern of separation. This pattern splits the view (also
called the presentation) into "dumb" templates that are primarily responsible
for inserting pre-built data in between HTML tags. The model contains the
"smart" domain objects (such as Account, Product, Person, Post) that holds all
the business logic and knows how to persist themselves to a database. The
controller handles the incoming requests (such as Save New Account, Update
Product, Show Post) by manipulating the model and directing data to the view.

In Rails, the model is handled by what's called an object-relational mapping
layer entitled Active Record. This layer allows you to present the data from
database rows as objects and embellish these data objects with business logic
methods. You can read more about Active Record in
link:files/vendor/rails/activerecord/README.html.

The controller and view are handled by the Action Pack, which handles both
layers by its two parts: Action View and Action Controller. These two layers
are bundled in a single package due to their heavy interdependence. This is
unlike the relationship between the Active Record and Action Pack that is much
more separate. Each of these packages can be used independently outside of
Rails.  You can read more about Action Pack in
link:files/vendor/rails/actionpack/README.html.


== Getting Started

1. At the command prompt, start a new Rails application using the rails command
   and your application name. Ex: rails myapp
   (If you've downloaded Rails in a complete tgz or zip, this step is already done)
2. Change directory into myapp and start the web server: script/server (run with --help for options)
3. Go to http://localhost:3000/ and get "Welcome aboard: You’re riding the Rails!"
4. Follow the guidelines to start developing your application


== Web Servers

By default, Rails will try to use Mongrel and lighttpd if they are installed, otherwise
Rails will use WEBrick, the webserver that ships with Ruby. When you run script/server,
Rails will check if Mongrel exists, then lighttpd and finally fall back to WEBrick. This ensures
that you can always get up and running quickly.

Mongrel is a Ruby-based webserver with a C component (which requires compilation) that is
suitable for development and deployment of Rails applications. If you have Ruby Gems installed,
getting up and running with mongrel is as easy as: gem install mongrel.
More info at: http://mongrel.rubyforge.org

If Mongrel is not installed, Rails will look for lighttpd. It's considerably faster than
Mongrel and WEBrick and also suited for production use, but requires additional
installation and currently only works well on OS X/Unix (Windows users are encouraged
to start with Mongrel). We recommend version 1.4.11 and higher. You can download it from
http://www.lighttpd.net.

And finally, if neither Mongrel or lighttpd are installed, Rails will use the built-in Ruby
web server, WEBrick. WEBrick is a small Ruby web server suitable for development, but not
for production.

But of course its also possible to run Rails on any platform that supports FCGI.
Apache, LiteSpeed, IIS are just a few. For more information on FCGI,
please visit: http://wiki.rubyonrails.com/rails/pages/FastCGI


== Debugging Rails

Sometimes your application goes wrong.  Fortunately there are a lot of tools that
will help you debug it and get it back on the rails.

First area to check is the application log files.  Have "tail -f" commands running
on the server.log and development.log. Rails will automatically display debugging
and runtime information to these files. Debugging info will also be shown in the
browser on requests from 127.0.0.1.

You can also log your own messages directly into the log file from your code using
the Ruby logger class from inside your controllers. Example:

  class WeblogController < ActionController::Base
    def destroy
      @weblog = Weblog.find(params[:id])
      @weblog.destroy
      logger.info("#{Time.now} Destroyed Weblog ID ##{@weblog.id}!")
    end
  end

The result will be a message in your log file along the lines of:

  Mon Oct 08 14:22:29 +1000 2007 Destroyed Weblog ID #1

More information on how to use the logger is at http://www.ruby-doc.org/core/

Also, Ruby documentation can be found at http://www.ruby-lang.org/ including:

* The Learning Ruby (Pickaxe) Book: http://www.ruby-doc.org/docs/ProgrammingRuby/
* Learn to Program: http://pine.fm/LearnToProgram/  (a beginners guide)

These two online (and free) books will bring you up to speed on the Ruby language
and also on programming in general.


== Debugger

Debugger support is available through the debugger command when you start your Mongrel or
Webrick server with --debugger. This means that you can break out of execution at any point
in the code, investigate and change the model, AND then resume execution! Example:

  class WeblogController < ActionController::Base
    def index
      @posts = Post.find(:all)
      debugger
    end
  end

So the controller will accept the action, run the first line, then present you
with a IRB prompt in the server window. Here you can do things like:

  >> @posts.inspect
  => "[#nil, \"body\"=>nil, \"id\"=>\"1\"}>,
       #\"Rails you know!\", \"body\"=>\"Only ten..\", \"id\"=>\"2\"}>]"
  >> @posts.first.title = "hello from a debugger"
  => "hello from a debugger"

...and even better is that you can examine how your runtime objects actually work:

  >> f = @posts.first
  => #nil, "body"=>nil, "id"=>"1"}>
  >> f.
  Display all 152 possibilities? (y or n)

Finally, when you're ready to resume execution, you enter "cont"


== Console

You can interact with the domain model by starting the console through script/console.
Here you'll have all parts of the application configured, just like it is when the
application is running. You can inspect domain models, change values, and save to the
database. Starting the script without arguments will launch it in the development environment.
Passing an argument will specify a different environment, like script/console production.

To reload your controllers and models after launching the console run reload!


== Description of Contents

app
  Holds all the code that's specific to this particular application.

app/controllers
  Holds controllers that should be named like weblogs_controller.rb for
  automated URL mapping. All controllers should descend from ApplicationController
  which itself descends from ActionController::Base.

app/models
  Holds models that should be named like post.rb.
  Most models will descend from ActiveRecord::Base.

app/views
  Holds the template files for the view that should be named like
  weblogs/index.erb for the WeblogsController#index action. All views use eRuby
  syntax.

app/views/layouts
  Holds the template files for layouts to be used with views. This models the common
  header/footer method of wrapping views. In your views, define a layout using the
  layout :default and create a file named default.erb. Inside default.erb,
  call <% yield %> to render the view using this layout.

app/helpers
  Holds view helpers that should be named like weblogs_helper.rb. These are generated
  for you automatically when using script/generate for controllers. Helpers can be used to
  wrap functionality for your views into methods.

config
  Configuration files for the Rails environment, the routing map, the database, and other dependencies.

db
  Contains the database schema in schema.rb.  db/migrate contains all
  the sequence of Migrations for your schema.

doc
  This directory is where your application documentation will be stored when generated
  using rake doc:app

lib
  Application specific libraries. Basically, any kind of custom code that doesn't
  belong under controllers, models, or helpers. This directory is in the load path.

public
  The directory available for the web server. Contains subdirectories for images, stylesheets,
  and javascripts. Also contains the dispatchers and the default HTML files. This should be
  set as the DOCUMENT_ROOT of your web server.

script
  Helper scripts for automation and generation.

test
  Unit and functional tests along with fixtures. When using the script/generate scripts, template
  test files will be generated for you and placed in this directory.

vendor
  External libraries that the application depends on. Also includes the plugins subdirectory.
  This directory is in the load path.

source: https://github.com/garalo/ecompages-cm 

Rails Ruby

But What Does It All Mean?

“Ruby on Rails” is catchy but confusing. Is Rails some type of magical drug that Ruby is on? (Depending on who you ask, yes.)

Ruby is a programming language, similar to Python and Perl. It is dynamically typed (no need for “int i”), interpreted, and can be modified at runtime (such as adding new methods to classes). It has dozens of shortcuts that make it very clean; methods are rarely over 10 lines. It has good RegEx support and works well for shell scripting.

Rails is a gem, or a Ruby library. Some gems let you use the Win32 API. Others handle networking. Rails helps make web applications, providing classes for saving to the database, handling URLs and displaying html (along with a webserver, maintenance tasks, a debugging console and much more).

IRB is the interactive Ruby console (type “irb” to use). Rails has a special IRB console to access your web app as it is running (excellent for live debugging).

Rake is Ruby’s version of Make. Define and run maintenance tasks like setting up databases, reloading data, backing up, or even deploying an app to your website.

Erb is embedded Ruby, which is like PHP. It lets you mix Ruby with HTML (for example):

Hello there, <%= get_user_name() %>

YAML (or YML) means “YAML Ain’t a Markup Language” — it’s a simple way to specify data:

{name: John Smith, age: 33}

It’s like JSON, much leaner than XML, and used by Rails for setting configuration options (like setting the database name and password).

Phew! Once Ruby is installed and in your path, you can add the rails gem using:

gem install rails

In general, use gem install “gem_name”, which searches online sources for that library. Although Rails is “just another gem”, it is the killer library that brought Ruby into the limelight.

Understanding The Model-View-Controller Pattern

Rails is built around the model-view-controller pattern. It’s a simple concept: separate the data, logic, and display layers of your program. This lets you split functionality cleanly, just like having separate HTML, CSS and Javascript files prevents your code from mushing together. Here’s the MVC breakdown:

• Models are classes that talk to the databse. You find, create and save models, so you don’t (usually) have to write SQL. Rails has a class to handle the magic of saving to a database when a model is updated.
• Controllers take user input (like a URL) and decide what to do (show a page, order an item, post a comment). They may initially have business logic, like finding the right models or changing data. As your rails ninjitsu improves, constantly refactor and move business logic into the model (fat model, skinny controller). Ideally, controllers just take inputs, call model methods, and pass outputs to the view (including error messages).
• Views display the output, usually HTML. They use ERB and this part of Rails is like PHP - you use HTML templates with some Ruby variables thrown in. Rails also makes it easy to create views as XML (for web services/RSS feeds) or JSON (for AJAX calls).

The MVC pattern is key to building a readable, maintainable and easily-updateable web app.

Understanding Rails’ Directory Structure

When you create your first rails app, the directories are laid out for you. The structure is well-organized: Models are in app/models, controllers in app/controllers, and views in app/my_local_views (just kidding).

The naming conventions are important – it lets rails applications “find their parts” easily, without additional configuration. Also, it’s very easy for another programmer to understand and learn from any rails app. I can take a look at Typo, the rails blogging software, and have a good idea of how it works in minutes. Consistency creates comprehension.

What is Ruby on Rails?

Ruby on Rails is a Model-View-Controller framework for creating database-driven websites in Ruby. That's a pretty jargon-heavy explanation of what Rails is, but, believe it or not, it's easier to understand what Rails is once you understand the pieces. At its core, Rails is built on simple concepts.

 

What is a Model-View-Controller Framework?

Model-View-Controller (or MVC) is a way of organizing your code to hide complexity and contain related code. It will separate a web application into three primary sections: the model, the controller and the the view. The model handles all the database interaction, the controller handles all the web server interaction and the view generates the HTML code that is actually displayed in the browser.

Keeping everything in its place is a big step toward making complex web applications easy to implement. If designed correctly, an MVC application in Rails will never have code in the Controller accessing the database and the View will never talk to the web server.

The Model's Job in Ruby on Rails

The Model abstracts database "objects" in your web application. So, for example, if your application is an online store, the "objects" could be items for sale, the customers or the user reviews. The Model section of the Ruby code is responsible for storing and retrieving all objects from the database as well as dealing with any "data integrity" issues. For instance, if you tried to save an item with a negative price, the code in the Model should check for that discrepancy and refuse to save it until the problem is fixed.

The Controller's Job in Ruby on Rails

The Controller is primarily what sits between the Model and the View. In its simplest form, the Controller fetches objects from the database using the Model and then hands them to the View to be rendered. It has direct access to the web server and all cookie and session variables. A Controller has different "actions" that the application can perform, such as adding an item to your cart or viewing an item.

The View's Job in Ruby on Rails

The View takes the data handed to it by the Controller and produces the HTML output for the web browser. All formatting and styles occur in the View, leaving the Model and Controller to focus on their specific jobs. The View is usually a mix of HTML and Ruby code. This type of code is a lot like PHP (where scripting code is embedded in HTML), which makes it easy to produce the formatting and layout tags needed in a web page and to add the dynamic content from the Model without relying on a third formatting language.

The MVC in Action

The entire process of accessing the example online shop works something like this. A web browser sends a request to the server for a URL, such as /items/show/45. This means the web browser wants to access the Show action of the Items controller. Rails sees this request, starts the Items controller and runs the Show action.

The Show action then uses the Model to fetch the item with the ID number of 45. At the same time, the Show action will also gather any other pertinent variables--such as login information and the cart with items currently in it--and pass all of these objects from the Model onto the View. The View uses mixed Ruby and HTML to produce HTML code with the information from the fetched objects. Rails captures this HTML output and sends it back to the web browser which displays the generated page.

MVC with Rails

Ruby on Rails takes all these concepts and bundles them into a single package for web developers. Every component of the MVC paradigm is implemented in Ruby and every effort is made to abstract and hide unnecessary complexity from the progammer. Rails developers only need to install the rails gem and they'll have a complete development environment for writing Rails web applications as well as a web server written in Ruby for running them.