Using TheSchwartz (Job Queue) with Catalyst

TheSchwartz is a job queue module written by the guy(s) behind LiveJournal. It simply allows you to add jobs to a queue, in this case a database, and then have scripts watching the queue (database) and running work tasks when jobs arrive that they know how to deal with.

I wrote down how I used it with Catalyst because there’s not a whole lot of documentation about how to use it available.

1. force install TheSchwartz – the module fails tests and according to a message on a thread it’s fine to force install, I haven’t had any problems.

2. You have to create the required database tables, which is mentioned in some hidden documentation.

3. Then what I did was to create a wrapper class for TheSchwartz so that I could invoke it from inside Catalyst and from my worker scripts outside of Cataylst. Mine looked like this, it may well be you know a better way of writing it:

package Norris::TheSchwartzWrapper;

use strict;
use warnings;
use base qw /TheSchwartz/;

sub new {
    my $class = shift;

    ## todo make the non database options OPTIONS
    return $class->SUPER::new(
        databases => [ {
            dsn => 'dbi:mysql:norris_jobs',
            user => 'root',
            pass => ''
        } ],
        verbose => 1,
        driver_cache_expiration => 300,
    );
}

1;

If you copy that you might wanna actually do the refactor allowing the passing of options.

3. Following a discussion on a mailing list I created a model wrapper [for the wrapper] for TheSchwartz. The post talks about wrapping TheScwhartz object directly but because we’ve already decoupled TheSchwartz from out Catalyst application we wrap the wrapper…

Basically, “use” your wrapper and then create a model with the following:

sub COMPONENT {
    return Norris::TheSchwartzWrapper->new();
}

That allows you to do $c->model(‘Model’)->insert() and access TheSchwartz API.

4. At this point it didn’t work for me – MAKE SURE YOU HAVE SOME DB DRIVERS INSTALLED – I did not and hence inserting jobs was failing.

5. Insert jobs from a controller or wherever:

$c->model('TheSchwartz')->insert( 'Norris::Scanner::Crawler', $job_args ); 

6. Create worker scripts.

7. Sit back and relax!

Related posts:

1. Open Melody, Homebrew, Catalyst/Mojo and Sinatra.
2. Mad-Lib Perl Snippet
3. Become a Perl rockstar… AKA Perl link dump

Tagged as: catalyst, perl