I just stumbled across this whole other Perl eco-system I wasn’t aware existed, which is very much a mirror of the exciting/interesting things going on in Ruby/Python land.

I tend to hear a lot about Ruby, and less about Perl, from hacker news and other places so I hadn’t seen any of this new Perl stuff.

I absolutely love Sinatra (a micro Ruby web DSL) but it turns out there is a perl port – Dancer! I haven’t had a chance to try it out yet but plan to ASAP.

Also, in Ruby land there is this whole RACK middleware layer thing which “provides a minimal interface between webservers supporting Ruby and Ruby frameworks.”

Well, turns out Perl has its own version to: PLACK. Exciting times!

Also, in Ruby land are a number of high performance application servers, Thin and Unicorn being two. Well guess what? We’ve got them too…!

Twiggy and Starman respectively.

I’d quite like to set up a little nginx proxying -> starman environment, with an application written in Dancer (or Catalyst).

Not only are there all these new HTTP based projects there’s a new cpan module installer – cpanminus – which has had some rave reviews.

These are exciting times to be involved in the Perl community. The Catalyst Book [aff] is an excellent reference manual about modern Perl web development and the up-coming Modern Perl Book promises to be equally enlightening.

Perhaps someone could implement rvm [oops fixed] for Perl, then I could easily try out the various Perl 5.x updates, or I should at least learn to use Local::Lib.

Honestly, I’m excited, stick around! Perl isn’t going anywhere…

[This was quite a rambling, whirl-wind tour of new Perl projects, that I've literally only just stumbled across. Subscribe [RSS] to find out more as I try them out for real over the next short while…].

Related posts:

1. Become a Perl rockstar… AKA Perl link dump
2. Open Melody, Homebrew, Catalyst/Mojo and Sinatra.
3. Mad-Lib Perl Snippet

I was trying to merge sort a list of files based on their last modified date when someone said to me “do a schwartzian”. Now, I had no idea what they were talking about so off to google I went.

It’s basically this (shamelessly ripped from the excellent Wikipedia article):

@sorted = map  { $_->[0] }
                  sort   { $a->[1] cmp $b->[1] }
                  map  { [$_, foo($_)] }
                           @unsorted;

Reading from right to left (or from the bottom to the top):

• the original list @unsorted is fed into a map operation that wraps each item into a (reference to an anonymous 2-element) array consisting of itself and the calculated value that will determine its sort order (list of item becomes a list of [item=>value]);
• then the list of lists produced by map is fed into sort, which sorts it according to the values previously calculated (list of [item, value] => sorted list of [item, value]);
• finally, another map operation unwraps the values (from the anonymous array) used for the sorting, producing the items of the original list in the sorted order (sorted list of [item, value] => sorted list of item).

The use of anonymous arrays ensures that memory will be reclaimed by the Perl garbage collector immediately after the sorting is done.

The computational savings obtained by the Schwartzian transform depend strongly on the structure of the inner function. For an efficient ordinary sort function, the number of invocations of the transform function goes from an average of 2nlogn to n; one should carefully consider on a case-by-case basis whether the extra implementation complexity is justified by this efficiency saving.

It’s worth noting that foo() is a method that returns the value that an element will be sorted against and that $a and $b are special Perl variables that you can read more about in the sort documentation.

Pretty neat!

Related posts:

1. Mad-Lib Perl Snippet

I stumbled across a blog post by digitalhobbit about building a little Sinatra App on top of the Twitter Streaming API.

I followed that guide and made the core of Tweetfactor a couple of weeks ago. Actually getting the thing deployed, however, was pretty involved for me, having never used Rack/Passenger/God etc. etc.

So the following are hopefully all the steps I took, and the appropriate config files to get stuff up and running:

1. So first off, I didn’t even have anywhere to host this app so I signed up to a 256 VPS from webbynode and then created an Ubuntu Rails Ready Stack (basically a one click install for an Rails Ready Ubuntu system).
2. Then I added a user and setup SSH and pub/priv keys from my laptop to the VPS.
3. Then I installed Capistrano – sudo gem install capistrano – for one click deploys of the app to the server. My code is on github in a private repo, so you’ll probably have to look through the Capistrano docs to edit this to your needs:

   
#========================
#CONFIG
#========================
default_run_options[:pty] = true
set :application, "tweetfactor.co.uk"
set :scm, :git
set :git_enable_submodules, 1
set :repository, "git@github.com:adamtaylor/Tweetfactor.git"
set :branch, "master"
set :ssh_options, { :forward_agent => true }
set :stage, :production
set :user, "xxxx"
set :scm_passphrase, "xxxxxxx"
set :use_sudo, false
set :runner, "deploy"
set :deploy_to, "/var/www/#{application}"
set :app_server, :passenger
set :domain, "www.tweetfactor.co.uk"
#========================
#ROLES
#========================
role :app, domain
role :web, domain
role :db, domain, :primary => true
#========================
#CUSTOM
#========================
namespace :deploy do
task :start, :roles => :app do
run "touch #{current_release}/tmp/restart.txt"
end
task :stop, :roles => :app do
# Do nothing.
end
desc "Restart Application"
task :restart, :roles => :app do
run "touch #{current_release}/tmp/restart.txt"
end
end


4. Then I had to download god, for process monitoring: sudo gem install god.
5. When I setup my Rails Ready Stack I told it to use Ruby Enterprise Edition, which means that my gems were installed in a funny place which I had to add to my $PATH.
6. With the help of this post I wrote a god init.d script as follows:

   
#!/bin/sh

   ### BEGIN INIT INFO
   # Provides: god
   # Required-Start: $all
   # Required-Stop: $all
   # Default-Start: 2 3 4 5
   # Default-Stop: 0 1 6
   # Short-Description: God
   ### END INIT INFO

   NAME=god
   DESC=god

   set -e

   # Make sure the binary and the config file are present before proceeding
   test -x /opt/ruby-enterprise/bin/god || exit 0

   # Create this file and put in a variable called GOD_CONFIG, pointing to
   # your God configuration file
   # test -f /etc/default/god && . /etc/default/god
   # [ $GOD_CONFIG ] || exit 0

   . /lib/lsb/init-functions

   RETVAL=0

   case "$1" in
   start)
   echo -n "Starting $DESC: "
   #/opt/ruby-enterprise/bin/god -c $GOD_CONFIG -P /var/run/god.pid -l /var/log/god.log
   /opt/ruby-enterprise/bin/god -c /etc/default/god -P /var/run/god.pid -l /var/log/god.log
   RETVAL=$?
   echo "$NAME."
   ;;
   stop)
   echo -n "Stopping $DESC: "
   kill `cat /var/run/god.pid`
   RETVAL=$?
   echo "$NAME."
   ;;
   restart)
   echo -n "Restarting $DESC: "
   kill `cat /var/run/god.pid`
   #/opt/ruby-enterprise/bin/god -c $GOD_CONFIG -P /var/run/god.pid -l /var/log/god.log
   /opt/ruby-enterprise/bin/god -c /etc/default/god -P /var/run/god.pid -l /var/log/god.log
   RETVAL=$?
   echo "$NAME."
   ;;
   status)
   /opt/ruby-enterprise/bin/god status
   RETVAL=$?
   ;;
   *)
   echo "Usage: god {start|stop|restart|status}"
   exit 1
   ;;
   esac

   exit $RETVAL

7. Then I setup a god config file in /etc/default/god – it has some code duplication so could probably be improved but it seems to work none-the-less:

   # run with: god -c /path/to/redis.god -D

   God.watch do |w|
   w.name = "redis-server"
   w.interval = 30.seconds # default
   w.start = "redis-server /var/www/tweetfactor.co.uk/current/redis.conf"
   w.stop = "kill `cat /var/run/redis.pid`"
   w.restart = "kill `cat /var/run/redis.pid`; redis-server /var/www/tweetfactor.co.uk/current/redis.conf"
   w.start_grace = 10.seconds
   w.restart_grace = 10.seconds
   w.pid_file = "/var/run/redis.pid"

   w.behavior(:clean_pid_file)

   w.start_if do |start|
   start.condition(:process_running) do |c|
   c.interval = 5.seconds
   c.running = false
   end
   end

   w.restart_if do |restart|
   restart.condition(:memory_usage) do |c|
   c.above = 30.megabytes
   c.times = [3, 5] # 3 out of 5 intervals
   end

   restart.condition(:cpu_usage) do |c|
   c.above = 30.percent
   c.times = 5
   end
   end

   # lifecycle
   w.lifecycle do |on|
   on.condition(:flapping) do |c|
   c.to_state = [:start, :restart]
   c.times = 5
   c.within = 5.minute
   c.transition = :unmonitored
   c.retry_in = 10.minutes
   c.retry_times = 5
   c.retry_within = 2.hours
   end
   end
   end

   God.watch do |w|
   w.name = "twitter_filter"
   w.interval = 30.seconds
   w.start = "cd /var/www/tweetfactor.co.uk/current/; ruby twitter_filter.rb start"
   w.stop = "cd /var/www/tweetfactor.co.uk/current/; ruby twitter_filter.rb stop"
   w.start_grace = 10.seconds
   w.restart_grace = 10.seconds
   w.pid_file = "/var/www/tweetfactor/releases/log/twitter_filter.pid"

   w.behavior(:clean_pid_file)

   w.start_if do |start|
   start.condition(:process_running) do |c|
   c.interval = 5.seconds
   c.running = false
   end
   end

   w.restart_if do |restart|
   restart.condition(:memory_usage) do |c|
   c.above = 30.megabytes
   c.times = [3, 5] # 3 out of 5 intervals
   end

   restart.condition(:cpu_usage) do |c|
   c.above = 30.percent
   c.times = 5
   end
   end

   # lifecycle
w.lifecycle do |on|
on.condition(:flapping) do |c|
c.to_state = [:start, :restart]
c.times = 5
c.within = 5.minute
c.transition = :unmonitored
c.retry_in = 10.minutes
c.retry_times = 5
c.retry_within = 2.hours
end
end
end


8. A quick chmod +x god /etc/init.d/god
9. And a sudo update-rc.d god defaults to have it added to startup.
10. I need to add a rackup config file in the root called config.ru so that Nginx/Passenger could handle the Sinatra app:
    
    require 'rubygems'
    require 'sinatra'

    require 'tweetfactor'

    set :environment, :production

    run Sinatra::Application


11. The last thing I had to do was modify my twitter_filter.rb from the original blog post so it could be daemonized, I messaged digitalhobbit for some help on this and he sent me a modified version which worked a charm. I also had to sudo gem install daemons

I had a fun few hours learning about a tonne of new stuff and I hope this may help you if you wanna deploy it too.

Any questions, shoot in the comments!

Related posts:

1. Kill A Process Using It’s PID File
2. Open Melody, Homebrew, Catalyst/Mojo and Sinatra.

Quick Tip: kill $(cat /var/run/redis.pid)

I needed to do that earlier and didn’t know how.

EDIT: Turns out you can just do this as well: kill `cat /var/run/redis.pid`

Related posts:

1. Finding which package a file belongs to on Debian
2. Running a unix command as a background process
3. Tweetfactor – An experiment using Ruby, Sinatra, Twitter and Redis

TheSchwartz is a job queue module written by the guy(s) behind LiveJournal. It simply allows you to add jobs to a queue, in this case a database, and then have scripts watching the queue (database) and running work tasks when jobs arrive that they know how to deal with.

I wrote down how I used it with Catalyst because there’s not a whole lot of documentation about how to use it available.

1. force install TheSchwartz – the module fails tests and according to a message on a thread it’s fine to force install, I haven’t had any problems.

2. You have to create the required database tables, which is mentioned in some hidden documentation.

3. Then what I did was to create a wrapper class for TheSchwartz so that I could invoke it from inside Catalyst and from my worker scripts outside of Cataylst. Mine looked like this, it may well be you know a better way of writing it:

package Norris::TheSchwartzWrapper;

use strict;
use warnings;
use base qw /TheSchwartz/;

sub new {
    my $class = shift;

    ## todo make the non database options OPTIONS
    return $class->SUPER::new(
        databases => [ {
            dsn => 'dbi:mysql:norris_jobs',
            user => 'root',
            pass => ''
        } ],
        verbose => 1,
        driver_cache_expiration => 300,
    );
}

1;

If you copy that you might wanna actually do the refactor allowing the passing of options.

3. Following a discussion on a mailing list I created a model wrapper [for the wrapper] for TheSchwartz. The post talks about wrapping TheScwhartz object directly but because we’ve already decoupled TheSchwartz from out Catalyst application we wrap the wrapper…

Basically, “use” your wrapper and then create a model with the following:

sub COMPONENT {
    return Norris::TheSchwartzWrapper->new();
}

That allows you to do $c->model(‘Model’)->insert() and access TheSchwartz API.

4. At this point it didn’t work for me – MAKE SURE YOU HAVE SOME DB DRIVERS INSTALLED – I did not and hence inserting jobs was failing.

5. Insert jobs from a controller or wherever:

$c->model('TheSchwartz')->insert( 'Norris::Scanner::Crawler', $job_args ); 

6. Create worker scripts.

7. Sit back and relax!

Related posts:

1. Open Melody, Homebrew, Catalyst/Mojo and Sinatra.
2. Mad-Lib Perl Snippet
3. Become a Perl rockstar… AKA Perl link dump

I just wanted to quickly highlight some interesting projects I’ve come across lately…

Open Melody

Open Melody is a fork of the popular blogging [/social media] platform Moveable Type. Its aim is to be completely backward compatible with Moveable Type but to have a more democratic development style.

One of the benefits of the Moveable Type system is that all posts are “published” rather than dynamically generated, i.e. from the posts a whole set of static html pages are created which can be served rather than database intense dynamic pages.

They want to encourage as much community participation as possible and all the code is on github for anyone to jump right in.

If nothing else they have a lovely looking website and I wish them all the best and look forward to seeing what becomes of this.

Homebrew

I love my MacBook Pro and OS X, I really do but there are times I miss being able to issue “sudo aptitude install apache” or whatever. Admittedly Fink [currently down] has worked well for me but I definitely plan to try out Homebrew, which is a new package manager for OS X. It was written by a guy I started stalkingfollowing on twitter because he works at last.fm!

Catalyst and Mojo

Mojo is a modern web framework written in Perl with an aim of being easy to install and having limited/no(?) dependencies. I actually applied to Google Summer of Code with Mojo but that’s another story…

Over the last week or two I’ve been working with Mojo’s big brother Catalyst. Catalyst is the opposite of Mojo in the “easy to install/lack of dependencies” but the guys in IRC are very helpful and the power and flexibility of Catalyst is awesome! It’s not as quick as bashing up a site in Rails admittedly but I am looking forward to using it more and also the book [aff link] is meant to be great!

Sinatra

Sinatra is a very simple and lightweight web framework written in Ruby. I’ve used it because it seems better than static HTML because you can set up simple URL routing:


get '/' do
erb :index
end



get '/about' do
erb :about
end


and you can have partial templates for footers etc. So basically better than PHP for a simple site.

Related posts:

1. Using TheSchwartz (Job Queue) with Catalyst
2. Become a Perl rockstar… AKA Perl link dump
3. Tweetfactor – An experiment using Ruby, Sinatra, Twitter and Redis

To celebrate the launch of Mobile Orchard Screencasts they are giving away the first one free for a week.

It is a seven minute long screencast about running code on the iPhone.

Get it while it’s hot, in a weeks time it’ll cost you $5.

Related posts:

1. Free laptop wtf?
2. In the news – 02/07/07

I’ve just spent the best part of a day, banging my head, trying to figure out why I couldn’t get LWP::UserAgent to connect to a URL (which happened to be an https url). I was looking at headers, manually in the browser, tried sending the request through a proxy, and finally, earlier today I used Affrus to step through the code visually and inspect LWP::UserAgent, Net::OAuth and my module (WWW::Yammer).

I found a small mention about Crypt::SSLeay and thought, no way, I bashed my head for a day and a half over a missing module that there was no obvious [to me] mention of?!

Well, I installed it, and hey-presto, it now works.

So if your banging your head trying to use LWP::UserAgent to connect to an https URL, ensure you have Crypt::SSLeay installed.

Related posts:

1. Textmate and Git

Photo: I’m not a rockstar I’m a poseur.

I’m coding quite a lot of Perl again at the moment and I’ve been saving quite a lot of interesting Perl links to my delicious account which I’m going to share with you. They probably won’t make you a rockstar but it might improve your Perl knowledge.

Modules

Moose is a collection of Perl modules that add a new, more modern object framework to Perl 5.x. Programming with Moose is a wikibook that discuses the uses and use of Moose. Actually, apparently the official documentation on CPAN is a better resource for using Moose.

Mojo is a new Perl web framework based off the Catalyst framework. It is designed to have no/limited dependencies so that a Perl newbie can install and get going fairly quickly. It is still early days but it will be interesting to see how this develops.

N.B. I just heard some skeptical views about Mojo, especially with a recent release which will have broken many peoples Mojo apps as it was there were major changes in the code base but I do hope that it stabilises and do think it is worth keeping track of.

Higher Order Perl

HIgher Order Perl is a blog about Perl, computing, maths etc. I haven’t read it too much yet but I was planning to trawl through the archives at somepoint because I have a feeling it could be quite interesting (in a nerdy way).

Profiling

Profiling, not something I’ve had to do yet but it is the process of analysing your code to see which routines/loops/blocks could benefit from optimisation rather than blindly optimising what you think may be running slowly. Here is a general introduction into profiling perl and here is a perl module released by the NYT for profiling Perl code. I plan to give that a look should I need to do any profiling.

Prototypes

I’m not talking about little wooden miniatures, prototypes in Perl are like, what’s the word, parameter declarations in function declarations in other languages? Except actually, I think they’re not. Well anyway, I read a post about Perl prototypes, which was pretty confusing, so then I asked my buddy Kristian to explain. I also read another long article about prototypes. Needless to say I still don’t fully understand the implications but basically don’t declare your parameters, I think..?

Object Orientated Perl

Even though I wanted to use Moose (see above) I still needed an overview of object orientation in Perl, as I knew from previous experience it was a bit wierd – bless / self etc… There was a pretty nice article by Chromatic, including some stuff about using Moose and another article about blessing your father.

Module Creation

I’m currently writing a module that I’m hopefully going to release to CPAN when finished but I’d never done that before so I read José’s guide to creating Perl modules and a guide on Perlmonks. Also I looked into Module::Builder::PBP on CPAN.

Misc. Perl Links

Exception handling in Perl.
Writing serious Perl: The absolute minimum you need to know.
Namespaces & Scoping
A link checking spider (I do not condone copyright infringement but that is a useful link for me as I’ll be writing a Perl spider soon..)
Intro to Test::More (Googledoc/PDF)

That’s your lot for today, if there are other great Perl articles or blogs I’d love to know about them..

Related posts:

1. Perl Ain’t Dead…
2. Mad-Lib Perl Snippet
3. Using TheSchwartz (Job Queue) with Catalyst

If you’ve installed git and use textmate you may receive errors about git not being installed correctly until you do the following:

1. run “which git” on the command line and note the output.
2. in textmate write “echo $PATH” and check to see that the path returned from which git is present – in my case it was
3. Open textmate preferences -> advanced -> shell variables and add a new shell variable: TM_GIT with the value of what was returned from “which git” e.g. /usr/local/git/bin/git

Now it should work.

Related posts:

1. TextMate Micro-Blog
2. Stacks, Heaps, Variables and Pointers