Bricolage daily activity summary.

Before I start I should preface this with a big "yuk". Its an extremely inelegant way to achieve what I want.

By default Bricolage sends email notifications with root relative URLs /like/this. Some time ago the NI techs were chatting about how it'd be nice to be able to click the URLs from the emails. This is a partial solution to that.

We're lucky enough to be running a dovecot imap server for our email. I filter bric email notifications into a single folder. Well, dovecot stores files inna maildir stylee. Which means they can be parsed like regular files.

So, I've built a script that trundles through my Bricolage folder checking for new files, parses them and prints a daily activity summary. Here's the code:

#!/usr/bin/perl use strict; use warnings; use Email::Simple; use Data::Dumper; # Used for testing



my $MAILDIR = '/home/charlie/Maildir/my_bric_folder'; # directory where files are stored
my %baseurl_lookup = ('New Internationalist main site'=>'http://newint.org','New Internationalist main site'=>'http://b
my @emails = glob "$MAILDIR/*"; # all our emails
my %hash; # 2d hash of story details keyed on {section}{action}{storytitle}

for(@emails) {
  my $age = -M; 
  next if($age>1); # ignore old emails
  chomp;
  open IN, $_ or die ("Couldn't open $\n$!");
  $/=undef;
  my $text = ;
  close IN; 
  $/="\n";
  my $email = Email::Simple->new($text);
  my $subject = $email->header('subject');
  my $body = $email->body;
  # Only certain subject lines require any action
  createdstory($body)    if($subject =~ /A story was just created/); 
  published_story($body)  if($subject =~ /A story was just published/); 
}

print Dumper %hash;

Print out daily summary

for(keys %hash) {
  print "\n\n$\n=============================\n";
  my $section=$;
  for(keys %{$hash{$section}}) {
    print "\n$_ stories\n-----------------\n";
    my $action = $; 
    for(keys %{$hash{$section}{$action}}) {
    print "\n$ stories\n-----------------\n";
    my $action = $; 
    for(keys %{$hash{$section}{$action}}) {
      print "* " . $ . ", by " . 
        $hash{$section}{$action}{$}{'author'} . " " . lc($action) . " on " .
        $hash{$section}{$action}{$}{'date'} . " at " .
        $hash{$section}{$action}{$_}{'url'} . "\n";
    }
  }
}

Parse "story created" email

sub created_story {
  my $body = shift;
  my @body = split(/[\r\n]+/,$body);
  $body = $body[0];
  $body =~ /A story by the name of "([^"]+)" was created by ([\w\s]+) and lives at ([^\s]+) (([^)]+)) and is sc
  my ($title,$author,$location,$section,$date) = ($1,$2,$3,$4,$5);
  my $action = "Created";
  my $url = $baseurl_lookup{$section} . $location;
  $hash{$section}{$action}{$title} = {'url' => $url, 'date' => $date, 'author' => $author };
}

Parse "story published" email

sub published_story { my $body = shift; my @body = split /[\r\n]+/,$body; # strips empty lines # line 1 $body[0] =~ /A story by the name of "([^"]+)" was published on ([\d-]+ [\d\:]+) by ([^.]+)./; my ($title,$date,$author) = ($1,$2,$3); # line 2 $body[1] =~ /You can view a live copy of this story at ([^(]+)(([^)]+))$/; my($location,$section) = ($1,$2); $location =~s/\s$//; my $action = "Published"; my $url = $baseurl_lookup{$section} . $location; # print "S: $section A: $action T: $title U: $url D: $date\n\n"; # testing $hash{$section}{$action}{$title} = {'url' => $url, 'date' => $date, 'author' => $author }; }

Yep, its pretty gnarly. But it does the job. I guess the ideal would be to rip out the parsing code and set it up as a filter in postfix. But that's for another day.

To install it I set up a simple crontab, using crontab -e:

0       0       *       *       * /path/to/bric_activity_summary.pl  | mail -s "FYI: Daily Bricolage activity summary" [email address scrubbed]

And now we can get daily summaries that look like this:

New Internationalist main site
=============================

Published stories
-----------------
* Cartoon Capture and Storage, by Bricolage Administrator published on 2009-02-03 09:45:09 at http://blog.newint.org/cantankerousfrank/2009/02/02/cartoon-capture-and-storage/
* Stealth wars, by Sokari Ekine published on 2009-02-05 01:25:10 at http://blog.newint.org/majority/2009/02/05/stealth-wars/
* Democratic Republic of Congo, by Sokari Ekine published on 2009-02-05 01:25:10 at http://blog.newint.org/tag/democratic-republic of congo/
* conflict, by Sokari Ekine published on 2009-02-05 01:25:10 at http://blog.newint.org/tag/conflict/
* BBC, by Sokari Ekine published on 2009-02-05 01:25:10 at http://blog.newint.org/tag/bbc/
* Singularity, by Bricolage Administrator published on 2009-02-04 09:45:07 at http://blog.newint.org/cantankerousfrank/2009/02/03/singularity/

Created stories
-----------------
* Cartoon Capture and Storage, by Bricolage Administrator created on 2009-02-02 06:13:00 at http://blog.newint.org/cantankerousfrank/2009/02/02/cartoon-capture-and-storage/
* Stealth wars, by Sokari Ekine created on 2009-02-05 01:20:06 at http://blog.newint.org/majority/2009/02/05/stealth-wars/
* Singularity, by Bricolage Administrator created on 2009-02-03 04:21:00 at http://blog.newint.org/cantankerousfrank/2009/02/03/singularity/


New Internationalist main site
=============================

Created stories
-----------------
* Fire in the soul, by Theo Sundh created on 2009-02-03 07:32:00 at http://newint.org/publications/fiction/poetry/

Comments on Bricolage daily activity summary.

Leave your comment

Maximum characters allowed: 5000

Simple HTML allowed: bold, italic, and links

Registration is quick and easy!
Register | Login

#1 Phillip Smith 05 Feb 09

#2 Adam Maanit 06 Feb 09

#3 Greg Heo 16 Feb 09

#4 Phillip Smith 17 Feb 09

#5 ciderpunx 24 Feb 09

#7 ascerycle 16 Jan 11

Guidelines: Please be respectful of others when posting your reply.

Maximum characters allowed: 5000
Simple HTML allowed: bold, italic, and links

#1 Phillip Smith 05 Feb 09

Hey there Charlie,

On a related topic, I asked our colleague Greg Heo to develop an administrative RSS feed of all system events, which could be used to monitor daily Bricolage system activity.

It basically aggregates any / all ’events’ in the Bricolage system, and writes them out to a publicly readable .rss file.

Unfortunately, Greg hasn't had time to release the code ’officially,’ and I haven't had time to check it into a public repository -- so I've e-mailed it to you for now. ;-)

Once it's online somewhere, I'll pop a link into this thread.

Phillip.
#2 Adam Maanit 06 Feb 09

Nice work

Great stuff Charlie!

Are you guys getting the blog post breaking out of the column past the sidebar on your systems?
#3 Greg Heo 16 Feb 09

I like the tech blog

Great blog! I love reading about the ’behind the scenes’ activity behind a web site, especially one powered by Bricolage. Could I do a guest post about Bricolite or the Schwartzian transform some day?

There's an updated version of the Bricolage events thing, adapted for a large, regional cable sports network. It allows you to select specific events, story types, date range, etc. but outputs a CSV rather than RSS. I'll send it along so you won't have to wait until I get my ’Open Source code by Greg’ web site up and running.
#4 Phillip Smith 17 Feb 09

’Could I do a guest post about Bricolite or the Schwartzian transform some day?’

Sure thing! That would be super. Just send it over to me once you've got something ready to go. :-)

’There's an updated version of the Bricolage events thing, adapted for a large, regional cable sports network. It allows you to select specific events, story types, date range, etc. but outputs a CSV rather than RSS. I'll send it along so you won't have to wait until I get my ’Open Source code by Greg’ web site up and running.’

Sounds great. I'm sure Charlie will be excited to play with it. :-)

Best,

Phillip.
#5 ciderpunx 24 Feb 09

Hey Greg,

Thanks for the CSV exporter code. Looks like it could be mega useful; the RSS event feed, too. I'm going to have a play at some point soon. I'm a big fan of RSS for monitoring; it somehow seems less annoying than email!

And it'd be awesome if you'd like to write a guest blog; just mail or silc it through to us.

Cheers,
Charlie
#7 ascerycle 16 Jan 11

What need those things olph dreaded ones would deserved beating ecstasy.

Blithesome Additional Year[url=http://sdjfh.in/flexpen/],[/url] everybody under the sun! :)

Subscribe to Comments for this article

New Internationalist

Bricolage daily activity summary.

print Dumper %hash;

Print out daily summary

Parse "story created" email

Parse "story published" email

Comments on Bricolage daily activity summary.

Leave your comment

#1 Phillip Smith 05 Feb 09

#2 Adam Maanit 06 Feb 09

#3 Greg Heo 16 Feb 09

#4 Phillip Smith 17 Feb 09

#5 ciderpunx 24 Feb 09

#7 ascerycle 16 Jan 11

About the author

Most popular

‘Frack free’: a Sussex village gets organized

Argument: Should prostitution be legalized?

Cameroon's women call time on breast ironing

The hypocrisy of sexism in India

India's elites have a ferocious sense of entitlement

Britain moves a step closer to combating caste

Argument: Is it time to ditch the pursuit of economic growth?

Leave mums alone

The hypocrisy of sexism in India

Wonder and frustration: living with the animals

Multimedia

Radio NI interviews - available in iTunes

Popular tags

Related entries

Obfuscating email addresses

The Tech Blog