New Internationalist

Bricolage daily activity summary.

Before I start I should preface this with a big "yuk". Its an extremely inelegant way to achieve what I want.

By default Bricolage sends email notifications with root relative URLs /like/this. Some time ago the NI techs were chatting about how it’d be nice to be able to click the URLs from the emails. This is a partial solution to that.

We’re lucky enough to be running a dovecot imap server for our email. I filter bric email notifications into a single folder. Well, dovecot stores files inna maildir stylee. Which means they can be parsed like regular files.

So, I’ve built a script that trundles through my Bricolage folder checking for new files, parses them and prints a daily activity summary. Here’s the code:

#!/usr/bin/perl use strict; use warnings; use Email::Simple; use Data::Dumper; # Used for testing

my $MAILDIR = '/home/charlie/Maildir/my_bric_folder'; # directory where files are stored my %baseurl_lookup = ('New Internationalist main site'=>'http://newint.org','New Internationalist main site'=>'http://b my @emails = glob "$MAILDIR/*"; # all our emails my %hash; # 2d hash of story details keyed on {section}{action}{storytitle}

for(@emails) { my $age = -M; next if($age>1); # ignore old emails chomp; open IN, $_ or die ("Couldn't open $\n$!"); $/=undef; my $text = ; close IN; $/="\n"; my $email = Email::Simple->new($text); my $subject = $email->header('subject'); my $body = $email->body; # Only certain subject lines require any action createdstory($body) if($subject =~ /A story was just created/); published_story($body) if($subject =~ /A story was just published/); }

print Dumper %hash;

Print out daily summary

for(keys %hash) { print "\n\n$\n=============================\n"; my $section=$; for(keys %{$hash{$section}}) { print "\n$_ stories\n-----------------\n"; my $action = $; for(keys %{$hash{$section}{$action}}) { print "\n$ stories\n-----------------\n"; my $action = $; for(keys %{$hash{$section}{$action}}) { print "* " . $ . ", by " . $hash{$section}{$action}{$}{'author'} . " " . lc($action) . " on " . $hash{$section}{$action}{$}{'date'} . " at " . $hash{$section}{$action}{$_}{'url'} . "\n"; } } }

Parse "story created" email

sub created_story { my $body = shift; my @body = split(/[\r\n]+/,$body); $body = $body[0]; $body =~ /A story by the name of "([^"]+)" was created by ([\w\s]+) and lives at ([^\s]+) (([^)]+)) and is sc my ($title,$author,$location,$section,$date) = ($1,$2,$3,$4,$5); my $action = "Created"; my $url = $baseurl_lookup{$section} . $location; $hash{$section}{$action}{$title} = {'url' => $url, 'date' => $date, 'author' => $author }; }

Parse "story published" email

sub published_story { my $body = shift; my @body = split /[\r\n]+/,$body; # strips empty lines # line 1 $body[0] =~ /A story by the name of "([^"]+)" was published on ([\d-]+ [\d\:]+) by ([^.]+)./; my ($title,$date,$author) = ($1,$2,$3); # line 2 $body[1] =~ /You can view a live copy of this story at ([^(]+)(([^)]+))$/; my($location,$section) = ($1,$2); $location =~s/\s$//; my $action = "Published"; my $url = $baseurl_lookup{$section} . $location; # print "S: $section A: $action T: $title U: $url D: $date\n\n"; # testing $hash{$section}{$action}{$title} = {'url' => $url, 'date' => $date, 'author' => $author }; }

Yep, its pretty gnarly. But it does the job. I guess the ideal would be to rip out the parsing code and set it up as a filter in postfix. But that’s for another day.

To install it I set up a simple crontab, using crontab -e:

0       0       *       *       * /path/to/bric_activity_summary.pl  | mail -s "FYI: Daily Bricolage activity summary" [email address scrubbed]

And now we can get daily summaries that look like this:

New Internationalist main site
=============================

Published stories
-----------------
* Cartoon Capture and Storage, by Bricolage Administrator published on 2009-02-03 09:45:09 at http://blog.newint.org/cantankerousfrank/2009/02/02/cartoon-capture-and-storage/
* Stealth wars, by Sokari Ekine published on 2009-02-05 01:25:10 at http://blog.newint.org/majority/2009/02/05/stealth-wars/
* Democratic Republic of Congo, by Sokari Ekine published on 2009-02-05 01:25:10 at http://blog.newint.org/tag/democratic-republic of congo/
* conflict, by Sokari Ekine published on 2009-02-05 01:25:10 at http://blog.newint.org/tag/conflict/
* BBC, by Sokari Ekine published on 2009-02-05 01:25:10 at http://blog.newint.org/tag/bbc/
* Singularity, by Bricolage Administrator published on 2009-02-04 09:45:07 at http://blog.newint.org/cantankerousfrank/2009/02/03/singularity/

Created stories
-----------------
* Cartoon Capture and Storage, by Bricolage Administrator created on 2009-02-02 06:13:00 at http://blog.newint.org/cantankerousfrank/2009/02/02/cartoon-capture-and-storage/
* Stealth wars, by Sokari Ekine created on 2009-02-05 01:20:06 at http://blog.newint.org/majority/2009/02/05/stealth-wars/
* Singularity, by Bricolage Administrator created on 2009-02-03 04:21:00 at http://blog.newint.org/cantankerousfrank/2009/02/03/singularity/


New Internationalist main site
=============================

Created stories
-----------------
* Fire in the soul, by Theo Sundh created on 2009-02-03 07:32:00 at http://newint.org/publications/fiction/poetry/

Comments on Bricolage daily activity summary.

Leave your comment







 

  • Maximum characters allowed: 5000
  • Simple HTML allowed: bold, italic, and links

Registration is quick and easy. Plus you won’t have to re-type the blurry words to comment!
Register | Login

  1. #1 Phillip Smith 05 Feb 09

    Hey there Charlie,

    On a related topic, I asked our colleague Greg Heo to develop an administrative RSS feed of all system events, which could be used to monitor daily Bricolage system activity.

    It basically aggregates any / all ’events’ in the Bricolage system, and writes them out to a publicly readable .rss file.

    Unfortunately, Greg hasn't had time to release the code ’officially,’ and I haven't had time to check it into a public repository -- so I've e-mailed it to you for now. ;-)

    Once it's online somewhere, I'll pop a link into this thread.

    Phillip.

  2. #2 Adam Maanit 06 Feb 09

    Nice work

    Great stuff Charlie!

    Are you guys getting the blog post breaking out of the column past the sidebar on your systems?

  3. #3 Greg Heo 16 Feb 09

    I like the tech blog

    Great blog! I love reading about the ’behind the scenes’ activity behind a web site, especially one powered by Bricolage. Could I do a guest post about Bricolite or the Schwartzian transform some day?

    There's an updated version of the Bricolage events thing, adapted for a large, regional cable sports network. It allows you to select specific events, story types, date range, etc. but outputs a CSV rather than RSS. I'll send it along so you won't have to wait until I get my ’Open Source code by Greg’ web site up and running.

  4. #4 Phillip Smith 17 Feb 09

    ’Could I do a guest post about Bricolite or the Schwartzian transform some day?’

    Sure thing! That would be super. Just send it over to me once you've got something ready to go. :-)

    ’There's an updated version of the Bricolage events thing, adapted for a large, regional cable sports network. It allows you to select specific events, story types, date range, etc. but outputs a CSV rather than RSS. I'll send it along so you won't have to wait until I get my ’Open Source code by Greg’ web site up and running.’

    Sounds great. I'm sure Charlie will be excited to play with it. :-)

    Best,

    Phillip.

  5. #5 ciderpunx 24 Feb 09

    Hey Greg,

    Thanks for the CSV exporter code. Looks like it could be mega useful; the RSS event feed, too. I'm going to have a play at some point soon. I'm a big fan of RSS for monitoring; it somehow seems less annoying than email!

    And it'd be awesome if you'd like to write a guest blog; just mail or silc it through to us.

    Cheers,
    Charlie

  6. #7 ascerycle 16 Jan 11

    What need those things olph dreaded ones would deserved beating ecstasy.

    Blithesome Additional Year[url=http://sdjfh.in/flexpen/],[/url] everybody under the sun! :)

Subscribe to Comments for this articleArticle Comment Feed RSS 2.0

Guidelines: Please be respectful of others when posting your reply.

About the author

Charlie Harvey a New Internationalist contributor

Charlie Harvey is the IT Manager here at New Internationalist. He's active in both the activist and tech communities and is a vocal advocate of Free Software. You can read more on his main Charlie Harvey site.

Read more by Charlie Harvey

Get our free fortnightly eNews

Multimedia

Videos from visionOntv’s globalviews channel.

Related articles

Popular tags

All tags

The Tech Blog

This is where New Internationalist Web Team documents the free and open software used to build this website and its services, discusses emerging issues in the technology space, and provide critical analysis, news and commentary on all things IT and web.

The Tech Blog