128km Melbourne radar rainfall animation

For a while I’ve been thinking it would be cool to scrape the Bureau of Meteorology rainfall images and make a movie out of them. The result was not quite as good as I imagined but interesting anyway. I’ll make a better one in the future with more data.

Digging through the page source I discovered they show six images in a cycle at a time. Images are removed from the loop location so I couldn’t just edit filenames and download the previous 1000 or so images – so I had to scrape them as they are used. A typical image looks like this:

DR023.T.201012030224.png

Clearly that didn’t have all the data so I dug a bit further and found three other images that this is overlayed on top of to make the image you see when you visit the site. These images contain the basic map of the area, the topography, the city locations and the key legend for the rainfall density.

I used ImageMagick ‘composite’ tool to overlay them. You can only do two images at a time so I prepared the four underlayed images into one and then wrote a script with a column-editor to batch the scraped images onto that background.

To get all the images I wrote the scraper in PHP and set a cron task. Images are uploaded every six minutes (10 an hour), so I just had a cron task running every 30min to make sure I didn’t miss any.

*/30 * * * *   root  /usr/bin/php /home/pinky/bom-data/index.php

The actual script to scrape the images is:


#!/usr/bin/php -q

include_once('myclass.web.php');
include_once('dBug.php');

//header("Content-Type: text/html; charset=UTF-8"); //no funny biznez, kthx

$siteroute = '<a class="linkification-ext" title="Linkification: http://www.bom.gov.au" href="http://www.bom.gov.au">http://www.bom.gov.au</a>'; //no trailing slash
//$imagesavepath = 'F:\MirroredFiles\Webdev\htdocs\bom-data\images';
$imagesavepath = '/home/pinky/bom-data/images'; //no traling slash

$data=Web::getPage('<a class="linkification-ext" title="Linkification: http://www.bom.gov.au/products/IDR023.loop.shtml" href="http://www.bom.gov.au/products/IDR023.loop.shtml">http://www.bom.gov.au/products/IDR023.loop.shtml</a>'); //Melbourne 128km radar cover

//go through page HTML source line by line and find the rainfall images in the JavaScript
$count = 0;
$imageurls = Array();
foreach(preg_split("/(\r?\n)/", $data) as $line){
if ( substr( $line, 0, strlen("theImageNames")) == "theImageNames" ) {
//e.g.: theImageNames[0] = "/radar/IDR023.T.201012021212.png";
$imageurls[$count] = substr($line, 20, 32);
$count++;
}
}

//if the image file does not exist locally already from a previous attempt then download it
$downloadcount = 0;
for ($ii=1; $ii&lt;sizeof($imageurls); $ii++) {
$imgname = substr($imageurls[$ii], 8, 25);
if ( !file_exists($imagesavepath . "/" . $imgname) ) {
$imgdata = file_get_contents($siteroute . $imageurls[$ii]);
file_put_contents($imagesavepath . "/" . $imgname , $imgdata);
$downloadcount++;
}
}

$logfile = "/tmp/bom-data.log";
$fh = fopen($logfile, 'a') or die("Can't open file");
fwrite($fh, date ("Y-m-d H:i:s", time()) . " Downloaded images: " . $downloadcount . "\n");
fclose($fh);

And my script to merge each image onto the background layer looks like:


#!/bin/sh
/usr/bin/composite DR023.T.201012030224.png background.png  DR023.T.201012030224.png
/usr/bin/composite DR023.T.201012030230.png background.png  DR023.T.201012030230.png
/usr/bin/composite DR023.T.201012030236.png background.png  DR023.T.201012030236.png
.
.
.

The nice time-stamped image names made everything easy. Overlaying 720 images took about a minute. I was surprised how long it took, even though each image is only 10kB or so.

To join all the images into a movie I used mencoder.

mencoder "mf://*.png" -mf fps=25 -ovc lavc -o output.avi

That just puts all the png’s in the directory into an encoded (compressed) AVI at 25fps. The resulting file was only 2.7MB and runs for 28 seconds. There wasn’t much rain in the last couple of days so it gets a bit boring to keep going further than that.

Here’s the resulting movie which shows the period 3rd Dec to 6th Dec.

Sorry for the very ordinary quality – I’d have to read up what encoding YouTube likes to make it nice because the source PNGs are very clear. Using mencoder with -lavc and then Windows Media Maker pretty much kills it.

To give credit to the soundtrack, the music is http://www.youtube.com/watch?v=aeYTBvanFmE

I’ll make a longer movie once I’ve collected a couple of months worth of data.

About Pinky

Computers, programming, new music and all things science, tech and sport
This entry was posted in Computer programming and tagged , , . Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>