A few years ago I moved house and became the owner of a nice backyard with a garden.  However with early arrival of summer, I quickly discovered the challenge of keeping everything alive!  As holidays and heat-waves approached it became pretty clear we needed an automated watering, and one that could keep track of tank levels to avoid running the tank dry and damaging the water pump. Add to that the fact I'm always out to save a buck, I was inspired to do a little IoT project... Final set-up, sorry about the spoilers I iterated on this idea for quite a while with ideas ranging from full computer controlled watering system to something much simpler.  In the end I decided my requirements were: Have a watering system with an automatic timer Be able to switch off the pump remotely  Be able to check tank levels remotely I eventually realised that two of these would be much easier (and probably no more expensive) to buy as off-the-shelf components but there was stil...

The great bike rush of 2020 is in full swing! COVID-19 restrictions means bike shops are sold out and there's a huge increase in cycling everywhere... or so media (social and otherwise) would have you believe, but I haven't seen any details on what is actually happening. So as a cyclist and data scientist I thought I would try and find out. I wrote a deconstruction of the reported 700% increase in cycling in Cycling, the new COVID-19 side effect? where found some serious sampling bias issues and proposed a more realistic results of 10-150% increase on weekends and a reduction on weekday at most location. While far more robust (in my option anyway) it still did not provide extensive coverage and did not look at changes in where people were riding. So here we are for round 2 so lets try and make it slightly scientific. My hypothesis (theory of what's happening) is that over the lockdown period: weekday cycle counts will be lower and weekend counts remain unchanged or higher,...

I've been reading a bit about the increased number of people cycling while the world is in lockdown but last week there were a few articles about the bicycle numbers in Melbourne that I was a little dubious about. So I did what no sane person does, I dived into the data... The article ( Car parks out, footpaths and cycling lanes in as city prepares for post-COVID commuters ) in The Age on 7th May 2020 cited cycling growth figures of 138% to 778% which seemed a little high to be believable. The figures apparently came from an article and report by the Bicycle Network detailing changes in cycling activity based on surveys they had conducted. Now firstly, I couldn't find matching figures in the report for all the sites listed in The Age article, but more importantly I had a few issues with the data being used for this kind of analysis. Bicycle network published the data from 4 surveys, one on Sunday 10th of November 2019 and three in April 2020 on a Wednesday, a ...

I noticed a post on linkedIn about a published report with a nice looking chart... that I couldn't really make sense of despite working in the industry for 13 years. So I opened the larger version and looked harder. While it didn't take too long to understand there was a lot of interpretation required and I think I decided it would be a good example for a makeover before I had even finished reading it. And yes, I totally stole the idea of doing a graph makeover . Chart makeover: left - original, right - makeover Ok so firstly, pie charts are generally avoided by the data visualisation community, perhaps most famously by Stephen Few , and for good reason, they are notoriously hard to read and interpret with only a few exceptions. The three rules of Pie I use and recommend are: Don't use a pie chart Don't use a pie chart Try using another chart first (FYI, the main exceptions to the rule are when you only 2-3 catagories and they are of very different sizes. ...

I recently spend about 8 weeks as a public transport commuter rather than cycle commuter and had the opportunity to listen to a lot of podcasts (some of which I've already posted about ). One that I really enjoyed was Story Telling With Data which is where I heard about the monthly challenge that Cole runs. The December challenge was to make a holiday themed visualisation so I decided to investigate how travel patterns change in the holiday period, which for Australia is about a quarter to half of December and all of January. The data set I found was for international departures from Australia however the destinations appear to be only to the first stop of the flight. Unfortunately the data did not include spatial information for the airports or the standard airport codes so I had to do some wrangling to make everything work. This and the fact it took me a while to draw out the interesting (and communicable) insights meant that I well and truly missed the challenge deadlin...

xkcd: Survivorship Bias Are you biased ? Would you know if you were? Statistics views biases as something to be avoided or corrected, and for good reason, because statistics typically seeks to use a sample to represent an entire 'population'. But I recently heard a new take on it which advocated that you should use your bias in doing data analysis, data visualisations and decision making. In order to use your biases the first thing to know is that everyone is biased due to their differing circumstances, and of course being human. The idea is that a person will have a unique point of view which is valuable, but that it won't be possible to fully understand others point of view or how that point of view will affect others. Therefore the key is collaborating with others that have different biases, with some level of understand of your biases and those of your collaborators. In many ways this is no different to the statistical approach but rather than trying to find...

My wife and I are currently house hunting and this will be the first house I've ever bought so I'm a little nervous about not messing it up (perfectionist and all that). So when we find a property we're interested in I dig into the details by checking for easements, looking at the drainage and checking planning overlays since I know a little about these things from my time in a Local Government engineering department many moons ago. Anyway it turns out that a lot of these things are available as open data, for the whole of Victoria! So to make my life easier I put together a little web app that also works on mobile devices for when you are actually at an inspection. Victorian Property Info Having recently built the Cycle Melbourne map , this one was super quick to put together. I used the same layout, almost the same background maps and just added 4 WMS map layers from land.vic.gov.au . The Leaflet Extras package also allowed me to add a search function and a...

I've been using R for a few years now and have found it fantastic for all the data manipulation and visualisation I was doing, plus you can even use it to build web apps! So to help people get going with R I've put together a pre-tutorial to show off a little of what it can do and help you work out if it is for you. Why use R? Overview and case studies One thing I missed in the video was to explain what R actually is, so here's my super brief definition... R is an open source data analysis scripting (or programming) language which was originally developed for statistical analysis but is now used more broadly in many scientific and business fields. It is my second ever video tutorial so please be kind but I do welcome suggestions for improving them or requests for topics/questions you are interested in. Also, what do you think of the new logo?

I'm big believer in beauty in simplicity... most of the time.  But regardless this is a great presentation that walks through some table formatting options that produce a much more readable table.  Interestingly the starting point is quite similar to the Microsoft defaults... Clear Off the Table

For years now I've been recommending QGIS to friend and colleagues with the caveat that there is bit of a learning curve, but that it's really great!  So to help these people along I've created a video tutorial that will get you up and running with QGIS. Intro to QGIS: Installing and Importing - Quickstart guide It is my first ever video tutorial so please be kind but I do welcome suggestions for improving them or requests for topics/questions you are interested in.  In future I probably won't record on quite so little sleep with a sleeping baby in the next room...

While delving into some data science communities I had hear of Kaggle many times over and had even created an account, but it wasn't until I got an email about a dataset of Kiva loans that I actually got interested. First of all, what is Kaggle? Kaggle is a platform for predictive modelling and analytics competitions in which statisticians and data miners compete to produce the best models for predicting and describing the datasets uploaded by companies and users. - Wikipedia Essentially it is a data science playground where you can access data, do analysis, look at other peoples analysis and enter competitions. If you submit an analysis then all your code is available for others to see which means you can 'be inspired' techniques you see others using and also makes it easier assist others when they need help. So what about Kiva? Kiva Microfunds is a 501 non-profit organization that allows people to lend money via the Internet to low-income entrepreneurs ...

 KML is a really common format, it is not usually one that GIS professionals use for a number of reasons but it is really handy for passing GIS data to online mapping systems, portable devices and it can even be created with a text editor.  If you want more information on it then the wiki page on Keyhole Markup Language (yes, that is the actual name) and Google Developers KML page are good places to start. Using KML with QGIS is super simple however there are a few tricks, particularly with KMZ files .  Here is a brief tutorial on opening KML files in QGIS. So getting them into QGIS is fine but sometimes I need to use the data within a KML file and QGIS doesn't deal with that very well as all the descriptive fields are groups in an HTML table in a single field.  While I'm sure it is possible to parse this in QGIS, here is a script in R since that was where I needed the data anyway. Save the KML file as a .csv file Open the file in R with read.csv ...

Region around Tidal River Wilsons Promontory is a beautiful national park in Victoria, Australia and actually contains the southern most point of the mainland. It is know for its beaches, camping, hiking and generally just being a nice place to visit.  While it is almost impossible to get lost there, I do love a good map and they do come in handy... Background I started making maps when I discovered there was a wealth of data on the data.vic.gov.au site and a friend mentioned needing some maps to go fly fishing. So I downloaded some data and set about putting something together. The first maps were not great, it was rather hard developing layer styles that work well together but it was a good starting point and a proof of concept. Tidal River looking towards Mt Ramsay Not long after that I decided to put together some maps to do the Coast to Coast in England based on UK Ordinance Survey data .  One of the really nice thing about this data is that they also off...

I had been looking for a good coding project for a while, to try and build some confidence before I attempted a hackathon, when a friend posted a link to a great map based representation of the  Australian Marriage Law Postal Survey results.  It was a nice clean map that provided a mouse-over pop-up of the yes/no percentage for each electorate which I found it really interesting, but I also wanted some more information.  Information on things like the number of voters, the member holding that seat, the party they are from, what suburbs where in each electorate, and maybe even some census data.  Thus the project was formed. As always, the 80:20 rule was proved true when I got the map up with the basic info the following day, but being a perfectionist (with a small child and a few other project on the go), it has taken me quite a bit longer to show it off but here it is: 2017 Australian Marriage Law Postal Survey - Results Map (The site is hosted on shinyapps.i...

I just got back from a weekend of camping near Walhalla which is a really beautify area if you ever get the chance to visit.  Anyway, I had heard a number of the people coming were planning to go four-wheel driving and I thought it might be a chance to find out what my new (to me) Freelander could do, so I set about making some maps of the area. The maps just re-use a set of styles I had previously developed to create some topographic maps of rural areas that had worked pretty well for me in the past, with almost all the data downloaded from data.vic.gov.au .  Everything was going well until realised I should check if there were any track closures in the area and after a quick google search I found Parks Victoria - Seasonal road closures , and it turns out there were quite a few, both seasonal and management vehicle only tracks.  In the past I had tried to use tracks from the Vicmap Transport - Road Network dataset that on arriving at the track, discovered they had cle...

​Recently I wrote a chapter on 'data' in a transport engineering book (coming out soon) and one particular aspect I found rather hard was to avoid being repetitive. Over and over I kept coming back to, "the reason for this is to better communicate x, y or z".  Communication is a critical and often overlook part of any analysis, data science or otherwise, so it is worth spending some time thinking about how to do it well.  T his article doesn't draw a lot of conclusions, it is more arguing for a need for a framework, however Roger does allude to one that I know he likes to use and that's the 'reproducible research' concept... Specialization and Communication in Data Science From the Simply Stats blog by Rafa Irizarry, Roger Peng, and Jeff Leek.  

Hi, I'm Will and I'm a map-aholic. I have been interest in maps since I started hiking in scouts, I'm one of those strange people that finds beauty in contour lines (because we all know the topographic map is the king of all maps).  But seriously, it is amazing how much information you can put on a map in an elegant way. I was first introduced GIS while doing a placement in local government in 2003, I was fascinated, even just plotting drainage pipes on a map, but it was the briefest of introductions and it was years till I came back to it.  After uni I ended up in road and transport research where I got my teeth into data analysis and was able to dabble in GIS again. 12 years on I'm still working in transport research and after commencing a data science course I'm doing more data analytics, management and GIS than ever.  The thing that has helped my skills along the most is being able to use open source software products means that I can also do GIS, map mak...