Extract KML data

 KML is a really common format, it is not usually one that GIS professionals use for a number of reasons but it is really handy for passing GIS data to online mapping systems, portable devices and it can even be created with a text editor.  If you want more information on it then the wiki page on Keyhole Markup Language (yes, that is the actual name) and Google Developers KML page are good places to start.

Using KML with QGIS is super simple however there are a few tricks, particularly with KMZ files.  Here is a brief tutorial on opening KML files in QGIS.

So getting them into QGIS is fine but sometimes I need to use the data within a KML file and QGIS doesn't deal with that very well as all the descriptive fields are groups in an HTML table in a single field.  While I'm sure it is possible to parse this in QGIS, here is a script in R since that was where I needed the data anyway.

  1. Save the KML file as a .csv file
  2. Open the file in R with read.csv
    3.        

            csvData <- read.csv("filename.csv")
  3.  Put the description data into a separate variable
    4.        

            descriptionData <- as.character(csvData$Description)
  4.  Create a function to split the character string on the line break tag '<br>' and on the colon (with space) ': ' which separates the field name from the field data.
    5.        

            splitbr <- function(x) {
    
                y <- strsplit(x,"<br>")
                lapply(y,function(z) strsplit(z,": "))
    
            }
  5.  Run function on each row of data using lapply
    6.        

            descriptionList <- lapply(descriptionData,splitbr)
  6.  This puts the data into a bit list which needs to be separate into a dataframe.  This happens using some nested apply functions to select only the data portion for the list
    7.        

            descriptionDF <- as.data.frame(do.call("rbind", lapply(descriptionList,function(z) sapply(z[[1]],function(y) y[2]))))
  7.  Next the names or headers are added to the data frame and it is ready to be used
    8.        

            names(descriptionDF) <- sapply(descriptionList[[1]][[1]],function(z) z[[1]][1])
Notes:
There are other ways to do this in R however this uses only base functions (and is still relatively simply) so there are no dependency issues.
It would probably also be possible to read the kml directly into R which I might investigate in another post.

Comments

Post a Comment

Popular posts from this blog

Victorian Property Overlay Map

Image
My wife and I are currently house hunting and this will be the first house I've ever bought so I'm a little nervous about not messing it up (perfectionist and all that). So when we find a property we're interested in I dig into the details by checking for easements, looking at the drainage and checking planning overlays since I know a little about these things from my time in a Local Government engineering department many moons ago. Anyway it turns out that a lot of these things are available as open data, for the whole of Victoria! So to make my life easier I put together a little web app that also works on mobile devices for when you are actually at an inspection. Victorian Property Info Having recently built the Cycle Melbourne map , this one was super quick to put together. I used the same layout, almost the same background maps and just added 4 WMS map layers from land.vic.gov.au . The Leaflet Extras package also allowed me to add a search function and a...
Read more

On your bike... to where?

Image
The great bike rush of 2020 is in full swing! COVID-19 restrictions means bike shops are sold out and there's a huge increase in cycling everywhere... or so media (social and otherwise) would have you believe, but I haven't seen any details on what is actually happening. So as a cyclist and data scientist I thought I would try and find out. I wrote a deconstruction of the reported 700% increase in cycling in Cycling, the new COVID-19 side effect? where found some serious sampling bias issues and proposed a more realistic results of 10-150% increase on weekends and a reduction on weekday at most location. While far more robust (in my option anyway) it still did not provide extensive coverage and did not look at changes in where people were riding. So here we are for round 2 so lets try and make it slightly scientific. My hypothesis (theory of what's happening) is that over the lockdown period: weekday cycle counts will be lower and weekend counts remain unchanged or higher,...
Read more

Everyday IoT: Tank Water Levels

Image
A few years ago I moved house and became the owner of a nice backyard with a garden.  However with early arrival of summer, I quickly discovered the challenge of keeping everything alive!  As holidays and heat-waves approached it became pretty clear we needed an automated watering, and one that could keep track of tank levels to avoid running the tank dry and damaging the water pump. Add to that the fact I'm always out to save a buck, I was inspired to do a little IoT project... Final set-up, sorry about the spoilers I iterated on this idea for quite a while with ideas ranging from full computer controlled watering system to something much simpler.  In the end I decided my requirements were: Have a watering system with an automatic timer Be able to switch off the pump remotely  Be able to check tank levels remotely I eventually realised that two of these would be much easier (and probably no more expensive) to buy as off-the-shelf components but there was stil...
Read more