How to manage maps which you sometimes edit on your smartphone, sometimes in Google Maps, and which you'd like to edit with even more powerful tools still?

... and you see me coming – by powerful tools I mean of course Vim, for diffing and merging maps coming from different sources. But I'm also thinking here about MAPS.ME on one end, Google Maps on the other end, and any UNIX tool somewhere in the middle.

1 OpenStreetMap and MAPS.ME

It's already back in 2008 that I first stumbled upon OpenStreetMap, a collaborative mapping project allowing you to freely download high-quality maps of any chartered part of the world. It was launched in 2004 and obviously didn't have very complete data back then. Things have had the time to change drastically in 10 years, however, and my interest in the project was kindled again when I saw to how much detail OpenStreetMaps nowadays go, and how well some applications use their potential.

I got a feel for this potential when I went looking for free Android applications allowing you to freely download offline OpenStreetMaps. After trying a few, the best I could find so far was MAPS.ME. It may not be perfect – it's in some cases absurdly slow and has a tendency to drain the battery, but I couldn't find anything better so far. It's particularly interesting in that it's useful when driving, walking, cycling, and that's why it's been my companion on many travels. It lets you save bookmarks which you can share between various media as KMZ files.

2 Google Maps

When it comes to comfortably editing things, a smartphone is no substitute for a real computer. The best tool I've found to work with real-world map data remains Google Maps. Besides, it's likely the few places OpenStreetMap doesn't know about can still be found in Google Maps. It also lets you do things MAPS.ME doesn't, such as placing markers in arbitrary locations without getting them snapped to any existing one. Google Maps can load and save KML/KMZ data, allowing for exchange, too.

3 Syncing and Merging XML Maps

So as you might have guessed by now, technically, they're really called KML. Or KMZ when compressed into a ZIP file.

Sharing maps between devices is easy enough with services such as Dropbox, and I should probably say at this point that, on the other hand, I've not been too impressed with Google Drive's offering which still lacks a decent Linux client and forces users to tediously go clicking every time a file needs syncing. So Dropbox it is, then, as it really makes it all effortlessly transparent.

If the idea is to merge data from maps which are edited with Google Maps on the one hand and MAPS.ME on the other hand, we need to make sure that we're dealing with XML that's easily compared with diff-like utilities. The problem is that the two tools structure data in rather different ways, and I'm not even talking about the fact that place marks – that's what they call locations you record – may very well be ordered differently. Fixing that goes well beyond any tool processing data on a per-line basis, and I needed something that can actually understand XML. I also noticed that several tags aren't actually necessary to work in either Google Maps and MAPS.ME, so I've removed them all to end up with rather canonical XML data.

3.1 An Attempt with XMLStarlet

XMLStarlet is a powerful tool which can certainly sort XML, remove unwanted tags and even move them around following XPath expressions. I rather effortlessly managed to clean up a map XML with a command like:

xmlstarlet ed \
    -d //_:Style \
    -d //_:TimeStamp \
    -d //_:styleUrl \
    -d //_:ExtendedData \
    -d //_:visibility \
    -d //_:description \
    -d //_:StyleMap \
    -m //_:Placemark //_:Document \
    -d //_:Folder

The ed subcommand is for editing (as opposed to some other commands which can be used e.g. for extracting data). The -d option removes tags and I noticed that removing those particular ones didn't disturb either Google Maps or MAPS.ME. The -m option moves tags, which was useful because Google Maps rather unnecessarily places Placemark tags into Folder tags, which MAPS.ME more wisely does not. The _: prefixing the tag name refers to the namespace declared at the root.

This all worked very nicely, but then I stumbled upon a rather annoying problem: newlines. How do I make sure I compare XML that follows the same style of placing newlines? Unfortunately, in spite of the obvious need for such a feature, I wasn't able to find a way in XMLStarlet to reformat XML according to a predictable style. And this, I'm afraid, is when I decided to reinvent the wheel and write a tool of my own.

3.2 Another Attempt with a Custom Tool

I wrote mapxml which loads KML files or extracts them from KMZ ones and rather brutally discards all tags other than Placemark, and some name elements before writing canonical temporary XML files which it opens into vimdiff for comparing them easily.

4 Result

It all works rather well, in practice. I've been able to clean up a backlog of several overlapping or incomplete maps without too much effort, editing them either with MAPS.ME, or Google Maps, or Vim, with Dropbox as a common place to store the maps. It's a process that's finally become straightforward – even fun.

5 References