Toby's Blog of open/mappy things

A Note About JOSM

noreply@blogger.com (Toby Murray) — Wed, 04 Feb 2015 00:43:00 +0000

Over the past couple of months I have been slowly working to add support for OSM notes into JOSM. Back in July 2013, Ian Dees and I initially did a rewrite of the OpenStreetBugs plugin to support notes. We actually both started out in a friendly competition to see who could get a working version out first as we traded words on IRC. But soon Ian gave me access to his GitHub repository where we continued to work on the plugin for a while.

This plugin basically worked but I decided that since notes are now a core feature of OSM, they should also be more integrated into JOSM. In particular I wanted the workflow of notes to follow the standard JOSM download-work-upload sequence. The plugin was designed to be always online and issued a new query against the notes API every time you panned or zoomed the map. It had an "offline" mode which would stop this behavior and allow you to work without constant network traffic. However in this mode, changes to notes and newly created notes were not shown on the map which was a bit confusing. And unfortunately it was not a simple fix because the plugin just wasn't designed that way. In offline mode, it held actions in queue which were then executed when you went back online. So there was no note object in queue to draw on the map until after the upload happened and another download was initiated from the API. Also, the most useful way of using it was to zoom out, let it download notes and then go into offline mode to work. This was not at all intuitive.

After receiving a favorable response from the josm-dev mailing list, I began submitting patches in September 2014. Throughout the rest of 2014 I slowly built up behind-the-scenes functionality to download, parse and display notes in JOSM. As of the December release, the notes functionality is actually usable. It is not turned on by default yet because there is still some polishing that needs to be done, especially in the user interface department. But I would love to get some more people testing it. The secret handshake to enable notes in JOSM is as follows:

Download some data. Anything, anywhere. This just initializes a setting that you will change
Open JOSM preferences
Go to the advanced preferences
Search for "notes"
Change the preference osm.notes.enableDownload to "true"
Restart JOSM

In graphical format:

When you open the download dialog after restarting JOSM, you will notice a new "Notes" checkbox alongside the "OpenStreetMap Data" and "Raw GPS data":

After your first download of notes, you will also need to open the note panel or you won't be able to manipulate notes. Click the note icon in the left hand panel. If you still have the notes plugin installed, there will be two identical buttons. Either figure out which one is which or uninstall the plugin. Here is a view of things:

The area download limit for the notes API is far greater than the limit on downloading OSM data. You can download 1 square degree at a time if you are only downloading notes. The limit for OSM data is 0.25 square degrees (although realistically you usually run into the 50,000 object limit before you run into the square degree limit). The above screenshot was downloaded in one request. It is about half the state of Kansas.

Protip: By default, JOSM will only download 1,000 notes per download request. The OSM API has a hard limit of 10,000 per request. What JOSM asks for is controlled by the osm.notes.downloadLimit setting in the advanced preferences. The other setting worth mentioning is osm.notes.daysClosed. This determines whether or not closed notes are downloaded. If set to -1, it will download all notes ever created within the requested bounding box. If it is set to 0 it will only download currently open notes. If it is set to a positive number, it will download notes which have been closed for that many days. By default, osm.org shows notes that have been closed within the last week. To copy this behavior, this setting should be set to 7. I believe the default right now is 1 day.

There is some discussion around these options as well as some of the other polishing that is still needed in the JOSM bug tracker. And of course that is also where any new issues can be reported.

In particular, one of the open tickets which I have already submitted a patch for, but hasn't been accepted yet is the ability to download notes via the note search API. I use this periodically to download all the notes created by Craigslist users (they all start with the word "bounds" so that's the query I send to the note search API) and close all the useless ones.

One last screenshot. One of the features of the notes code is that it can read note dump files. It is actually a byproduct of being able to save and load the note layer. Anyway, here is a screenshot of the latest note dump from http://planet.osm.org/notes/ loaded into JOSM. All 309,000 of them.

Happy noting!

P.S. Dear iD: Your move!

Workflow For Fixing County Borders

noreply@blogger.com (Toby Murray) — Wed, 05 Feb 2014 18:32:00 +0000

Well, my last post got some requests for workflow and I got another snow day off from work today so why not! Follow along as I fix Wyoming's county borders.

Two things you will need to have open: JOSM with the mirrored_download plugin installed and the Wikipedia page for the list of counties in the state. Do a search on wikipedia for "List of counties in <state>" to find it. Here is the one for Wyoming.

Next you need to download all the county relations and their members. I do this using the Overpass API. JOSM has a "Download from Overpass API" option in the File menu (Update: turns out this is a feature of the mirrored_download plugin, not in JOSM core) but we aren't going to use that one just yet because it always adds a bbox parameter to the query which we don't want in this one. Instead, use the "Open Location" option and paste in the following URL after changing the 56 to the appropriate FIPS code for your state which should be listed on the Wikipedia page.

http://overpass-api.de/api/interpreter?data=relation["nist:state_fips"="56"];>>;out meta;

This tells the Overpass API to return all relations with a nist:state_fips=56 tag plus all of the relation members (>>) and finally to add OSM metadata that is needed to edit and upload (out meta). This includes things like version number, last edit timestamp and last user to touch the object. By default it excludes this metadata to reduce download size.

After that finishes you should have a nice outline of the state and all its counties. Make sure the number of downloaded relations matches the number of counties in the state. If not, it probably means the nist:state_fips tag is missing from some counties. If you can mange to find the relation through another means, download it by ID using the "Download object" option in JOSM. Or... I don't know... Ping me on IRC I guess :)

Next, let's load up all the place nodes in the state. For this I use the "Download from Overpass API" option in JOSM. Start off by drawing a box around the desired state on the map. So far I have only done mostly square states. If your state is an odd shape then you can download the bounding box and then select things outside of your state and use the JOSM "Purge" option in the edit menu to remove them from your dataset. Or just ignore them. Then paste this in this text into the query box:

[timeout:300];node["place"];out meta;

Now the screen will look a little busy so I set up a filter to only show me place=county nodes until I need the other stuff. To do this, create a new filter with the filter string

place=county or type:way or type:relation

This will initially hide the county borders and nodes that you want to see so after the filter is created, make sure all three checkboxes in the filter panel are checked. The far right one will invert the filter so instead of hiding whatever matches the filter string it will hide everything except the things that match the filter string. The second checkbox on the left makes them completely disappear instead of just making them inactive but still visible. The first checkbox can be used to quickly disable the filter when you need to see everything.

At this point my JOSM window looks like this:

Note the "Relations: 23" on the relation toolbox matches the expected number of counties in Wyoming.

At this point I usually remove the is_in tag from all the place=county nodes just to get that out of the way. Do a search (CTRL-F) for "type:node place=county" and it should select all of the county nodes. If you have some for a neighboring state loaded that you don't want to touch you can deselect them by holding CTRL while clicking. Then just delete the is_in tag.

Now for the real meat. I just go down the list of counties in Wikipedia. Bring up the first county's Wikipedia page and copy the page title for use in the wikipedia=* tag and note the name of the county seat.

Now in JOSM hit CTRL-F to bring up the search dialog and search for the county seat. If you are lucky it will find exactly one match. Note that because of the filter, you won't actually see the selected object but it is still selected! This could be considered a bug but in this case it works out. If the search found multiple things then you will have to disable the filter and find the right one.

Now:

Hold down shift and select the county node in addition to the city node that is already selected
Select the county relation in the relation list and click the "Edit" button
Add the two selected nodes to the bottom of the relation member list using the 4th button down on the lower right panel of the window
Add the role "admin_centre" to the city node
Add the role "label" to the county node
Add the alt_name tag to the relation in the top panel
Add the wikipedia tag. Remember how you copied the page title from Wikipedia? Just type "en:" and then paste the title.

Here is a shot of Albany county (Wyoming, not New York!) when I'm done with it:

And that's pretty much it. Now just repeat the process through the list of counties.

After I'm done I go back and look for any place=hamlet or place=village county seats and bump them up to place=town. I do this by doing CTRL-A in the relation list, then right clicking and doing a "Select members" which selects all the relation members. Then I use the find dialog to find all place=village and make sure to select the "find in selection" radio button so you are only finding things that are already selected. Otherwise you will promote ALL villages in the whole state to towns! As a sanity check, use the selection toolbox to make sure the number of objects selected matches your expectation. It should be fewer than the number of counties in the state.

I will leave you with a quick video of me editing the next county in Wyoming.

County Borders in OpenStreetMap

noreply@blogger.com (Toby Murray) — Wed, 05 Feb 2014 07:21:00 +0000

I have had a fairly steady relationship with county borders in OSM. They were originally imported before I joined the project in 2010. However they were imported as overlapping closed ways which is not ideal. In some places people had started working on de-duplicating the ways and turning them into relations. However this is tedious work and some of the people who attempted it didn't quite understand how relations worked so it was pretty much a big mess.

Throughout 2011 and 2012 I periodically came back and worked on county borders and eventually got to a place where there was a relation for every county in the nation. There are still some minor differences in tagging but I wasn't as concerned with that and tried not to change the work of other mappers as long as the relation existed and had valid geometry. At the same time I also reduced complexity at state borders. There used to be a mess of admin boundary ways. One for the state, one for each of the counties on either side of the state border and possibly also some city boundaries. They usually didn't match up very well either so it looked terrible. I didn't mess with cities but at least I made the state and county borders share a single way to reduce clutter.

At some point I also noticed (thanks to the MapQuest Open rendering which renders county names) that the place=county nodes were all placed at the extreme eastern border of all the counties. This caused the county names to be rendered in unexpected locations. So I ended up fixing those as well. Here is a screenshot of the last changeset where I moved over 1,100 nodes to the middle(ish) of their respective counties:

The Effects

So that was nice. But there was still some weirdness with counties, particularly when geocoding in Nominatim. Especially if there was a city with the same name as a county. For example, I searched for "Kearney, NE" and the first result was the place=county node for Kearney County in Nebraska. What I was looking for and expected to find first was the city of Kearney which is in Buffalo county. Then there is the issue of Nominatim finding both the county node and the boundary relation when you search for a county. The same thing often happens with cities. Really, they are the same thing and should show up as one result. And sometimes it might. Nominatim tries to be smart about this and does remarkably well but sometimes it needs help.

You might say we should just delete the node since we already have the boundary information. But very few renderers use boundary relations to render place names from. They all rely on these nodes. The same goes for cities, states and even countries I believe. There is also the oddness of what to use for county name. The relations all have a "name=Kearney County" tag while the nodes just have "name=Kearney".

After poking around a little and bugging lonvia (maintainer of nominatim.osm.org) about it a couple of times, I learned a few things about Nominatim and was able to make some fixes. It was my intent to come back to this some day and figure it all out and fix up county relations once and for all. I still hadn't gotten around to that but just in the last few days, new user revent was getting annoyed at the geocoding problems in Texas and really dove into Nominatim to figure out the best practices. He wrote up a diary entry on the subject and we have since then talked together some more on IRC and since I had work cancelled today because of snow I thought I would write up this post about my current understanding of things:

Nominatim is able to "link" multiple elements into a single feature. It tries to do this automatically but doesn't always get it right.
Nominatim uses Wikipedia to help determine the importance of a place.
Nominatim makes use of the alt_name tag.
After editing the map, Nominatim does not always fully reindex an area. It might depend on current workload of the server so things can seem inconsistent. But a manual reindex can be forced by an admin after a large area has undergone a lot of edits.

The linking is important because this allows the county border and the node to be linked into a single feature so that only one result is returned when you search for a particular county. It also improves addressing since Nominatim no longer has to choose between two different things to associate addresses with. This linking can be accomplished by adding the node to the county border relation with a role of "label".

The use of Wikipedia is interesting and (I think) the source of my problems with Kearney, NE that I mentioned above. Nominatim was linking the county node to the Kearney, Nebraska Wikipedia page based on name tag. This helped bump its importance up to where it was returned before the city. Now that I have added the Kearney county node to the Kearney county relation and specified a "wikipedia=en:Kearney County, Nebraska" tag, it links things to the correct Wikipedia page and the search results return the city first, as expected.

The Fix

So to summarize, I believe revent and I have settled on the ideal way to map a county. Using Kearney county as an example:
One relation with the following tags:

type=boundary
boundary=administrative
admin_level=6
name=Kearney County
alt_name=Kearney
wikipedia=en:Kearney County, Nebraska

In addition, the following tags are in widespread use:

border_type=county
nist:state_fips=31
nist:fips_code=31099

I'm not sure the border_type tag is actually used for much but it has a fair amount of existing use and seems to further differentiate borders that are the same admin_level based on local conventions - county vs borough and such. The two nist: tags were part of the import and I have actually used the state_fips as an easy way to download all county relations in a given state to work on. These FIPS codes have been withdrawn by NIST recently but are still commonly used. We may need to look at replacing or augmenting these tags with an ISO standard at some point but for now they work.

In addition to the relation tags the following relation memberships are recommended:

Obviously the ways that make up the border with a role of "outer" in the correct order so that they form a closed ring
The node with the following two tags: place=county and name=Kearney with a role of "label"
The place=city/town node of the county seat with a role of "admin_centre"

The county seat city node as an admin centre (yes, use the British spelling) is kind of a new thing to me. I'm not 100% sold on it but it does seem like useful information to have.

If you have looked at very many county nodes you might notice that the "is_in" tag is conspicuously absent. It turns out, it causes more problems than it solves and should probably be left off or removed. I have always found it a bit silly anyway. OSM is a geodatabase - there is no need to indicate location in a tag.

I have also been promoting any county seats that were place=village to place=town. I know I saw this as a thing on the OSM wiki at one point although I can't find it right now. While they may not meet the population guidelines of a "town" according to the wiki, I think county seats carry extra importance due to their governmental functions. It also means that at least some city names are rendered on maps at low zoom levels in low population areas. Without this promotion you could pan around in the Kansas/Nebraska/Dakotas area for several screens at zoom level 9 or 10 and not see a single city name on the map on osm.org. In some ways this is a rendering issue but I don't feel like this is an egregious case of tagging for the renderer.

Your Turn!

So if you are looking for something to do, jump in and make sure your county is correctly linked and tagged. I have already done Kansas, Nebraska and South Dakota but I probably won't have time to do everything in the near future and I will concentrate on lower population areas first so please feel free to jump in! If there is interest, I may do another post about my workflow to do a state-wide edit. Otherwise you can always find me on the #osm or #osm-us IRC channels if you have questions.

UPDATE! I have since written another post detailing my workflow: http://ksmapper.blogspot.com/2014/02/workflow-for-fixing-county-borders.html

Incidentally, a lot of the same concepts apply to cities as well. If you do a Nominatim search for "Los Angeles, CA" right now it will return two results. One for the node and one for the boundary relation. Making the node a member of the relation should fix that.
(sigh. In doing that search I just saw that the relation for LA is completely broken anyway. Admin boundaries suck.)

In closing, here is a screen shot of the completed county relations in South Dakota:

Too Much of a Good Thing

noreply@blogger.com (Toby Murray) — Wed, 23 Jan 2013 07:13:00 +0000

What makes OpenStreetMap great? Well there may be multiple answers from different people. But one thing is certain: Our geodata is awesome. I mean it is kind of the reason that we exist. So more is better, right? We are, after all, trying to map the entire planet.

But it turns out, there can actually be too much of a good thing. One of my first encounters with such a situation was while doing some fixing up of imported TIGER data somewhere out west. I came across this:

Notice the scale. We are looking at a little over 100 meters of country highway and there are over 300 nodes in this image. This is obviously beyond ridiculous and I can assure you that this particular instance was cleaned up long ago and those nodes met their appropriate end in a special level of /dev/null.

But this is not an isolated incident. I keep coming across such over-noding. So I decided to use my local planet database (which I previously blogged about setting up) to try and find out how much this really happens and where such things come from.

Well, the "ways" table in the database is 85 GB in size and has over 165 million entries in it. Doing anything with the whole data set tends to be slow. So I decided to filter out some things. While discussing it on IRC, Andrew Buck suggested starting by ignoring ways with fewer than 10 nodes in them. After some consideration this actually seemed reasonable. It is hard to over-node something with 10 nodes. So I made a new table with only these ways in it and with only a subset of columns to make it smaller and more manageable. This actually ended up eliminating a lot more ways than I anticipated. It turns out, 75% of the ways in the OSM database have fewer than 10 nodes in them. This left me with about 40.5 million ways to investigate.

The next question is what does it mean to be "over-noded" anyway? Since this is a geospatial database, I can use functions to determine the length of a given way. And I also know how many nodes each way has in it. So a logical unit is nodes per meter.

Next up is to determine what is "normal" for typical mapping. Taking the 40 million ways with more than 10 nodes, I ran some basic statistical functions and found the following numbers:
Minimum nodes/meter: 0.00000282
Median nodes/meter: 0.043
Average nodes/meter: 0.088
Maximum nodes/meter: 65.23

Well that is quite a range. Let's see what is going on here. The way with the fewest nodes/meter turns out to be way 105818922 which is part of the border of Indonesia. It has 14 nodes and stretches almost 5,000 kilometers, averaging one node every 350 kilometers. It could maybe use a little more refinement but it's a national border in the middle of the ocean... it is probably not too bad.

On the other end of the spectrum is way 44342937. It supposedly represents a round "building" with a diameter of about 1.5 meters. It has 339 nodes. What the... WHAT?! What is this, the world's most accurately mapped dog house? Since it is likely that this way is going to be deleted in the very near future, I include here a screen shot of it loaded up in JOSM:

Yes, that is a tiny circle in the middle of someone's back yard. All that red around it is JOSM trying to draw its normal way direction arrows on such a ridiculous object.

Now that we have examined both ends of the spectrum, what's in the middle? Well, there is way 101521179. 174 meters long with 14 nodes which yields a nodes/meter reading of 0.08000019. That is... well... entirely reasonable. The fact that the median is half of the average would seem to indicate that the numbers are skewed towards the high end by a comparatively few number of ways with a very high number of nodes - like that one with 65 per meter.

So if 65 nodes/meter is the extreme upper limit and most people map at less than 0.1 then what exactly is "too much"? Well I don't really know but I'm just going to throw out a number for further analysis. Let's make it... 1 node per meter. This limits the number of ways down to about 105 thousand.

Since my database has geographic knowledge about these ways and I have the number down to a reasonable level, let's hook up QGIS to it and make a map of the ways with more than 1 node per meter. Let's start with the US:

Well that's actually not as dense as I was expecting. It looks like some of the TIGER over-noding might actually come in at under 1 node/meter. Which kind of makes sense if you think about it. A lot of TIGER ways are really long and the over-noding is sometimes clumped into one section so on average the nodes/meter ratio will be less than 1. But for fun let's look at what other interesting things might be in this data set. How about Europe?

Well now that's different. What's with the outbreak in France?! I have my suspicions. Let's see how this works out...

There is a "source" tag that is often used in OSM data to indicate where a certain map object came from. It is especially often used during imports. Technically it usually makes more sense to put the source tag on the changeset instead of the map object itself, especially for large imports but a lot of people still put it on the map objects anyway. So let's see what these over-noded ways reveal in their source tags. Here are the results of a query that groups things by source tag and counts how many ways with a node/meter ratio of over 1 have that source tag. I truncated the source information at 70 characters for display purposes.

 WAYS  | SOURCE
-------|-----------------------------------------------------------------------
 62995 | cadastre-dgi-fr source : Direction Générale des Impôts - Cadastre. Mis
 22713 | 
  3535 | extraction vectorielle v1 cadastre-dgi-fr source : Direction Générale 
  1313 | Bing
  1045 | 3dShapes
   744 | bing
   687 | Kolding Kommune
   628 | NHD
   608 | dcgis
   602 | WakeGIS
   585 | WroclawGIS
   526 | Planimetria de Vitoria
   482 | MassGIS Buildings (http://www.mass.gov/mgis/lidarbuildingfp2d.htm)
   454 | http://www.bakersfieldcity.us/gis/downloads/gis_spatial_data.htm
   393 | NextView
   390 | Regione Emilia Romagna
   369 | cadastre-dgi-fr source : Direction Générale des Impôts - Cadastre ; mi
   349 | kapor2
   336 | Bing Sat
   324 | SO!GIS Import
   260 | geoimage.at
   244 | Regione_del_Veneto_LR28_16.7.1976_Formazione_CTR_auth_39164-5700-1100_
   230 | http://www.sogis.ch SO!GIS Import
   225 | MGC
   214 | Kreis_Viersen_Katasteramt_2012_06
   214 | lukr
   206 | Ajuntament de Girona
   184 | CCH
   183 | NRCan-CanVec-10.0
   175 | Orthophotos 2011 du SITG (Système d'Information du Territoire Genevois
   133 | DEP Wetlands (1:12,000) - April 2007 (http://www.mass.gov/mgis/wetdep.
   128 | vuv:dibavod:a05
   125 | http://www.roseville.ca.us/services/maps_n_data/data_clearinghouse.asp
   115 | City of Kamloops
   112 | cadastre-dgi-fr source : PaysDeBrest - 20100331
   105 | OS_OpenData_VectorMapDistrict

Well then. I recognize that top source tag as the ongoing import of french building outlines and some other features from some cadastre data they got their hands on over there. A spot check shows that some of this isn't actually because of excessive nodes being used but rather the odd way in which the import is creating building geometries. For example, here is way 67157454.

Note the wall=no tag. I believe this implies that it is some kind of porch or veranda. It has a rounded front which is where all the nodes are that make it have a nodes/meter ratio of just over 1.0. If a human mapper had mapped this building it would have likely been a single way that included the porch as well as the rest of the house instead of the 8 individual areas that the import created. This would have resulted in a single 70 meter long way with 47 nodes which comes out to 0.67 nodes/meter.

Checking a few more of these ways in France, it does look like some care was taken to prevent over-noding. Most of them are just barely over 1.0 nodes/meter. And at this point it should also be noted that closed ways are actually having their first/last node counted twice because of the way the database stores the node membership information. Since I was originally looking for over-noded highways I didn't take this into account. So technically a lot of these French buildings might be just under 1 node/meter. But the fact that there are so many of them right on this (arbitrary) limit is still interesting.

There are a few other source tags I recognize as well. CanVec is imported from Canadian government data. NHD is the National Hydrography Dataset here in the US from which some people have imported rivers and lakes. I know several of these water feature imports have had problems with ridiculous over-noding. Some of it has been fixed but a lot remains. MassGIS is the Massachusetts GIS office which has been used for some local imports. In particular, the most over-noded way that I showcased above is from this MassGIS building import.

Of course the second most popular source tag is blank which doesn't tell us much. A tiny random sample shows there are a couple of imports that didn't use any source tag and some are just very detailed manual mapping.

I think finding what I was originally looking for (over-noded TIGER ways) is going to take some more digging. This post kind of got hijacked by the big red blob in France but I think I'll call it quits for now and do some more poking to find the TIGERs I'm after. If there is a lesson so far, I think it can be summed up as:

Sanity check your imports! Tiny objects with a large number of nodes are an obvious sign that there is something weird going on in your conversion to .osm format. It is possible that the object was represented as a parametric curve in the source and the conversion to OSM format tried to recreate that as closely as possible within the constraints of our x/y coordinate system.
Imports are not human data. Even in France where this building import is generally viewed as a good thing and a lot of checking and verifying of the data has taken place, the data is still very distinctly different from most of the OSM database that has been created by hand. This may not be a bad thing in every case but it is definitely something to think about when proposing and executing an import.

Lastly, I leave you with one more way. As I was randomly browsing the 105 thousand ways, I happened to click on way number 129485933 which has 1.00526 nodes per meter. (again, with the first/ last node double counted) If you look back at version 1 and 2 you will see a certain someone's user name. Turns out I am the culprit! This way was originally created on my phone during the baseball game we attended during SOTM 2011 in Denver. How random is that?

Preparing for Operation Cowboy

noreply@blogger.com (Toby Murray) — Wed, 21 Nov 2012 17:10:00 +0000

If you haven't heard, there's a major mapping event about to happen. Dubbed Operation Cowboy, this will be the second event of its kind, the first being Night of the Living Maps which happened back in February. As described on the wiki, these events are kind of like a global LAN party for maps. For one weekend people who choose to participate will focus their mapping efforts on a common goal. This time the organizers decided to focus on the USA.

We of course have plenty of mapping to do around here. Cleaning up and updating the TIGER road import is kind of a never-ending task. In some of the sparsely populated areas of the country there aren't really any local mappers to speak of so there is ample work for "armchair mappers" to come in and do some map maintenance.

Considering the audience and the event, I decided to try and get a few things tweaked before Operation Cowboy goes down.

TIGER 2012 road name tiles

First up was improving the TIGER 2012 road name tiles being served up by the OSM-US tile server. They were originally set up by Ian Dees and are a great resource for tracing roads that were built since the original TIGER data was imported. The only issue I had with them is that they show the street name in the Census Bureau's abbreviated format. In OSM we prefer to use unabbreviated names because it reduces ambiguity. Does "St" mean "Street" or "Saint"? While that one might not be too hard to figure out and expanding "Ave" to "Avenue" might be trivial, there are a lot of abbreviations in the data and some of them are not easy to guess if you aren't familiar with the area.

Any guesses as to what the following mean? Brg. Cll. Natl Lkshr. Ofc. Pso. RM. BIA Rte? (see answer key at end of post) Most people in the US can probably guess some of them but there are definitely some regional ones in there that many people may not be familiar with. Then add in abbreviated directional prefixes. And now suppose you are a European trying to figure all of this out during Operation Cowboy. It's going to look like a secret code.

I won't go into the technical details in this post (might save that for another one after I clean up the process doumentation) but it did involve 27 GB of data in 3,200 .dbf files, about 10 different command line utilties and several large SQL queries. The end result was a mapping of 10.8 million line IDs (linearid field in TIGER speak) to fully expanded road names which I applied to the rendering database. The result can be seen on this tile preview page.

Imagery preset updates

Related to the road name tiles, I noticed that Potlatch2 had a TIGER road name tiles preset in the "Background" menu but it was still pointing at tiles with 2011 data in them. Thanks to the magic of github, I was able to get that fixed easily.

Then I noticed that the JOSM imagery sources page didn't have this tile set listed at all. That one was a little trickier to fix. It involved coming up with a simplified polygon of the US border to define the available extent of the tile layer as well as putting a base64 encoded image into an XML tag on the JOSM wiki. The image+base64+XML+wiki thing seems a little nasty but the JOSM functionality that this enables is actually pretty neat. If you download OSM data inside of the US, it will pop up a notice that informs you that that there is additional background imagery available and lets you add it to the "Imagery" menu with a single click. The base64 encoded image is displayed next to the menu option. In this case it is the TIGER logo:

RAWR!

TIGER tag removal

The TIGER road import put some tags on imported objects that weren't really necessary. As people edit roads, some of the tags become even less useful as the information from the tiger tags diverges from the "normal" OSM tags. Some of the tags were useful at the time for the upload process itself but are now just annoying bloat that confuses new users.

Back in July I started a mailing list thread about having editors automatically drop some of these tags like they already drop the created_by tag. Out of that discussion came a JOSM trac ticket and patch that does exactly this. Right before uploading, JOSM checks to see if any of the modified objects it is about to upload have "discardable" TIGER tags on them and removes them.

In September, I opened a ticket against Potlatch2 with the same feature request. Since nothing had happened with that ticket and I already had the Potlatch2 repository cloned from the tile URL change pull request, I decided to take matters into my own hands. I had never so much as glanced at ActionScript but fortunately P2 was a breeze to get running in a development environment. The developer documentation page on the wiki got me up and running in a matter of minutes. Although I did have to give ant more PermGen space to get through compilation. I should probably update the wiki with that tidbit...

After a little over an hour I had another pull request fashioned which Richard was kind enough to merge the next day. I believe this went live on osm.org within the last few hours. So now every way that is touched during Operation Cowboy will help to clean this clutter out of the database without the user having to do anything.

YEEEEHAW!

So that's how I prepared for Operation Cowboy. Unfortunately the actual event will take place over the Thanksgiving weekend which means I will be in Nebraska with very limited internet access so I won't be able to participate much with actual mapping. I will probably manage to hang out in the IRC channel a bit to try and field some questions if they come up. Please join me! (#opc2012 on irc.oftc.net)

Answer key for TIGER abbreviations:

Brg = Bridge

Cll = Calle (Spanish)

Natl Lkshr = National Lakeshore

Ofc = Office

Pso = Paseo (Spanish)

RM = Ranch to Market Road

BIA Rte = Bureau of Indian Affairs Route

Licensed to Map (What happened to Los Angeles?!)

noreply@blogger.com (Toby Murray) — Wed, 24 Oct 2012 03:53:00 +0000

I recently got back from the State of the Map - USA conference. It was great and you should have been there! But for those who weren't... I presented one session on Saturday afternoon with the same title as this post. Instead of just putting my slides out there I thought I would write a blog post to tell more of the story. This is a fairly long post but don't worry, it has a lot of pictures! (click to see full resolution)

So, the license change. It happened. We lost some data. But what happened before that to try and save as much as we could? And what exactly did we lose?

First, a brief timeline:

Many moons ago, the OSM Foundation voted to change our license from Creative Commons to the Open Database License. This actually happened before I knew that OSM existed.
In order to make the change, permission had to be secured from everyone who had contributed map data to the database.
Contributions from anyone who did not agree to the new terms had to be removed from the database.
At the end of 2011 the board set April 1st, 2012 as a target date for the license switch. This turned out to be wildly optimistic but at least it was a concrete goal to shoot for instead of just an ongoing process with no end in sight.
The process of removing non-relicensable data was done by a bot that went through the entire database in the 2nd half of June
After a little more cleanup, the first ODbL planet file was finally delivered in September.

Even though the April 1st target date was not really reasonable to hit, (especially in hindsight of course) it still gave the community something to work towards. And work we did.

Contacting inactive mappers

The first priority was to contact undecided users to make them aware of the change and that they needed to log in and indicate a decision. There were several rounds of emails sent out from the foundation to undecided users, trying to get them to respond. In addition to the emails from the foundation, users actively contacted other people in their area to try and make them aware of the decision. Lastly, the foundation supplied the account email address of some of the accounts with the most map data to a small group of volunteers for more targeted contact.

The result of this contact effort can be seen in this graph that I have been presenting since the beginning of the process:

The green line corresponds to the scale on the left while the red line follows the scale on the right. Of those who responded, it was a 99.5% landslide in favor of re-licensing their data. You can see that the people who were opposed to the change were very quick to log in and enter their decision. The big bumps in the green line clearly show when the mass emails were being sent out by the foundation.

I was the volunteer who did targeted contact here in the U.S. It was an interesting project. Simon Poole came up with a prioritized list of users to contact per country/region based on his site which shows undecided users ranked by how much map data they have in the database. (http://odbl.poole.ch/)

I ended up contacting people through a variety of websites including obvious social networking sites like Facebook, Google+, Twitter, Flickr and LinkedIn. I also sent a few though more obscure sites like tripadvisor and soundcloud. Beyond that, I attempted to determine their real name and any contact info I could get my hands on. A simple web search with their OSM user name plus the location of their first edit was remarkably effective. Often it would yield a twitter account which would then give me a real name and lead to accounts on other sites. Here is a screenshot of the spreadsheet I used to track my contact with users:

I kept track of dates I sent messages, when I heard back and any accounts around the internet that were possibly related to the user in question. All in all, I attempted to contact a total of 168 users. 71% of them accepted the new contributor terms. Only one or two of those who responded did not accept the new terms. The rest I was unable to contact.

As part of my contact efforts I ended up calling 7 people on the telephone. This led to a few awkward conversations. One person said "I don't remember giving OSM my phone number" to which I replied "Well... you didn't. I stalked you across the internet." Another call landed me in a conversation with a high school student's mother. When I asked to speak to him she asked "who is this?" to which I responded that her son did not know me but that I am from a website that her son had used. I am happy to report that both of these users agreed to the new terms within 5 minutes of talking to me! However a few times I felt kind of like this:

Found on http://www.freewebs.com/jhnbytwoo/

Remapping

Besides contacting undecided users, there was also an effort to remove and replace the contributions of users who would not agree to the new terms. There were several tools to assist us in this effort. Unfortunately I didn't get good screenshots of most of them. At the time I wasn't planning on doing a presentation about this and now that the license change is over, they are either shut down or not displaying useful information.

But they basically all had a different way of highlighting map objects that were likely going to be either deleted or modified by the removal of license tainted data. Both JOSM and Potlatch 2 had a license change plugin/mode that would draw red halos around objects that were in danger of deletion and yellow ones around things that would be modified. Next was the OSM Inspector license change view. It also displayed objects that would be touched by the bot and let you click on them to get details as well as links to the object history and edit links to go fix it. These were based on the "Quick History Service" which was a pretty good approximation of the algorithm the bot would use to remove data. There was also cleanmap/badmap which was two different map renderings. Cleanmap showed a rendering of what the map would look like after the license change and badmap showed only things that needed to be remapped.

The license bot

In the second half of July, the "license bot" was run over the planet to remove all license tainted data from the database. After the full bot run there was some additional cleanup to do. For example there were some obvious cases of copy/paste remapping where people "remapped" by straight up copying the data of users who had not agreed to the new terms. This is obviously copyright infringement and needed to be taken care of. In the bot removal process, data which was originally created by decliners or non-responders was completely removed from the database. Objects which had been modified by decliners/non-responders were reverted to an older state to remove the data that couldn't be relicensed.

The official number I heard after the process was done is that OpenStreetMap lost 1.2% of its data. Of course removing 1% of the Interstate system makes it completely useless for routing across the country so the true impact of the bot was much higher but I'm not sure it is possible to put a specific number on it. The effects of the bot were also highly variable by location. Australia and Poland got hit disproportionately hard as did some cities here in the U.S.

The bot had a slight blind spot having to do with splitting ways. When a way is split in OSM, a brand new way is created and attributed entirely to the user who split the way. No link is kept to the user who originally created the long way. This issue was known about ahead of time but the effort required to try and guess on way splits was deemed too high in comparison to the difference it would have made on the data.

So, what does 1.2% of OSM data look like? Here are two graphs from http://osmstats.alltogetherlost.com/ which show the total number of nodes and ways in the OSM database over time:

You can clearly see a small bump in July. But really, it's little more than a blip. The growth of data in OSM appears to be kind of unstoppable.

The bot in America

Since I presented this information at SOTM-US, I focused on some U.S. specific issues. We have some unique considerations, mostly having to do with the fact that a lot of our road network is imported from the TIGER/Line data set which is public domain. This means that the vast majority of our road network is not impacted by license considerations. However some of the TIGER fixup work that has happened since the import was definitely impacted. For example, there was one armchair mapper from Europe who did a lot of work on the interstate system in the eastern 1/3 of the country which was lost in the license change. As I mentioned above, way splitting was not taken into account by the license bot. Since splitting ways is a relatively common thing to do when fixing up TIGER data, this did lead to some data being removed that was technically from a public domain source. Luckily it tends to be relatively easy to restore this data, especially on interstates where a lot of the splitting happened to map bridges which are clearly visible in aerial imagery.

To visualize the bot's activities, I made a map by analyzing the daily diffs that were published during the time the license bot was running in America. It only shows node edits. Red means the bot deleted a node, blue means it modified a node, likely reverting it to an earlier version.

Well that looks kind of bad. But it is exaggerated a bit because of the rendering style and low zoom. Let's look at some individual locations. First up: Kansas.

Well that's pretty good. I got most of the license problems here in Kansas taken care of either by contacting users and getting them to agree to the change or by remapping things.

Next up is Dallas. On the national map it looks like a fairly decent sized blob.

Upon closer inspection, it isn't really that bad. There are a few neighborhoods that were removed from the map and some damage to that highway heading east. But the overall structure of the road network isn't too bad.

Next, South Carolina:

That's a bit more serious. Obviously there was a lot of damage to the interstate system here.

And finally, Los Angeles:

OK, now that's pretty heavy. Most of it is due to a single, very prolific, mapper who refused to agree to the new terms.

So we know where the license bot was most active. But what exactly did it do to our data? Turns out, many different things. Here is an example of some pretty obvious damage:

The original TIGER import was pretty terrible in this area. A declining user corrected a bunch of the road geometry. Then an accepting user came along and added some details like turning circles and gates. When the bot ran, it did not touch the new additions made by the accepting user but moved all the nodes touched by the declining user back to their original positions, shearing the map rather severely.

Here is a somewhat amusing example:

In this case, the original TIGER had this onramp going the wrong direction. A declining user reversed the direction of the way. Then an accepting user came along and refined the shape of the way by adding more nodes to it. The bot reversed the order of the original nodes but not of the new nodes added by the accepting user, leading to the odd "zig" in the middle.

Now for a little more subtle damage:

You can see some obviously weird zigs and zags in the roads but nothing seems to be completely out of whack here. However, when viewed in JOSM, the damage becomes clearer.

In the upper left, there are nodes that used to be part of a way. Both the nodes themselves and the way made it through the license bot. But the fact that the nodes were a part of the way was lost leaving empty nodes and a misshapen way. In the lower part of the image you can see that there is a way which does not actually connect to any of the ways it crosses. So much for routability.

Fixing the damage

After the license bot ran, the community immediately started focusing on how to fix the damage. Several tools were developed or adapted to help guide and focus this effort. At the end of the day, we have risen to the challenge and I have actually been surprised at how quickly most of the damage has been fixed.

I made a rough graph of remapping activity based on changeset comments. I used the weekly changeset dump along with my my ChangesetMD tool to create a database of changeset metadata which I then queried for these keywords in the changeset comments: odbl, license, remapping, redaction.

It starts back in December 2011 when remapping started in earnest. The orange line follows the axis on the left and indicates the number of objects touched in these remapping changesets. This includes deletes, modifications and additions. The blue line follows the axis on the right and is just the number of remapping changesets created per day.

I believe the huge spike in the orange line at the end of January was me and a few other people really putting our backs into getting the coastlines cleaned up. Paul Norman came up with a good way to highlight shoreline ways that were in danger of being damaged by the bot and it turns out there were a lot of them. I nuked most of the shoreline along the west coast and replaced it with NHD data. Some others did the same or manually remapped the shoreline around the Great Lakes and the east coast.

In May and June, remapping activity seems to have slowed to a crawl. I think people were just getting tired of it. I certainly was. The big blue spike on the right happened a couple of days after the license bot started running. It was actually still in progress at the time. But since Europe was hit first, it seems that the Europeans made quick work of cleaning up after it.

My own contribution to the remapping effort was a map that highlighted ways that seemed to be missing their highway=* tag. I noticed that a fair number of interstates and on/off ramps had a lot of their tags stripped but one of the following tags remained: oneway, lanes, bridge, tunnel. Any of these tags without a highway tag is rather suspicious so I put them on a map to highlight and guide people to them for remapping. Once in the area of such a way I often found other problems in addition to the missing highway tag. Here is an animation of how my map evolved between August 8th and September 30th:

It started out only showing ways with a oneway tag. You can see some new red appear on the map when I added the other tags to the list. By August 28th, the U.S. was completely clear on this map. The Europeans still have quite a mess to clean up. You can find the map still online here: http://ni.kwsn.net/~toby/OSM/maps/redaction.html

Another effort to detect problems on major roads was the routing grid that Kai Krueger set up. It queried OSRM periodically for a route between 40 cities and compared the distance to a reference distance. If it was off by a lot, it indicated a likely problem along an interstate or US highway between the two cities. The difference was color coded and put into a display matrix. Here is another animation showing how the routing grid evolved between July 21st and September 30th:

Again, you can see that things got fixed up pretty quickly. Also, I'm not sure why everything went orange again right at the very end.

One other strategy I used to fill in hard hit areas was to re-import the area from new TIGER 2011 and even some 2012 data when it became available. To do this, I downloaded both the shapefile and the feature names relationship file from the Census Bureau. The feature names file allowed me to do a better job of expanding street names from their abbreviated forms. I joined the two files in QGIS and then ran them through ogr2osm using my own translation file which extracted only a few of the shapefile attributes into OSM tags and expanded abbreviations. This gave me a .osm file which I was able to load into JOSM and then selectively copy roads from there into OSM.

Here is a screen shot of JOSM while I had some TIGER data loaded in the background with my "inactive" color set to a bright green to contrast the red of selected ways in the OSM layer on top of it. This highlighted areas that were missing in OSM. My apologies to any red-green colorblind readers.

After zooming in on the desired area I used the search string "type:way allinview" to automatically select all the ways in the TIGER layer, copied them and then pasted them into the OSM layer. Then of course I had to do some quality assurance to connect the new ways to the existing data and remove some common TIGER problems such as over-noded ways and duplicate ways that were created when TIGER had more than one name for a road. Typically this was just things like "Drive" vs "Avenue" or some other minor spelling difference.

One excellent tool that Martijn van Exel came up with is called "Remap-a-tron." I did not talk about it at the conference at all because Martijn did a presentation about it right before my presentation. But you can read about it in his own blog post. Basically, he did some digging to find ways that had been deleted by the license bot. Then when you hit the web page, it would display one deleted way at a time and give you the opportunity to go remap it in your favorite editor using aerial imagery, surrounding data that wasn't removed or whatever other source of license acceptable data you could find. After remapping the way you would come back and indicate that you had remapped it. It would record the remapping and immediately show you the next way that needed attention. This tool has now been rebranded as "Maproulette" and is available at http://maproulette.org where it has been retasked to find other common map errors that are easy to fix.

Finally, there is also a "Redaction Bot" view in the Geofabrik OSM Inspector tool. It shows all objects that were touched by the bot. You can click on them to get details about what the bot did to a particular object. I don't think it tries to match things that were deleted by the bot to new mapping done to replace them so over time the signal to noise ratio of this tool will decrease. Here is a screenshot of it with one way selected.

The End

In conclusion, the license change definitely had a major impact on OpenStreetMap. Both our map data and the community were impacted, sometimes to great detriment. However we are getting past it remarkably quickly thanks to the hard work of mappers like you. Some damage will remain in the map for a long time to come. Deleted points of interest aren't nearly as obvious as broken interstates and will take much longer to fill back in. But at the end of the day, we will survive and continue to thrive. In some ways, 2012 really was the year of OpenStreetMap, despite the license change. Let's make 2013 even better!

Map the planet!

Making a location badge using OsmAnd and the MapQuest Open static map API

noreply@blogger.com (Toby Murray) — Fri, 08 Jun 2012 02:50:00 +0000

I'm about to embark on my second Biking Across Kansas tour. When I did it last year, I used Google Latitude to create a little "My Location" badge which I put here on my blog to let people track me across the state. The Google one looks like this:

However at some point during the last year I noticed that OsmAnd added a feature that allows it to periodically ping a web service of your choosing with the current location of the phone. This allows me to build my own location publishing service.

I did a little searching for how other people have used this feature in OsmAnd and found a few people talking about feeding the data into a Google map display. The example in the OsmAnd wiki posts the location to a Google doc which then somehow magically updates a Google map display. No offense to Google but I'd rather use OSM based products. Fortunately the good people over at MapQuest Open provide a service that suits this situation: The static map API.

Behold my creation (and location):

Note that this is no longer an active link since BAK is over. This is just a screenshot of what it looked like.

Now for the details. The system has two components. A web service to collect the location information from the phone and a display page that creates a pretty map to look at.

I hacked together some PHP in about an hour. Not the biggest fan of PHP and I'm pretty sure I'm doing some things wrong but I got something working and don't have time to make any improvements before I leave for BAK in under 12 hours so there you have it. The code is on github.

The location web service takes the GET request from the phone and writes the information contained in it to a file. The display page reads the file, pulls out the coordinates and uses them to construct a URL that retrieves a static map image from the MapQuest Open static map API. It isn't an interactive map, just a quick "I am here" image. However I have wrapped the image in a link that takes youto an interactive map on openstreetmap.org. Throw that into an iframe tag and you're good to go!

Possible future improvements might include the following

Perform a nominatim query to get a nearby city name and display that
Better controls over the accuracy of the position displayed. The google badge lets you only reveal which city you are in if you wish. Currently I only allow rounding coordinates to a given number of decimals (default is two)
Dynamic sizing of the map based on parameters passed in to the iframe from the parent page. This example is rather huge and not really a "badge" but that's what I want in this situation.
Track more than just the last location ping and display distance traveled between updates or something else interesting. The static map API is pretty flexible with displaying multiple POIs or even lines or shapes.
...
PROFIT!

I am packing my bags as I finish this post so I don't really have time to brainstorm anything else. I'm sure others will have ideas :)

See you in 500 miles!

Remapping using TIGER 2011

noreply@blogger.com (Toby Murray) — Sun, 11 Mar 2012 22:36:00 +0000

The license change mapping process I talked about in my previous post works well where there is still some clean data to go off of and you're just having to redo some work that a decliner did in splitting ways or adding some nodes to refine the geometry. But what about when an entire area is on the chopping block? I present the OSM Inspector view in Glendale, CA:

The area I worked on was just south of here and was actually even worse although I didn't get a "before" screenshot. Less yellow, more red. Here is what it looked like on Simon Poole's "badmap" rendering. Pretty much the entire road network is going to get nuked:

Most of this data is from a single, very prolific contributor who is not likely to agree to the new terms. It is also highly unlikely that anyone in the area is going to be able to redo all of his work before April 1st. So I reached for a bigger paintbrush: TIGER 2011.

To use TIGER data in OSM is no simple matter. For one thing, the Census Bureau uses abbreviations rather heavily and OSM is strongly anti-abbreviation, and for good reason. Unfortunately it is not a simple matter of throwing a regular expression at it and expanding all "Ave" to "Avenue" and "N" to "North" - what if there is a street named "N Street" which runs parallel to "O Street"? But then again, there may be a legitimate "North Street" somewhere. The TIGER data does indicate what pre- and post-fixes are present on a given road so the full name can be reconstructed with some effort. But to use this information is a little cumbersome. And of course the accuracy is always an issue but that's kind of a given with this data.

First I downloaded the shapefile for Los Angeles county. Then I had to get the "Feature Names Relationship File" and use QGIS to join the two based on the LINEARID field. I ended up with a shapefile with all the name information present. The complication here is that sometimes roads have multiple entries in the relationship file and I'm not sure what QGIS does in this case. Since I was only working on a small area, I reviewed the data and made sure it all looked sane. Since a whole county's worth of data can be hard to work with in JOSM, I clipped out just the area I wanted to work on and exported that as a new shapefile in QGIS.

The next step was to convert this data to .osm format. I used a new version of the ogr2osm python script written by Andrew Guertin to do this. To translate shapefile attributes to OSM tags requires a translation file. I cooked one up based on some previous traffic on the talk-us mailing list for mapping TIGER road types to OSM highway=* tags. For the name, I did some more looking at the data and came up with a bare minimum implementation for the few hundred highways I had to work with. I don't expect this to work well in other places without modification. But here is what I came up with. Patches welcome :)
https://github.com/ToeBee/ogr2osm-translations/blob/master/tiger2011.py

And that was the easy part.

Now to integrate this data into OSM. First I downloaded the entire area in JOSM, selected everything and ran the LicenseChange plugin against it. Of course the whole screen lit up in red. Then I opened the .osm file generated by ogr2osm. I also changed my JOSM settings to render the "inactive" layer in a bright color so I could easily see where things were missing.

Then it was just a matter of spending a few hours deleting ways in the area, switching to the TIGER layer, copying the missing roads and then pasting them into the "live" layer of OSM data. Then of course I had to find all the places where the old and the new data intersected and recombine them to maintain correct topology. The JOSM validator can help with this although I'm sure some mistakes made it in. The larger an area you can do at once, the less fiddling you have to do with the topology since all the data you copy/paste from the TIGER data will (for the most part) have correct internal topology. But bigger areas are harder to keep track of as you go. I usually tried to find relatively self-contained areas with as few as possible connections to the rest of the world. This spot on the map is a little unique in that it seems to be in a hilly area and it isn't your average grid pattern of streets. So there were actually isolated networks that I could deal with in chunks.

Of course I always had to keep an eye out for TIGER strangeness. Some fictional roads and that kind of thing. 99% of what I touched was just residential roads. Where I did touch tertiary/secondary ones I did some additional comparison since TIGER's road classifications don't always match up with reality.

This whole process was kind of complicated when there was actually something I didn't have to replace. I always prefer local mapper contributions over TIGER data so I had to make sure I didn't steamroll over what little clean data I did manage to find.

So at the end of the day (literally) the area now looks like this on badmap. Obviously I was only concentrating on the road network so the parks are still there.

And even from a lower zoom, there is a noticeable hole in OSMI:

Definitely an improvement but there is still much, MUCH more work to do :(

The changeset I did this in can be seen here:
http://www.openstreetmap.org/browse/changeset/10937678

License Change Mapping

noreply@blogger.com (Toby Murray) — Sun, 08 Jan 2012 06:04:00 +0000

So far I have avoided saying anything about the OpenStreetMap license change on this blog. It is a very sensitive subject for a lot of people for a million different reasons. This is not a post about how right/wrong the license change is. I personally have agreed to the new terms but then again I don't really care what license my data gets redistributed under. While I prefer (in most situations) a share-alike clause, if we were switching to public domain instead of ODbL, I would probably still agree. Whatever.

But the license change is happening and we must prepare the map for it, ahead of the April 1 cutover date set by the license working group. Today I decided to take a look at what it would take to get I-70 in Kansas cleaned up, license-wise, and thought I'd share my findings. Note that this particular process may or may not apply to other areas. The interstate system is kind of special in that it was imported from a public domain source and has very long stretches that are identical except for tags that don't really matter. This means that there isn't really very much unique or unrecoverable information present. If there is a small section of road (say a bridge) that is dirty, it can just be removed and replaced from clean data that surrounds it plus imagery to verify that it is indeed a bridge. However whenever possible it is preferred to maintain the history of objects in OSM so I don't want to just nuke and replace large sections of I-70. Plus, deleting and recreating ways gets more complicated because of the route relations that they may be a part of.

A majority of I-70 in Kansas is already clean but there is still a noticeable fraction of dirty objects. The goal is to end up with all objects being "clean" to re-license under ODbL. First task: find dirty objects. For this I use the "License Change" view in the OSM inspector tool created by Geofabrik: http://tools.geofabrik.de/osmi/?view=wtfe

Here is a section of I-70 I will be working on.

You can mouse over the things on the left to find out exactly what the colors and symbols mean. Basically red and orange are bad. In this screen shot I have already clicked on the right-most red dot. This brought up the information in the upper right panel which indicates that there are two items of interest near where I clicked. Node 388499694 and way 33918862 were both created by declining user moonwashed.

Now I click on the clock icon of the way. This brings up Ian Dees' "Deep Diff" service which displays a nice visual history of the way.

From this I can see that the declining user created the way and made one edit to it. The subsequent 11 edits have all been by users who have accepted the new terms. The only original tags that have survived unchanged from the declining user are highway=motorway and oneway=yes. In this case, I consider these both trivial tags that should not prevent this way from being re-licensed. I have yet to encountered this along I-70 but there may be non-trivial tags that were added by a declining user. In this case, such tags would have to be removed to make the way "clean."

Now, what about the nodes? I have the "remote control" feature enabled in JOSM which means that I can go back to the OSM inspector and click on the icon with the little "J" on it next to the way. This will cause JOSM to automatically download a bounding box around that way and select the way after it is downloaded. (~~This doesn't seem to work from Opera though~~ see comment below) I also have the License Change Plugin installed which I used to do a license check on the selected way. Here is the resulting JOSM window:

I have also selected all of the dirty nodes in the license problem list. Now if I click the "Select" button, it will select all the dirty nodes. Next I inspect them to make sure none of them are nodes that connect to other ways. This could cause problems in the next step. And in fact, the one at the left end of the way is a junction node with a bridge. So I will unselect it using a CTRL+click and deal with it later. Now that only dirty nodes are selected, I delete them all. This will leave me with a way that is of course horribly wrong in terms of geometry. But now I can turn on the Bing imagery and go back and re-improve the way from Bing using the ImproveWayAccuracy plugin:

And of course the screen shot software intentionally left out the mouse cursor... but it is where the green line makes a bend which is where I am about to add a node. Now I check along the rest of the way to make sure everything else is ok.

Now back to that dirty node I left out. The difficulty in this case is that the node is part of a 2-node way (a short bridge) so if I delete the dirty node, the bridge will get deleted as well. Since I don't want that, I will create a new node in the middle of the bridge, then delete the dirty node. This will create a gap between the two ways which needs to be fixed by re-extending one of the ways to connect with the other. Then all that I need to do is fix the position of the junction node according to imagery. This leaves all the nodes clean.

Note that for untagged nodes, simply moving might be enough to clean them, depending on what rules end up being applied during the license chage. But simply moving a node by 1 meter to make sure it is ODbL clean doesn't give me warm fuzzy feelings. I'd rather just scrap them and start from scratch to avoid uncertainty.

But the way itself is still considered dirty. For the following 3 reasons, I believe that the way can still be re-licensed:

The way was originally created from a public domain import. All the declining user did was split it, probably to make the bridge.
All tags are either trivial (as mentioned above) or have been added/changed by agreeing users
The geometry is now completely clean as I removed all dirty nodes and recreated them as I would when mapping something from scratch: aerial imagery and GPS traces

So to indicate this, I will add the following two tags to the way:

odbl=clean
odbl:note=decliner touched tags that have been overwritten or are trivial. All decliner nodes have been removed

The first tag is documented on the Remapping/License Change View on OSM Inspector wiki page. I'm not sure if there has been an official nod to obeying this tag in the final license change process but I'm not the only one using it so hopefully it will. And if not, it gives me a simple tag that I can query on to retrieve all ways that need further work. The second tag is just something I decided to use to briefly explain why I think this way is clean.

And that's it. After the OSM inspector updates its data, this way should show up as a green line instead of a red one.

This might seem like a lot of work but once I worked out a process I was able to move along pretty quickly. Plus, there were only a few ways where I had to deal with more than a few nodes. I got all of I-70 in eastern Kansas done in a little over an hour and I should be able to finish the rest within another hour or two. And along the way I of course get sucked into improving other things too so it isn't entirely busywork.

If only we lived in a perfect world where everyone got along and no licenses were required. But then what would the lawyers do?

The Most Interesting Mapper

noreply@blogger.com (Toby Murray) — Sun, 18 Dec 2011 06:09:00 +0000

The Most Interesting Mapper in the World

In case you live under a rock, this particular meme has its origins in a set of beer commercials that I find kind of amusing. Here is a decent compilation of them.

But seriously, the ImproveWayAccuracy plugin is a wonderful tool. Perhaps the prime use case to show it off is a smoothly curving road that was originally either mapped from low resolution imagery or (around here) imported from TIGER data that may have used far too few nodes to make a smooth curve. To fix this by hand is highly annoying. You have to drag nodes around and then grab the + between two existing nodes to create a new one and drag it to where you want it, repeating that as often as needed to achieve a smooth enough curve. Each node requires several precise mouse movements.

It is much easier to draw a way from scratch where you just move your mouse along the curve and click as needed to form a smooth looking curve. Well, the ImproveWayAccuracy plugin makes this possible to do with existing nodes/ways!

For basic usage you can look at the plugin's wiki page: http://wiki.openstreetmap.org/wiki/JOSM/Plugins/ImproveWayAccuracy

The first thing you will want to do after installing the plugin is to change JOSM's "highlight" color to something that has a high contrast to whatever the "selected" color is. By default they are both shades of red which makes the plugin harder to use. I changed mine to a bright green (#66FF33).

Basically you activate the plugin by pressing k and selecting a way. Then if you move the mouse around you will see that the node that is closest to your mouse cursor is highlighted and there are temporary lines going from your cursor to the two nodes next to that one. If you click, the highlighted node will be moved to your cursor's current location. It is kind of hard to describe but pretty easy to get the hang of if you try it.

Holding down CTRL and clicking will add a new node to the way between the two nearest nodes. Holding down SHIFT will lock the node selection to the current node even if you move the mouse so that another node is now closer to the cursor. This can be helpful while trying to fix some weird geometries.

Here is a short clip of me aligning and smoothing out a curve using this plugin. You can tell when I am holding down CTRL because the cursor gets a small + symbol added to it:

Now... I do have one beef with this plugin. The wiki page states that deleting nodes is not a feature and that you should re-use nodes whenever possible. While I generally agree that as much history as possible should be preserved while editing, it is not always practical. Especially with the TIGER imports that I am frequently editing. Sometimes imported ways have too many nodes along them. This makes editing them more of a pain and the correct way to handle it really is to delete nodes.

Of course the source code for this plugin is available in the OSM SVN repository (see here) so I hacked this feature in myself. Just like the CTRL key adds nodes to the way, the ALT key removes them in my version. However it performs a sanity checks first. If you try to delete a node that is a member of two different ways (a junction node) then the plugin will throw up an error message. The ImproveWayAccuracy plugin is only intended to modify the geometry of a way. If you want to change the topology of the data, you should not be in the IWA mode.

One gotacha: Some linux window managers use ALT plus a mouse drag to move windows. So my window manager was swallowing my ALT+click event and it never got passed on to JOSM so the plugin just didn't do anything. But I don't use ALT+drag to move my windows around so I just disabled it. I looked at trying to use a different key but the way the existing code was written, it was easiest to just use another meta key and since CTRL and SHIFT were already taken... I guess I could have maybe used CTRL+SHIFT.

Update: Turns out window managers only listen for the left ALT key but JOSM will take the left or right one. So you can use the meta keys on the right side of the keyboard without problems. Oh and CTRL+SHIFT is already used by the plugin although I have never actually used this feature.

If you want to use my version of the plugin, the easiest way is to first install the "official" plugin through the JOSM plugin options. Then close JOSM and copy my ImproveWayAccuracy.jar file over the one in the JOSM plugin directory (~/.josm/plugins/ on linux or C:\Documents and Settings\<username>\Application Data\JOSM\plugins\ on windows) You can ~~download the jar file here. Or if you want to see the code,~~ ~~here is the svn diff~~ ~~for the IWAMode.java file. (as of revision 27186)~~

I submitted the patch to the original author of the plugin so at some point this might get rolled in to the real plugin. But he said he is very busy right now and probably won't get a chance to look at it any time soon. So I decided to just put it out there in this form for now. The only problem is that letting JOSM update plugins may overwrite the custom .jar file in the future.

Update: I submitted a patch to the JOSM trac system. There was some discussion about keys and other checks to do before deleting nodes. But at the end of the day my changes (plus a tweak or two) have been incorporated into the live code and a new version of the plugin has been released. So just update your plugins and you will be able to take advantage of this feature!

Update 2: Turns out, this code is useful! The JOSM developers saw fit to pull this functionality into JOSM core. They even rearranged some keyboard shortcuts to make room for it on W instead of K. So no more plugin required. Just hit "W" and go! You can see the discussion in ticket #7184 over on the JOSM trac.

Enjoy.

Random computer upgrade notes

noreply@blogger.com (Toby Murray) — Tue, 06 Dec 2011 03:50:00 +0000

I recently did some reshuffling of the computers in my home. I built a brand new gaming system from parts I bought on newegg. Intel i5 Sandybridge, 16 GB of memory and an nvidia video card that not only has a 120mm fan, but has two of them. The insanity. Then I used the parts from the old gaming system (first generation i5, 12 GB, ATI graphics) to give my ancient linux box a major upgrade.

Of Linux and Distros
The Linux box was so old (AMD Athlon x2 4400+ with 3 GB) that it was becoming a pain to use for any kind of desktop applications. I ended up neglecting it and spending most of my time on my windows box (I know! The horror!) with multiple putty sessions open. Also, it was running Gentoo Linux. In my local Linux user group, we have determined that people who are over 30 don't use Gentoo. It is a worthy distro and has many strengths. I have been using it for over 10 years on some system or another and because of its minimal nature, I learned a lot about Linux that I would otherwise be oblivious to. But it can be a real pain, especially to maintain for desktop use. I was getting tired of every other emerge --sync generating package conflicts, lengthy config file merges and the occasional system toolchain breakage that you had to resolve before rebooting (or loosing power!) otherwise the system would completely eat itself. So, being a distinguished 31 years of age now, I decided it was time to drop Gentoo. So long and thanks for all the fish.

So what to replace it with? Well I've been an Ubuntu user for a while as well. But they really worked hard to tick me off with the whole Unity business. It's not that I am completely opposed to new things. But I think Ubuntu messed up pretty good in this transition. First, they rolled out Unity to the netbook edition a year ago, before it was ready. It crashed the first time I logged in. Then it continued to be buggy. Also, I really, really don't like the mac-style combined launch/task bar that is central to Unity. I want a taskbar that displays currently running programs. Not some auto-hiding, moving-with-your-mouse contraption that shows a whole host of programs, some of which may be running and some of which may not. The running ones have a dot on them or something? Whatver. Just give me a taskbar. And now with Ubuntu 11.10 the option to go back to regular old gnome 2 is mostly gone. It's either Unity or Gnome3 with gnome-shell. On my netbook I now use gnome-shell and it works well enough for such a limited device with a small screen. But I'm not really a fan of some of the unilateral decisions they made when it comes to running it on a dual monitor desktop.

Enter this blog post from Linux Mint. The condensed version: "Gnome 2 is dead so we have to move to something newer. But we'll ease the transition by giving you similar functionality to what you've been used to for the last 15 years and let you choose how much new stuff you want." Perfect. So right now I'm typing this on a Linux Mint 12 release candidate install. So far I'm pretty happy. I even find myself using some of the new gnome-shell features from time to time. But I still have my taskbar, traditional alt-tab and always-visible notification area. I'm still trying to hammer out a few bugs I'm encountering. One thing that annoyed me greatly was the default workspace switching for dual monitors. Only my left monitor switched. The right one always displayed the same workspace. Turns out it is just a dumb default setting made by a gnome developer. Unfortunately they don't provide an easy checkbox to change this in the settings. Luckily I found this blog post that fixed it for me in 30 seconds.

Gnome-shell does seem to be a little bit crashy but I'm I'm going to blame that on the ATI video card I have in this thing. I bought this card several years ago right after ATI announced that they would open up their hardware specs to allow people to write decent open source drivers for ATI cards. It seems that hasn't happened as quickly as I had hoped. Notice I'm back to nvidia for my new gaming card. Luckily when gnome-shell crashes it doesn't take down the whole system and I can usually hit ATL+F2 to bring up a run dialog (even though it doesn't appear on the screen) and execute "r" which tells gnome-shell to restart itself. Otherwise, I can also switch to a virtual terminal run a gnome-shell --replace command to make everything happy again.

Update: I think I have tracked down the gnome 3 crashiness. It is in the Mint media player plugin. This bug on launchpad indicates that there is a fix which should be rolled out to the masses shortly. The workaround until then is to just disable the extension.

Of Zabbix and Migrating Old Data
I have been using Zabbix to monitor and track system metrics for a while. Not that I have anything super critical running on my boxes here at home but sometimes it yields some interesting data and I'm kind of a sucker for graphs.

I installed the latest zabbix on my new machine but I didn't really want to migrate my whole database from the old one because there was some cruft in it and several old hosts that haven't existed for a while. But there are a few monitored items that I wanted to migrate. In particular, the data that generates my license acceptance graphs. Zabbix does have an export feature that will export a given item but it only exports the definition, not the data. Fortunately it keeps all its data in a pretty plain mysql database and I'm kind of handy with databases so I managed to transition the data as follows:

Create the item in the new Zabbix install. I did an export from the old system and imported it into the new one to create the item. In this case I also had to copy a shell script that zabbix calls to update the counts and then configure the new zabbix agent to poll it.
Export the data from the old mysql database. First you need the item ID. This can be found by looking at the item you are exporting in the zabbix web web UI and examining the URL. You are looking for an itemid=???? parameter. Then plug that in to the following query which will give you a CSV file. Note that the mysql user you connect as must have the file permission - I just logged in as the mysql root user.
```
select * 
into outfile '/tmp/history_uint_agreed.csv'
fields terminated by ','
lines terminated by '\n'
from history_uint 
where itemid = 19913;
```
Now, find the item ID in the target zabbix install. Then use sed or vim or something to do a string substitution. I used vim with the following command:
```
:%s/19913/22304/
```

Import the data into the new database:

load data infile '/tmp/history_uint_agreed.csv'
into table history_uint
fields terminated by ','
lines terminated by '\n'

Finally, activate the item in the new zabbix so that it starts updating with new data.

Of PostGIS, osmosis and the jxapi
Since I was using PostgreSQL 8.4 on my old system and 9.1 is out now, I ended up just dumping my existing OSM database that I blogged about in the past and setting up a brand new one from scratch. I set up the database right before I left for Thanksgiving. I started the planet file torrent and then drove to Nebraska where my only access to the internet was tethering through my phone which only gets EDGE data service up there. Plus, I was of course being (a little) social with family so I didn't have as much time to tinker with it. For a few tweaks dealing with PostgreSQL 9.1 vs 8.4 I followed Paul Norman's post on the subject. But I didn't want to do the dump files with osmosis for a couple of reasons so I went with the --write-pgsql task instead of the --write-pgsql-dump one so that the data would get read out of the planet file and put straight into the database. Well almost. It still creates some temporary files along the way but you don't have to mess with loading them into the database after it finishes. Unfortunately the planet file has grown too much since I last did this and the nodes no longer fit in my 12 GB of memory so I had to use the TempFile option for node location storage. It actually didn't impact the speed nearly as much as I thought it would. I'm guessing Linux's aggressive caching probably paid off and a good chunk of the temporary node file ended up in the 12 GB of memory the system has. My final osmosis command line looked like this:

bzcat planet-111123.osm.bz2 | 
osmosis --fast-read-xml file=- 
--log-progress interval=30 
--write-pgsql nodeLocationStoreType=TempFile 
              host=localhost database=xapi user=xapi password=xapi

Osmosis ran for 53 hours but when it was done, the database was ready to go.

For fun, I ran a few benchmark queries that Paul came up with a while ago against my old PostgreSQL 8.4 and then again against the 9.1 database on new hardware. Before running each query I restarted PostgreSQL and flushed the disk cache using the command

sync; echo 3 > /proc/sys/vm/drop_caches

to make sure I was getting the raw hardware speed, not assisted by the various caches that the operating system and the database both have. Vital stats of the old and new systems:
Old: AMD Athlon x2 4400+ (2.2 GHz) with 3 GB of memory
New: Intel Core i5 (2.67 GHz) with 12 GB of memory
The I/O system remained pretty much the same: software RAID 5 of four 7,200 RPM 1TB drives with LVM on top of it.

Query	Old system	New system	speedup
map?bbox=11.54,48.14,11.543,48.145	61 sec	46 sec	25%
*[bbox=11.54,48.14,11.543,48.145]	56 sec	42 sec	25%
*[amenity=library][bbox=-0.57,51.24,0.31,51.75]	172 sec	153 sec	11%
node[power=generator]	55 sec	85 sec	-55%
[amenity=parking][capacity:disabled=][bbox=-0.57,51.24,0.31,51.75]	192 sec	177 sec	8%
*[waterway=sluice_gate]	40 sec	9 sec	78%
*[bdad6d21a=1378b02bc]	6.3 sec	3.3 sec	47%
*[power=generator]	405 sec	367 sec	9%
map?bbox=11.54,48.14,11.543,48.145	45 sec	43 sec	4%
map?bbox=-124,49,-123,50	272 sec	227 sec	17%

Several noticeably faster ones but nothing too earth shattering. Then there is that one that is 55% slower. No clue what's up with that. Of course this is all pretty anecdotal. So much changed between the two runs that it is hard to pinpoint whether the speedup came from the faster CPU, more memory, improvements in PostgreSQL or changes in the data. Probably a little from all of the above.

I leave you with a picture of the ridiculous video card fans that I mentioned in the first paragraph. Notice at the upper right corner of the card that it also requires two power connections. I hope it will help me achieve some epic frags at the upcoming semiannual LAN party I have with my college friends. It has already propelled my BOINC score to new heights.

Simple OpenStreetMap tile rendering with Maperitive

noreply@blogger.com (Toby Murray) — Sun, 21 Aug 2011 03:44:00 +0000

Lately there have been a lot of people popping into the OSM IRC channel (#osm on oftc.net) because they have been getting the "you've been bad" tile from the OSM tile server:

The reason is that they are using tile scraping applications that violate the OSM tile usage policy. Typically people just want to download tiles for their city or a vacation destination or travel route in order to pre-load them on a mobile device for offline use.

This is of course a perfect use case for OSM. The problem is that OSM is a project run by volunteers and with limited server capacity. Our primary product is geographic data, not map tiles. So when these tile scraping programs start hitting high zoom levels, it can cause serious problems for the tile server. The solution is to download OSM data instead of rendered tiles. Then you can render your own tiles or even use offline vector rendering.

One tool to make your own tiles is Maperitive. It is pretty easy to use and by default generates tiles that are fairly similar in style to the default mapnik style you see on osm.org although it is very customizable. It is a .NET application although it claims to work in Linux/OSX with mono. Also, while it is free to download, it is not open source which is kind of rare for the OSM world. These two things may cause some people to have an allergic reaction (I'm looking at you, Serge) but if you're still reading, here is a quick lesson on how to use it to render your own tiles. If you keep reading, you should be able to start rendering your own tiles in about 10 minutes. There is a tutorial video and some detailed documentation on the Maperitive website but I don't see a basic text based tutorial that walks you through the common "render some tiles for this area" use case.

Getting Started
Installation is pretty straightforward. Just head over to http://maperitive.net/ and find the "Download" link. Grab the latest .zip file (Maperitive-1228.zip right now) and download it. Once the download finishes, just unzip the Maperitive folder inside of the zip file somewhere and you're done. Open the Maperitive.exe file to fire it up.

Here is a screen shot of my Maperitive with some OSM data already loaded in. It should all make sense by the end of this post.

The first time you fire up Maperitive, you will notice there is already a map on the screen! By default it just starts loading map tiles from osm.org so you can get your bearings and navigate to an area of interest. Next, look at the bottom of the window. On the left is a text area (labeled "Commander" in the tab above it) where status messages and command outputs are displayed. Below that is a command line. This is a graphical program but a lot of the advanced functionality is done via this command line. A note about typing commands: Maperitive automatically completes commands for you which is nice... but it does it in a rather annoying way. As soon as you have typed enough for it to identify a unique command, it immediately fills in the rest of the command. I am much more used to a user-driven tab completion style as seen in most linux shells so the automatic completion in the middle of typing a command means my fingers keep typing the command after Maperitive auto-completes it for me and I have to backspace to erase extra characters.

On the right side is a "Sources" section that lists what map sources are currently being displayed in the main window. You can turn each one on and off by clicking the star by it. In the menus are various options. When you select one it will execute a command that you can see being executed in the Commander pane at the bottom. For example, find your home city and then go to the View menu and select "Set Home Location." You will see the text "> set-home" appear in the Commander. You can do the same thing by typing "set-home" in the command prompt and hitting enter. Now you can use the "Go Home" option to snap back to your "home" location from anywhere.

Selecting an area
Next, you must select an area of the world to render. You can navigate with your mouse just like on osm.org. Holding down shift while dragging with the mouse will draw a box. After you release the mouse button the view will zoom to that box. I'm not really sure what the limits of Maperitive are as far as volume of data. Note that the way it downloads data from the XAPI puts a limit of 10 square degrees on the download area. But this translates to a box of over 100 kilometers on a side and I'm guessing Maperitive will have a hard time with that much data anyway. I tried my county which is about 30x60 kilometers (18x37 miles). It contains about 26 MB of data in .osm format. I was able to render tiles on a 2.8 GHz Core i5 system down to zoom level 18 in about an hour with a memory footprint of under 300 MB. This should be a big enough example to cover many use cases so I will be using this area in my examples.

By default, Maperitive renders everything you see in the map on your screen. So if you are happy with that, you can go to the next step. To customize the area to something more specific than the shape of the window, read on.

You can set a more specific bounding box for all your operations by using the command prompt. One easy way to find a bounding box is to use the XAPI frontend. Browse the map and then hold shift while dragging the mouse to draw a box. After drawing the box, check the "Search by Area" checkbox above the map. Now look down at the bottom of the page and copy the latitude/longitude coordinates after "bbox=" - in my case: -96.964,39.025,-96.547,39.573 (do NOT copy the ] at the end!)

Now go back to Maperitive and use the bounds-set command:

bounds-set -96.964,39.025,-96.547,39.573

Maperitive will reply with "Locking the bounds to ..." and will draw a thick dotted line on the map to show the bounding box you have selected. It is clearly visible in the screenshot up above.

Downloading data
Now that you have your area of operations defined, it is time to download data. This should be as simple as selecting the "Download OSM Data" option in the Map menu. This will hit the java XAPI service hosted by MapQuest Open and download the data within your specified bounds. One problem with Maperitive is that its network timeout is very short. So if you get an error that says "The operation has timed out" then the XAPI server just responded too slowly. You can try again but you may need to reduce the size of your bounding box or try one of the other XAPI servers listed on the XAPI wiki page. To specify the server you add an "xapi-url=" argument to the command. I hit my own XAPI server that I have set up here at home. To do this I issued this command:

download-osm xapi-url=http://localhost:8080/xapi/api/0.6/*

Alternatively you can download the data using another method. (in JOSM or the "Export" tab on osm.org to name two) Then you can go to the File menu and use the "Open Map Sources" option to load the file.

Once the data is loaded, turn off the Web map display by clicking the star over in the Map Sources tab. This will remove the tiles downloaded from osm.org from the display and show only what Maperitive has rendered from the downloaded data. It will be fairly similar but not identical.

Generating the tiles
Almost done! Now all you have to do is generate tiles. First, make sure that the web map layer is disabled or deleted in the Map Sources tab. Otherwise Maperitive uses map tiles from osm.org as a background layer. This causes it to scrape tiles which is what we are trying to avoid in the first place! There is a command in the Tools menu that will generate tiles. By default it uses how far you are zoomed in on the map window and renders 5 zoom levels of tiles based on that. If you want to generate a wider range of zoom levels, go back to the command line. This will render your bounding area from zoom level 8 down to 18:

generate-tiles minzoom=8 maxzoom=18

The tiles will be generated in a directory named "Tiles" inside of the Maperitive directory, wherever you unzipped it to. If you are going all the way to zoom level 17 and 18, it could take a while to render so go outside and collect some map data or something.

When the process finishes, you will have a standard tile directory tree just like you would have had by scraping tiles from osm.org with Mobile Atlass Creator or whatever other scraping software people have come up with. Load it onto a phone for offline viewing or throw it up on a web server using OpenLayers... whatever your heart desires.

Easy updating
Maperitive supports making simple scripts. So if you intend on updating your tiles periodically you might want to make a script so you can just fire and forget. Scripts are run from the File menu. Here is one that performs all the steps listed above for my county but only renders tiles between level 8 and 15. It takes about 90 seconds to execute including download time from the XAPI which can vary depending on current server load and size of area. You can copy this script, change the numbers in the bounds-set command and the zoom levels you want to render and then run it to generate tiles for your desired area. Maperitive expects script files to have a .mscript extension.

clear-map
use-ruleset alias=default
bounds-set -96.964,39.025,-96.547,39.573
download-osm
generate-tiles minzoom=8 maxzoom=15

Maperitive can do a lot of other things like adding hillshading or relief contours (elevation information) to the map.There are also several styles that come with Maperitive besides the mapnik look-alike one. You can poke around the menus to see some of this functionality or execute the help-commands command on the command line to get a list of all available commands. More information about a specific command can be seen by using help-commands again and adding the specific command you want more information about. Example: help-commands generate-tiles will give you more details about the command that renders tiles.

I think I am going to do another post about Maperitive that will go in to more details about customizing rendering rules for a specific purpose. Until then, happy rendering!

Cleanup on the Map Aisle

noreply@blogger.com (Toby Murray) — Sat, 25 Jun 2011 22:04:00 +0000

Spills happen. A sticky honey jar in the grocery store. A glass of orange juice at breakfast. Oil spills in the Gulf of Mexico. Node spills in OpenStreetMap. Some can be cleaned up with a mop and some water. Others require a more technical solution. What is a "node spill" you ask? It's when nodes get uploaded to OSM that don't have any tags on them and aren't part of a way. These tagless, unconnected nodes add no useful information to the map and are just dead weight in the database. Where do they come from? There are at least a couple of common sources of node spills. One is editor bugs and simple user error. These are generally pretty small spills of 10s or maybe 100s of nodes. The bigger problem is imports.

The topic of imports deserves its own post. For now I'll just say that badly performed imports and insufficient checking after imports can lead to 10s of thousands of empty nodes. The basic problem is that nodes get uploaded first and don't become part of a way until the way is uploaded later. So if something goes wrong with the way upload, the nodes end up in an orphaned state.

I have come across a few of these nodes before but what really caught my attention was a failed NHD (National Hydrography Dataset) import in Oklahoma. It happened to poke up into Kansas a little bit where I noticed it next to a state highway I was editing. After a lengthy thread on the talk-us mailing list I eventually found a good way to detect these nodes. Since then I have made my way across the US and some of Canada deleting these useless nodes from the database. As detailed in this message it goes something like this:

Use the XAPI to perform a query of the form /node[not(way)][bbox=a,b,c,d]
Open the result of this query in JOSM and apply a filter to hide all nodes with tags
Perform some checks to make sure there really is no useful data
Delete nodes and upload

XAPI Query
The [not(way)] xapi predicate is documented on the OSM wiki XAPI page. In addition you need to add a bounding box to the query. An easy way to come up with a bounding box is to use the uixapi page found at http://openstreetmap.us/uixapi/xapi.html. Hold down shift and draw a box around the area you wish to query. Then tick the "Search by Area" checkbox above the map. The URL at the bottom of the page will change to include a [bbox=...] portion. Note that the bbox won't update automatically if you draw another box. You need to uncheck and recheck the box for it to update.

If you are using one of the public XAPI services then the size of the bounding box will be limited to 10 square degrees. Since I am running these queries against my own XAPI installation, I removed that limitation so I've just been doing entire states at a time. Depending on how much data is in a given area, the queries take very different times to return. All of Alaska took under 10 minutes while some of the small states on the east coast took about an hour.

I started out downloading the data straight into JOSM using the "Open Location..." feature in the File menu but for particularly dense areas this caused some strange problems so I switched to using wget to download the data to a file and then opened the file in JOSM. I learned something about wget: it has a default timeout of 15 minutes. So if the query takes longer than 15 minutes, wget will give up and try again which will result in two identical queries running simultaneously and slowing things down even more. So remember to use the --timeout=0 option to disable the timeout in wget. Here is a sample wget command for Kansas:

wget --timeout=0 -O KS.osm "http://localhost:8080/xapi/api/0.6/node[not(way)][bbox=-102.12,36.87,-94.46,40.11]"

Loading into JOSM
The previous step gives you a .osm file that contains all nodes that are not part of a way in the area you queried. Obviously this includes all POIs that contain useful data. The file can be rather large and may require that you give JOSM a couple of GB of memory to use. Here is a screen shot of Maine:

Now let's filter out all nodes with useful information. In this case you can see a filter in the bottom right of the screen of the form "-untagged" which hides all nodes with tags except for nodes that only have the following tags: attribution, created_by, source, fixme and note. (I won't guarantee that this is a complete list) While nodes with only these tags aren't really useful map data, you do need to be careful because nodes with a fixme or a note tag could be useful information to other mappers. The other stricter JOSM filter that I used sometimes was "tags:1-999" which filtered out all nodes with any tags on them. After filtering, things are a little more manageable:

Now for a bit of data analysis. Over on the right side of the window you can see I have the "Authors" panel open which lists who the last person was to touch the currently selected objects. The most frequent user in the US was woodpeck_fixbot. After some digging I found out why. The Census Bureau TIGER data was originally imported with several tags such as source and an upload_uuid tag on every single node. After the import, it was determined that these tags on the 10s of millions of nodes in the TIGER data were unnecessary and bloated the database and the planet file so much that it was decided to remove them. This removal was done by woodpeck_fixbot. So all these nodes are originally from the TIGER import and I deleted these without a second thought since they have been sitting in the database for several years.

However for other cases I did more checking. I investigated the changesets that left a lot of empty nodes to determine what was being done in these uploads and whether the user noticed the failure. Sometimes it was obvious that the failure was noticed and the data was re-uploaded so that the empty nodes were actually duplicates of data that was successfully uploaded later. In this case deletion was also obviously warranted. One way to check this is to just zoom in on some of the nodes with the OSM tiles in the background:

Obviously these nodes were intended to be part of the border of this lake which exists. In this case it was doubly confirmed by looking at the changeset comment where it stated that this was an upload of NHD data.

Unfortunately sometimes the uploader did not notice the error so no re-uploading took place and there were no features present. In some of these cases I contacted the user to make sure they knew about the error and see if they had a way of recovering from it. A surprising number of them knew there was a problem during the upload process and thought they had fixed it but they didn't get everything. Just about everyone I contacted said to go ahead and delete the nodes.

The other thing you really have to watch out for is currently running imports. I hit one of those in St Louis. Someone was actively importing NHD data while I was cleaning up empty nodes in the area. If I hadn't noticed this and blindly deleted the nodes it would have caused their upload to fail horribly and led to all kinds of headaches including even more empty nodes. One good way to do this is to select all the nodes you are thinking about deleting and then using the JOSM search feature with the string "timestamp:2011" to find nodes last touched in 2011 within your current selection. These can then be more thoroughly investigated.

Uploading
After sufficient checking had been performed, I deleted the nodes and uploaded them with a helpful changeset comment.

There is one final kink that popped up a couple of times during upload. While all these nodes are not used in a way, a few of them were members of a relation. Most of these cases were clear errors. For example some empty node in the middle of Nebraska was a member of an administrative boundary in France. No clue how that happened. I'm assuming some editor or import bug. There were several in California that were members of turn restriction relations with a "location_hint" role. Not sure what that's supposed to mean but of course I left them alone. In these cases the API will return an error saying that the node is still in use by a relation and you can't delete it. To resolve the conflict you can take the relation ID from that error message and download the relation using JOSM's "Download Object" feature. It will complain that there is a conflict which can then be resolved in the conflict editor. Then the upload can continue.

Here is a shot of LiveMapViewer after a day of work near the Great Lakes. Obviously this includes the work of other mappers as well but most of that red is the result of my deletions.

Conclusion
As stated above, I have covered the entire US with this cleanup. In most places I was pretty conservative in what I deleted in that I only deleted those nodes that I positively identified as part of an import or otherwise determined what had happened and why. Other nodes that I wasn't sure about I left in place. I have yet to receive a single complaint about any of these edits so I think I did ok. Despite the conservative deletion I have nuked just over 411,000 nodes. This translates to a space savings of about 50 MB in the planet.osm file assuming each empty node takes about 125 bytes of space. Guess that's not even a drop in the bucket. But still a worth while cleanup effort. It even got me pretty high up on the monthly list of most active OSM users on the data stats page!

Some of the more extreme cases of bad data I found were a failed upload of parcel data in Arkansas that was supposed to have been cleaned up by the original uploader but was not due to a bug in the editor. It took three changesets to clean that up. Then there was the case of ridiculous over-noding. There were 14,000 nodes that were mostly contained in about a 4 block area near San Bernadino in California. That amount of bad data can actually make it difficult for casual mappers to do anything in the area since some editors may not handle that volume of data very well.

Time to go get some fresh mop water and continue the cleanup.

The Main Attraction

noreply@blogger.com (Toby Murray) — Sun, 15 May 2011 08:29:00 +0000

I have mentioned the TIGER import several times. The initial import tagged roads with abbreviations in their name. For example "Ave" instead of "Avenue" or "St" instead of "Street." However the general rule in OSM is to not abbreviate things. So a script was devised to automatically expand abbreviations. This script was run by the user balrog-kun. Until tonight I assumed that it had been run over all of the TIGER data. But then I saw a few abbreviations while working on some highways in Kansas and decided to dig a little deeper. I did some XAPI queries (using the database I talked about in a previous post!) to see if anything interesting stood out. I did one query for all streets named "Main St" and another for all roads named "Main Street" and here is what I got back.

Green is "Main St" and the rest is "Main Street"

First of all, I am amused at just how many "Main" streets there are in the US. But also it looks like the name expansion bot was only set loose west of the Mississippi. The bot was run long before I joined OSM so I was not around for any discussions about why this was done. I'm sure it is buried somewhere in the mailing list archives...

I haven't done any substantial editing east of the Mississippi which explains why I hadn't noticed abbreviated street names before. I'm still not sure why I found some in western Kansas. The most likely explanation is that a few of the bot uploads failed for some reason, leaving pockets of abbreviations around.

I pinged balrog-kun about it on IRC so the mystery may be solved in a few hours but I thought the image was amusing enough to share anyway. And for a few more giggles, here is the whole world:

It seems the UK has a fair share of Main streets as well, plus a few scattered mostly in former British colonies. I'm sure they started it and then I guess we here in the US just turned it up to 11.

Update After talking to balrog-kun: Apparently he was moving west-to-east and at some point some people started complaining for some reason so he stopped. Fair enough I guess. He's not sure about the ones I found in western Kansas though.

One Mapper, One Year

noreply@blogger.com (Toby Murray) — Sun, 10 Apr 2011 20:36:00 +0000

Today is the one year anniversary of my first OpenStreetMap edit! I thought I would take the opportunity to reflect on how OSM and myself have had an effect on each other. My intention is not to brag about all the mapping I have done but rather to point out some different ways in which contributions can be made and some of the tools I use when mapping.

My very first edit was to add a foot/bike path near my apartment so that it would show up on OpenCycleMap. Unfortunately the imported TIGER roads were so far off that I had to realign them before I could add the path in the correct location. And of course correcting the ones right around the path led me to correct the ones connecting to those and the ones connecting to those and so on... At the time I didn't even know what TIGER was or what all those tiger:* tags were on all the roads.

Some highlights since then:

GPS trackpoints uploaded: 283,271 in 75 traces (most from my Garmin Edge 305)
410 changesets created using Potlatch, JOSM, OsmAnd and Vespucci
Pictures taken: over 2,000 (4.5 GB saved) Most were geotagged to a GPS trace to aid in mapping features (uploaded some of them to openstreetview.org)
Some of the locations I have edited:

My university campus
My city
My state
My country
My high school
My childhood home
Some random place where there was an earthquake

Map objects affected (according to hdyc.neis-one.org - last updated in January)

Nodes: about 165,000
Ways: about 24,500
Relations: about 500 (I created over 200 county relations in 3 states since the last update)

Contacted multiple government agencies in Kansas to get explicit permission to use their data to assist in mapping. Documented these data sources on the Kansas page in the wiki
Started a blog (self referring link!)

Now for a pretty picture. Here are all the GPS traces in my city. I think there may be one or two from other people but most of it is me.

GPS traces in Manhattan, KS

Most of them were collected on my bicycle although there are a few roads in town that I do not bike on because I do not have a death wish so those were done in the car. But I still have work to do! On Saturday I was talking to some people in the Flint Hills Area Bike Club and heard laments of new streets missing from maps. They were of course talking about google maps but if these new streets are added to OSM first, I may have some converts!

And speaking of converts, I have also had the pleasure of working with some other locals to enhance and use OSM data in interesting applications. For example, Michael Wesch is a professor at Kansas State University. He used OSM data to make a bicycle safety map for the city. In trying to get the map into a usable state, he added a lot of the bigger landuse areas in the city that you now see on the map. He also convinced the College of Architecture, Planning and Design to focus its "Design Days" project on bicycle friendliness and to use OSM as their map. Almost 200 students were sent out into the city with walking papers to find any bicycle oriented path or amenity that wasn't on the map and report back to people on campus who then entered the information into the OSM database. I had about 10 minutes to teach 10 people how to use JOSM. Oh the things I would now do differently.

Several new paths were found that connect neighborhoods and open up the option of safely commuting on a bicycle or by foot to school and the university. Many of these paths will never make it to google/yahoo/bing/etc maps because they are not officially maintained or even known about by city officials. They are just paths that people have worn into the ground by repeated use. At the end, each group presented plans for improving one section of the city and most used OSM renderings in their presentations. I will admit that I felt a sense of satisfaction seeing mapnik renderings of the city printed out on three foot wide paper and being displayed in projected presentations.

A giant printed map for people to mark on at the Design Days public presentation

And speaking of the university, my involvement in OSM got me interested in geographic data in general and I decided to take advantage of the university employee tuition discount. I took a cartography course last fall and am now in the middle of the GIS I course. It is interesting to see differences in the way OSM and "professional" GIS applications (mostly ESRI ArcGIS for these courses) handle data and allow you to interact with it. Both have different strengths and weaknesses.

After my dramatic and internationally anticipated* entrance into the blogosphere with my TIGER ways in Kansas post, I was made aware of and encouraged to attend WhereCamp5280 in Denver. Here I met Steve and Hurricane Coast, Richard Weait and Samat Jain among others. I learned several things that day and had some fun while I was at it. Ski mapping in Winter Park the next day didn't hurt the "fun" aspect either!
*for some definition of "dramatic" and "internationally anticipated"

A lot of my mapping has obviously been local to Manhattan, KS. (Note: I have heard rumors that some other "Manhattan" exists up in the northeast of the country. I have been unable to verify this rumor personally and assume it is false based on my own empirical evidence) This is OSM's strength. Get people who know their own neighborhood to add detail to the map and then update it as it changes. As evidenced by the GPS traces above, I have explored much of the city. Doing so on a bicycle has been really fun. I have discovered interesting neighborhoods and parts of the city that I had never seen before. It surprised me how much I did not know about this relatively small city even after having lived here for over 10 years.

Current map of Manhattan, KS

Other edits have touched places that I have visited on trips. After the trip to Denver, I edited all along I-70. Mostly just improving alignment plus adding the occasional gas station, hotel or other business along the interstate that I managed to snap a picture of as I went past. I also ended up fixing a major cloverleaf intersection in Denver that the local OSMers hadn't gotten to yet. Tsk. Tsk.

Some of the more regional editing started with tracing the Kansas River from USGS imagery. Someone else happened to be doing the same thing, starting from the east. I started from the west and we met up in the middle. More recently I have started doing some edits that touch much larger areas. For example, while browsing the MapQuest Open map, I noticed that county name labels were being placed towards the southeast of the county instead of the middle where you would expect them to be. After some investigation I realized that they are rendering the county name from the place=county nodes and that when these were originally imported, they were all off in the extreme southeast of their counties for some reason. So I used an XAPI query to get all the county nodes in Kansas and another one to get all the county borders. Then I moved all the nodes to be approximately in the center of their respective counties. Now the MapQuest site shows county names in the correct location.

While local mappers are key to OSM's value, people with some data analysis skills and above average technical abilities are also needed to tend to some of the "bigger" data that local mappers may not notice or care about. In the US, a lot of this data that needs tending to comes from data imports. Administrative boundaries, TIGER data, etc. Imports is an entirely separate topic that I won't get in to here though.

Another issue that requires some data analysis is tagging. Luckily we have some great tools such as tagstat and taginfo to help explore tag usage. One problem with tags is typos. Recently I got a message from a mapper who discovered through one of these tools that I had mistyped the word "petroleum" when mapping some of the oil wells south of the city. Thanks to JOSM's auto complete feature when entering tags, the typo had been applied to all of the wells in the area. Thanks to this other mapper discovering the outlying values in this tag, the typo was discovered and corrected.

Another problem with tags is inconsistencies. Unfortunately, as with many open source projects, documentation seems to be a low priority. When I first tried to figure out how to tag a doctor's office, I found 3 different pages on the OSM wiki describing how to tag them. One of them was just a stub page. The other two were fairly well fleshed out. But by looking at the tagstat I found that the one that was being used the most by far was the tag corresponding to the stub page. It looks like this has been cleaned up somewhat now. There are still two different pages with a proposal to merge them.

Initially I was really frustrated by the disorganized tagging rules. In my day job, inconsistent data is sometimes worse than missing data. I still haven't entirely gotten over it but at the end of the day, I consider the various sources and tag things however it makes the most sense to me. The bad thing about such an approach is that it makes the OSM data harder for our consumers to use because they have to consider all the tagging variations for a given type of object. The good news is that since I had these problems, a lot of commonly mapped objects have been added to presets that various editors have so most users don't need to know about the intricacies of tag values or the impressive debates that surround them on various mailing lists. Just use the preset and you're done - and the data is consistent.

In summary: It has been a fun year learning how to map, meeting other mappers, getting involved in the mapping community and experimenting with map data. I plan to continue! In fact, I just recently decided to make the relationship official and became a card-carrying member of the OSM-US foundation. Although I'm still waiting on the actual card to arrive. In the meantime...

Map the planet!

Keeping a database up to date with osmosis

noreply@blogger.com (Toby Murray) — Sat, 09 Apr 2011 04:34:00 +0000

This is a follow-up to my previous post about importing a planet file into a pgsnapshot database schema. However the same basic process applies to other schemas that osmosis can write to.

To briefly set the stage: After downloading the weekly planet file and processing/importing it into a database, the data is now probably 3 or 4 days old. OpenStreetMap provides both minutely and hourly change files that can be applied to the database to bring it up to date with what is live right now in the master OSM database. If you think about it, that is actually a pretty sweet deal. Someone on the other side of the planet can spot a new restaurant while on their lunch break, use an application on their phone to upload it to OSM and you will get the change applied to your local database in less than 120 seconds. Does this blow anyone else's mind just a little? Open data is awesome like that.

Initializing the working directory

So what does it take to pull of this feat of mad wizardry? The "Replication Tasks" section of the osmosis usage page is of interest. Again, not the easiest read in the world. Let's break it down. First, you need to set up a working directory for osmosis to keep some information about the current state of the data. To do this, create a directory. I chose to put this in a hidden directory in my home dir: ~/.osmosis-minutely/ Next, enter this directory and execute this command (again, I am assuming you have osmosis installed in your home directory):

~/osmosis/bin/osmosis --rrii

This will create two files: download.lock and configuration.txt. You can ignore download.lock. It is used by osmosis to ensure that only one copy is running at a time. The interesting bits are in configuration.txt. First of all, this is where you have to decide if you want to do hourly or minutely updates. Minutely allows you to stay in sync with live changes to the map as people upload them as described above. However this means that your computer will always be spending anywhere from maybe 10 seconds to a full minute every minute importing OSM data. This won't make it unusable for other things but will certainly have an impact on the system. In theory using the hourly files might be slightly more efficient. I haven't played with this to determine if this is true or not but Paul Norman indicated that it didn't seem to be for him. Once you have decided which to use, put either http://planet.openstreetmap.org/minute-replicate or http://planet.openstreetmap.org/hour-replicate in the baseUrl line of the configuration.txt file.

Now you need to determine the time stamp of your planet file. It is in the 2nd line of the planet file. You can get it like this:

bunzip2 -c planet-110309.osm.bz2 | head

In my case it reads timestamp="2011-03-09T01:11:06Z"

Now you can do it the easy way or the hard way. The easy way is to use the replicate-sequences tool. Note: this tool only supports minutely updates. As mentioned in the osmosis documentation, I would suggest taking the time stamp from above and subtracting an hour and using that time just to make sure you don't miss an update. After you enter the time and click the button, take the text it returns and copy/paste it into a file named state.txt in the working directory (~/.osmosis-minutely/)

If you want to know what is going on behind the scenes, read on here. Otherwise feel free to skip to the next section. Every minute (and every hour) files are created on http://planet.openstreetmap.org that contain the changes that have been uploaded to the database since the last minute/hourly file was generated. These files contain only the new attributes of objects that have been changed. So applying a change twice is fine. It just updates the data to the same values it had before. This is why it is safe to go back an hour from your time stamp. However if you miss an update, all the objects affected by that update will be out of sync with the master database until (if ever) they are updated again.

Along with the change file itself, there is also file containing metadata about the change. For example, this one. It has a sequence number that increments for each change file plus a time stamp of when exactly the last change in the associated change file was applied. If you want to find the correct state file without using the tool listed above (or if you want to use hourly updates) you will need to browse through the directories to find an appropriate state file to start off from, again going back in time at least a few minutes or an hour. The time stamps listed next to the files are one hour off from the time stamp inside of the file (the time stamp in the file is in UTC and the server is one time zone away from UTC) so you can use it as a rough guide for locating the correct file but be sure to check the time stamp inside of the file before using it! As above, paste the contents of the appropriate state file into your ~/.osmosis-minutely/ directory as state.txt.

Catching up

You should have everything in place now! Time to fire up osmosis! Paul reached this point long before I did (because I was playing around with things) so I swiped this command from him:

~/osmosis/bin/osmosis --rri workingDirectory=~/.osmosis-minutely/ 
                      --sc 
                      --wpc user="xapi" database="xapi" password="xapi"

Let's look at the command:

--rri or "read replication interval:" Not much to add to the wiki description. This makes osmosis download a set of change files and tells it where to look for the state.txt file to know which files to download.
--sc or "sort change:" I'm not 100% sure about the function here. I'm assuming it puts the changes in chronological order so that you don't end up applying a newer change before an older change which would lead to an outdated object in your database.
--wpc or "write pgsql change:" writes the change to the database using the given credentials. (modify to match your database/username/password) This task can be replaced by other "write change" tasks like --wdc or --wxc to write changes to other destinations.

By default osmosis will download one hours worth of changes (even if you are using minutely updates - it will just download 60 at a time) and then sit there applying them. This will take anywhere from a few minutes to a full hour. It just depends on how many changes are in the files it is processing and how complex the changes are. Applying changes to huge multipolygons is known to be slow and weekends are always a lot busier than week days.

If you want quicker feedback to make sure it is working, open the ~/.osmosis-minutely/configuration.txt file and change the maxInterval value to something lower. This controls the maximum number of changes that osmosis will download in one go. As mentioned above, the default is one hour (3600 seconds) so you can set it to 60 seconds to just download one minute at a time which should only take a few seconds to apply. Once the process finishes you can check the state.txt file. The time stamp and sequence number should have changed to reflect the update that was applied.

Due to my tinkering around and multiple runs I did with osmosis, I had to apply 488 hours worth of updates to my database. I did it by hand in 10 and 48 hour chunks by changing maxInterval to 36000 and 172800, respectively. After it was all said and done, it took about 163 hours to apply 488 hours of updates. That averages out to about 20 minutes of processing time per hour of updates. But I saw variation, especially in the 10 hour runs. Anywhere from 1.8 to 4.5 hours to apply 10 hours of updates. As before, disk I/O tends to be the bottleneck and I'm using a software RAID5 composed of three 1TB 7,200RPM drives so not exactly tuned for speed.

Now all you need to do is put the osmosis command into a minutely or hourly cron job and you're done! And really you can do this before the database is all caught up. I just did the catching up by hand to monitor and record progress. As I mentioned above, it is safe to call osmosis every minute. Osmosis puts a lock on the download.lock file and if the previous run is still executing, the next one will exit immediately without doing anything. I chose to put the command in a script and then put a call to the script into cron. You might want to redirect stdout and stderr to either a log file or /dev/null to avoid cron sending out emails every time the script runs. My cron job looks like this, minus the newline put in for readability here:

* * * * * /home/toby/bin/osmosis-update.bash >> 
          /home/toby/.osmosis-minutely/osmosis.log 2>&1

And that's all, folks! You now have an automatically updating OSM database to query as you please. Have fun!

Parse the Planet! (into a pgsnapshot database)

noreply@blogger.com (Toby Murray) — Sun, 03 Apr 2011 22:35:00 +0000

What makes OpenStreetMap... well... Open? In a link: http://planet.openstreetmap.org/

Yes, of course there are the legal issues of having an open license, the transparent community processes to govern the project and a whole host of open source software to create, manipulate, consume and display map data. But at the end of the day, none of that would matter if the raw data wasn't available.

As is obvious from my previous posts, I have dabbled in manipulating this raw data in the past, with varying degrees of success. After learning from my previous attempts and hanging around on the mailing lists and IRC channel, picking up information here and there I decided to take another crack at processing a planet file. I ended up finding a partner in crime: Paul Norman. He was wanting to import a planet file for use in a private XAPI server using the new java based XAPI implementation written by Ian Dees. This sounded like a good idea so I decided to follow along. We bounced commands and results off of each other for a while. I ended up going a slightly different route in some of the processing because I had some different hardware available to me.

The first thing to know about processing planet data is that there are several different database schemas that can be used to hold the data. This topic probably deserves its own post at some point. Here is a brief list of the ones I am at least minimally familiar with. I know there are some others (like the gazetteer database used by nominatim):

API database: Creates a duplicate of the master OSM database. Does not use any GIS features. Nodes are stored with lat/lon values as simple numbers. Contains tables for: ways, tags, users, changesets. Can track history of objects once you start applying updates.
"pgsnapshot" schema: creates a schema that utilizes the GIS features of postgis that allows you to run spatial queries against OSM data. Still maintains all tag information but they are in a single "hstore" column, not in their own relational tables. Changesets are not stored nor is any information about users except the ID and username. History is not maintained as updates are applied. There is also a related "pgsimple" schema that seems to be an older version of the pgsnapshot schema that may have some advantages. Differences are discussed in the "pgsnapshot_and_pgsimple.txt" file in the osmosis script directory.
osm2pgsql rendering schema: This schema is created by the osm2pgsql tool and used by mapnik to render map tiles. It uses GIS features as well but it is lossy and very de-normalized. Only objects and tags that you care about rendering are imported. This means it is also smaller and faster to get set up.

The XAPI uses the pgsnapshot schema so this is what I set up. The instructions on how to get going are provided in the xapi readme file but I will go ahead and copy it here with some additional notes.

Setting up the database

First of all, you will need a postgresql server with some spare drive space. If you are processing the entire planet file, you will end up creating a database that is about 430 GB in size. However during processing, you may end up needing anywhere between 200 and 400 GB of additional space for temporary files. Some of this depends on exactly how you execute the process. And some of the storage doesn't need to be fast. I used an external USB drive for some of the temporary files and it worked great.

Next, you might want to take a quick look at the "Configure the PostGIS database" section in this post by Richard Weait. I am pretty new to postgresql so I have no tuning recommendations of my own. I used the values he suggested and they seem to work.

As the postgres user on your database system execute the following commands from a shell:

createdb xapi
createlang plpgsql xapi
createuser xapi

Note: this may not be the best way to do things, but answer "yes" to the superuser question. Otherwise you have to modify postgres ownership which wasn't quite straightforward. Mostly because I'm new to postgres.

Now to set up the GIS part of postgis and the hstore for tags. This step sets up the database to hold spatial data instead of just being a bland relational database. Note that your exact path may vary depending on what Linux distro you are using. You can always try a quick locate command. On Ubuntu you have to install the postgresql-8.4-hstore-new package first. On Gentoo I actually had to compile the hstore-new plugin myself.

psql -d xapi -f /usr/share/postgresql/8.4/contrib/postgis.sql
psql -d xapi -f /usr/share/postgresql/8.4/contrib/spatial_ref_sys.sql
psql -d xapi -f /usr/share/postgresql/8.4/contrib/hstore-new.sql

Now we set up the actual OSM tables. You will need to download osmosis. The pgsnapshot schema was just recently introduced. I had to use a development snapshot but as of April 2, it is included in the release of osmosis 0.39. In this post I will assume that you have osmosis unpacked in your home directory.

psql -d xapi -f ~/osmosis/script/pgsnapshot_schema_0.6.sql
psql -d xapi -f ~/osmosis/script/pgsnapshot_schema_0.6_linestring.sql

Getting the data

Now that the database is ready, let's get some data to work on. For the boring people, just go to http://planet.openstreetmap.org/ and download the file planet-latest.osm.bz2. It will likely take several hours to get the 15 GB file. For the cool kids, check out http://osm-torrent.torres.voyager.hr/ where you can find a torrent file with the latest planet dump in it. Planet files are typically produced every Wednesday. The torrent is updated a day or two later. I downloaded several planet files via the torrent and each time I truly did get a torrent of data. During most of the transfer, my downstream bandwidth was maxed out at 2 megabytes/sec.

If this is your first time working with OSM data I would strongly recommend testing the waters with a country or state extract. Extract providers are listed on the Planet page on the wiki. I would look at geofabrik and cloudmade. Geofabrik has the data available in a format called PBF which is more compact and quicker to work with than the bzip2 compressed XML file you get from planet.osm.

Processing with Osmosis

You might want to go refill your drink. This is going to be long.

Osmosis is an amazing tool. Also, amazingly confusing. Just take a look at the "Detailed Usage" page on the wiki. The page summary is two full screens tall in my browser. And once you start to actually read things, it just gets worse. It talks about tasks, pipes and streams as casually as most people would discuss the weather. These are all useful things if you're an osmosis expert wanting to look up how to do a specific task like splitting a planet file into 16 polygon defined extracts, all at the same time - while filtering out relations. But it is a little daunting to new users.

One thing to keep in mind: argument order matters. For example, If you want to use the --log-progress option to output progress information, it must go between the read and write tasks - you can't simply add it to the end of the command line. There are also multiple ways to do some things. This can be a good thing as it lets you optimize the process based on what kind of hardware you have available to you.

Osmosis is a java program. As such, you may need to specify certain options to the JVM to make it do what you want. These JVM level options are controlled by a file. On linux this file is named .osmosis and needs to be in your home directory. On windows 7 it needs to be at C:\Users\<user name>\osmosis.bat. The most common option you may need to use is to give java more memory using the -Xmx option. The other one I needed for one of my attempts was -Djava.io.tmpdir=/some/other/path/ to set the location of the temporary directory to something other than /tmp/

So my ~/.osmosis file in linux looks like this to give it 2 GB of memory and redirect the temp directory

export JAVACMD_OPTIONS="-Xmx2G -Djava.io.tmpdir=/mnt/database/tmp/"

The format is identical in windows except you replace "export" with "set"

One more important note: Osmosis claims to be able to read bzip2 compressed files natively and will happily set about doing this for a few minutes. But do not attempt this! It will end in tears and very odd "invalid XML" errors being thrown. The reason is that java's native bzip2 implementation can not read files with multiple streams in them. So it reads the first stream and thinks it is done, truncating any XML elements it might be in the middle of at the time. Instead, use an external utility to unzip the file to standard out and then tell osmosis to read the uncompressed data from standard in. There is a note about this at the bottom of the osmosis page on the wiki but it is easy to miss (I did at first) so it bears repeating.

Now that we have some basics down (I told you this was going to be long) let's look at what it takes to actually process the data. The first thing we need to do is read the XML using the --read-xml task.

Next, there is a decision to make. There are two basic ways to get the data into the database. One is to let osmosis do it all automatically using the --write-pgsql task. This will cause osmosis to read the XML into some temporary files in your tmp directory and then shove it into the database. The first step may take up to several days (depending on if you are using memory node storage or not - keep reading) and create close to 200 GB of temporary files before it even touches the database. If something fails during the database load, the temporary files are deleted and you start over from scratch. Why the temporary files? Osmosis has to do some processing of the data to correctly construct ways out of nodes and relations out of ways and nodes before it can put them in the database.

The other option is to use the --write-pgsql-dump task to process the XML into a series of files that can be loaded into postgres using the \copy command which is much faster than issuing INSERT queries. It also gives you a fall back point if something goes wrong during the database import. The primary reason I chose to use it is that it lets you easily run this process on another computer and then take the resulting files and import them into postgres on a different computer. This is also the only reason I have been mentioning Windows. My file server that has the disk space and is running postgres is an older machine with only 3 GB of memory and a slower CPU. My windows desktop is a much newer machine since I use it for gaming. So it has a quad core CPU and 12 GB of memory. I found a good use for the memory.

I will focus on the --write-pgsql-dump task that I used but some of the same things apply to the --write-pgsql task. I will start out with the full command line I used and then explain the parts. As I mentioned above, I ran this on my windows box because it has the memory. So please excuse the ugly file paths :)

c:\bzcat.exe o:\planet-110309.osm.bz2 | bin\osmosis.bat 
  --read-xml file=- --log-progress interval=30 
  --write-pgsql-dump directory=g:\osmosis 
    enableLinestringBuilder=yes 
    nodeLocationStoreType=InMemory

Now for the parts:

c:\bzcat.exe o:\planet-110309.osm.bz2 | bin\osmosis.bat: As I mentioned, it is a bad idea to let osmosis unzip the file so we must do this using an external utility and pipe the result into osmosis
--read-xml file=-: This tells osmosis to read an XML document from standard in. It is easy to miss, but don't forget the - at the end!
--log-progress interval=30: This will cause osmosis to report its progress every 30 seconds. The output simply tells you which node/way/relation it is working on and how many per second it is processing so you can make sure it is actually doing something and possibly estimate completion time.
--write-pgsql-dump directory=g:\osmosis: Tells osmosis to format its output as a series of postgres dump files in the given directory. The following three sub-options affect the behavior of this task.
enableLinestringBuilder=yes: This builds a geometric representation of the way which is then stored in a "geometry" type column in postgres that can be used for spatial queries. Building the linestring can be done in the database but you may not live to see it finish if you are processing the entire planet. Seriously.
nodeLocationStoreType=InMemory: This tells osmosis to hold geometric information about every node in memory. By default it writes this to a temporary file on disk. If you have enough memory you really, really, really want to do this as it cuts processing time from about a week down to, in my case, about 8 hours.

I mentioned "enough" memory. I did this on a computer with 12 GB of memory. The first time I tried this command I gave java 12 GB (-Xmx12G) and java threw an out of memory exception even though my system monitoring (I have zabbix set up) showed that there was still 1GB free. So the second time I gave java 16GB of memory. Yes, that is more memory than the system has and I would not recommend doing this on a regular basis... but it worked in this case. In fact, zabbix still showed that there was almost 1 GB free except for a brief dip to 200 MB. I'm guessing there is some interplay between the JVM and the operating system going on that complicated things a bit. Also, keep in mind that my 12 GB was able to process the planet file today with about 1 billion, 100 million nodes in it but the map is constantly growing (at about a million nodes per day!) so hardware requirements will go up over time. If you don't have enough memory, you can still use osmosis but you will need to use a different node location store type and this process will take several days - maybe up to a week - instead of 8 hours. Anyway, here is a picture of memory usage as this process ran:

The initial downhill slide is where osmosis reads all the nodes into memory. After that, memory usage is relatively constant as it processes ways and relations using the nodes it has in memory. The first upwards bump in the graph is when I closed Starcraft 2 and went to bed :)

One tip if you are doing this across the network like I was: have your output directory be a local disk or even an external USB drive. My windows system didn't have enough disk space so at first I tried putting the output on a network mounted drive that pointed to my file server which is running a samba share. Bandwidth was not a problem but network latency killed performance. Switching to an external USB drive almost tripled the speed at which osmosis processed objects. The planet file can reside on the network. But the output directory needs to be something local.

Here are the resulting files after this 8 hour long memory sucking excursion:

nodes.txt            105 GB
relation_members.txt 231 MB
relations.txt        165 MB
users.txt              2 MB
way_nodes.txt         26 GB
ways.txt              79 GB
---------------------------
total:               210 GB

Loading into postgres

Now for the long part. These 6 files need to be loaded into a database. Luckily there is another script in the osmosis distribution that will assist you. However it will need some modifications. The file is ~/osmosis/script/pgsnapshot_load_0.6.sql. If you open this file you will see that it first drops all constraints and indexes to make the mass inserts go faster. Then it actually imports the data, rebuilds indexes and constraints and finishes up with some database maintenance. The first thing I did was enable query timing just so I could see how long things took. To do this, insert the following line. All of the index/constraint drops will be instantaneous so you can put it after those:

\timing

Next, comment out the next two lines that start with SELECT DropGeometryColumn. The bbox one isn't needed for the xapi and, assuming you followed my advice above about building the linestring in osmosis, that should already be there. You will also need to comment out the two SELECT AddGeometryColumn statements and the two bigger UPDATE queries below that. These sections are all preceded by comments indicating that they may need to be commented out. To comment them out, just add two dashes at the beginning of the line.

I am also unconvinced as to the necessity of the CLUSTER queries. I let the one for the nodes table run and it took almost 3 days. Then I lost power during a spring thunderstorm while the one on the ways table was running. I didn't go back and re run it and my database still seems to be working fine. From what I learned in the postgres documentation on the subject, in theory it makes things faster by copying the entire table in an ordered fashion so that nodes that are geographically near each other are also close to each other in the database. This can increase the chances that a single disk read operation will load many of the nodes that you need into memory at once. In practice my use of the database via the XAPI tends to touch a wide area at once so I doubt I'm seeing much benefit from this. You will also see that the postgres documentation states that this needs to be run on a regular basis to maintain the theoretical benefit. As objects are added/deleted/changed, the clustering is slowly lost. And the CLUSTER query puts an exclusive lock on the table so your database will effectively be offline for at least a day any time you do this. So feel free to comment these out as well if you want to save some time. You can always run them by hand later if you determine that they are actually needed.

I also did not run the VACUUM command. VACUUM is a database maintenance task that compacts your tables to remove "holes" left by deleted and changed records. In my case, the CLUSTER query completely copied my nodes table, so I don't think that a VACUUM immediately following this would have done anything. Even without the CLUSTER, I'm not sure it would have done much to a brand new database. I may be wrong about that but I have yet to see any adverse effects from not running it. Again, you can always run it by hand later - and may need to do so periodically anyway.

However it is very important to run the ANALYZE query! This will cause postgres to do a quick (10 minutes or so) analysis of all the tables in the database and will store some basic information that postgres can use later to optimize your joins. I initially didn't do this either and some simple queries I tried looked like they were going to take days to finish. After running the ANALYZE query, they finished in seconds. It is THAT important!

But back to the import process itself. Now that you have tweaked the pgsnapshot_load_0.6.sql file, it is time to wield it. Change to the directory where the .txt files from osmosis are. Then execute:

psql -U xapi -d xapi -f ~/osmosis/script/pgsnapshot_load_0.6.sql

At this point you should go to sleep or spend time with your family or something. Here are the timings from my load:

users.txt:            20 seconds
nodes.txt:            13 hours
ways.txt:             20 hours
way_nodes.txt:         2 hours
relations.txt:        40 seconds
relation_members.txt: 80 seconds

Creating indexes:
node primary key:              1.5 hours
way primary key:               30 minutes
way_nodes primary key:         1.5 hours
relations primary key:         40 seconds
relation_members primary key:  50 seconds
nodes geometry index:          25 hours
way_nodes node ID index:       2 hours
realtion_members member index: 45 seconds
ways bounding box index:       1.7 hours
way linestring index:          1.6 hours

This ads up to about 70 hours. And that's without the CLUSTER query. If something goes wrong, you can truncate the tables (only the 6 affected by the load... leave the other GIS tables alone!) and then just rerun the script. It will spit out errors when it tries to drop the indexes at the beginning because they don't exist but that is fine.

You can monitor the progress of the \copy commands in linux by using

lsof -o -p <pid>

where <pid> is the process ID of the pgsql process. It will give you a readout the position of the current read within the file in bytes - in hex. Convert to decimal and divide by 1073741824 to get it in gigabytes.

If you are tight on space, it is safe to delete or compress the input files after pgsql has finished the \copy command on that file.

Here is a pretty picture of my disk usage during the import process:

As you can see, this covers a 6 day period. 3 days (from some time on the 16th to pretty much the end) is the CLUSTER query that I'm not sure is needed. The occasional drops in space usage towards the end is me deleting old files from previous osmosis test runs to make sure it didn't fill up the partition. The big drop at the end is where it finally finished the CLUSTER query and was able to drop the temporary nodes table. Total partition size is 670 GB.

Conclusion

And that's it! Once that process finishes, you will have a complete pgsnapshot database containing the OpenStreetMap database! As I mentioned up towards the top, one use for this is to run your own XAPI server to do specific map queries however you could also use this database to do other things with the OSM data. Depending on what you are doing, you may need to set osmosis up to keep your database in sync with new edits. Obviously by the time this process finishes, the data will already be several days out of date. I will cover the syncing process in a separate post soon.

The total processing time for me was about 80 hours. To make that at least somewhat meaningful I should list the specs of the computer doing all the importing. I already mentioned that the computer I ran osmosis on was a quad core with 12 GB of memory. The computer that is running the database and that actually ran the import process is a little older. It has an AMD 4400+ (first generation dual core) with 3 GB of memory. But the most important part for the database processes is disk. Mine aren't really spectacular: 3 1TB Samsung Spinpoint F1 drives (7,200 RPM) in a software RAID-5 (so total of 2TB usable storage). I am also running LVM on top of the RAID which splits the space into two partitions. Pretty much all of the database related processes are slowed down by disk access times. Faster disks or a better RAID setup (0 or 10) would drop these times. Or SSDs. Donations welcome :)

As I said above, the total size of the database after it is all said and done is 430 GB. However if you execute the CLUSTER query on the nodes table, you will need an additional 180 GB or so because it makes a complete copy of the nodes table. Plus remember the temporary files created by osmosis need another 210 GB although they can be deleted/moved/compressed as the load script finishes using them.

~~Check back soon (or subscribe!) to catch~~ Check out my next post about keeping the database up to date.

Dual Carriageways

noreply@blogger.com (Toby Murray) — Wed, 02 Mar 2011 06:11:00 +0000

When I first started editing OSM, I was mostly doing residential roads and bike trails. Eventually I worked my way to major roads. Some of them had dual carriageways. In my case these were also US/State highways. Somehow I got the two mixed up in my head and thought that mapping any dual carriageway road required relations. Since I didn't know much about relations at the time I was kind of scared to touch them. Turns out relations ARE used to describe highways however they are not required for dual carriageway roads.

Since I've recently been working on "dualizing" roads in the Kansas City area, I thought I would throw up a quick post about the process. It has probably already been done (I seem to recall a "TIGER fixup" tutorial that described the process) but here is my version.

To Start off with, let's define a dual carriageway road. There may be some difference of opinion (there usually is in OSM) but in general it is any road where the opposing streams of traffic are separated by a physical barrier. A grass median (think interstates) or a strip of trees in the middle of a boulevard are prime examples. Sometimes the barrier is just a small strip of raised pavement or bricks. Regardless, it is something that would make your car go "bump" or even cause damage if you were to try and drive through/over it.

The Census TIGER data that was imported for all of the US just has a single centerline for any given road. The Census Bureau doesn't care about routing topology. They just want basic road centerlines that they can use to carve the country up into their tracts for statistical purposes. There is a tiger:separated tag that I presume is supposed to indicate dual carriageway roads but I have only seen this set to yes a handful of times so as far as I am concerned, that tag can be ignored.

So why bother splitting things up? Well for interstates it is pretty obvious. The ways are far enough apart from each other that you really need to be able to distinguish between the two ways for navigation purposes. They may also be part of different routes (where highways overlap interstates) or have other differentiating characteristics. But even on smaller roads, it can be important detail. For example, see this intersection in Overland Park (part of the Kansas City metro area):

This is obviously going to cause problems for routing and is likely to set off a flurry of annoying "recalculating" messages in navigation devices if you are trying to go east from that residential road.

Medians in the middle of roads are also a visual navigation aid for drivers so having the map show where the road is split is a good thing.

Here is another short section of road that I will convert to a dual carriageway. In this case it won't really affect routing very heavily but if we are going to have the best map in the world then we need details!

The first task is to split the way at the beginning and end of the dual carriageway section and mark the section as a one-way street using the oneway=yes tag. Now adjust the geometry to fit into one lane or the other.

You will see that JOSM (as of the style update last week!) puts black arrows along the way to indicate its one-way-ness. Potlatch does something similar. You may need to reverse the direction of the way to make sure the arrows are pointing in the right direction.

The next step is to draw in the other side of the road. Then copy the highway=* classification as well as the name and the oneway=yes tag (and possibly other tags, as appropriate) from the old way to the new one. To reduce typo errors, I usually just copy the old way and then paste the tags onto the new one using CTRL+SHIFT+V in JOSM. Then I remove any tags I don't want to duplicate, like all the tiger: tags since my new way is not TIGER data.

You will notice that they look slightly different. The old way still is still tagged with tiger:reviewed=no but the new one doesn't have a tiger:reviewed tag. JOSM renders an extra casing around unreviewed TIGER roads to highlight them. I usually only remove this tag if I have done a thorough survey on the ground. In this case I am just using aerial imagery so I will leave it alone. I also usually try to tag the new way I created with a source tag (source=bing in this case) to indicate where I got this information from. That way if someone has a source that they know is better or more up to date than bing, they can update my work in the future.

Finally, I need to fix the connecting road so that the topology is still correct. In this case it actually involves dualizing that road as well.

In this case I brought the intersecting road to a point on the north way. Depending on the situation I sometimes do this or leave them straight and have two intersection points instead of one. Here I felt like using a single node more closely matched the path a car would actually take to turn from one street onto the other. There is always a balance to be struck between topology, geometry and level of detail.

And that's it. Nothing too complicated really. TL;DNR:

Split way, tag with oneway=yes
Draw new way
Copy name, highway and oneway tags to new way
Fix intersecting roads
PROFIT! (optional)

On the miles long roads I've been working on in Kansas City I often make several passes back and forth, mapping additional things like traffic lights (highway=traffic_signals) which can be identified from pavement markings and shadows on the ground (obviously only if imagery is good enough).

Some intersections take a bit of work. Here is a pretty complex one involving US 69, College Boulevard, on/off ramps and a couple of bridges for good measure.

Here is a simple one but it involves a smaller road that splits right before it intersects the big road:

And as the last image for this post, I give you the rendered version of the area I used as an editing example:

That's all!

Airports, Part 2

noreply@blogger.com (Toby Murray) — Tue, 01 Feb 2011 03:43:00 +0000

In my first installment of the airport series I presented some basic mapping techniques I used while mapping rural airports in Kansas. One comment I got on IRC suggested using a slightly different tagging scheme for displaced thresholds. The suggestion (I believe it was from pnorman) was to tag it with two tags. One being aeroway=runway and the other being runway=displaced_threshold. While I haven't gone back and changed my previous work, this may be a better way of doing it. First of all, they would get rendered on the default map. I know, I know... "don't tag for the renderer!" But here's the thing: This is technically not incorrect. Displaced thresholds are considered part of the runway and are included when the runway length is specified. So tagging it as a runway but then specifying that it is a specific type of runway is not incorrect tagging. On standard maps it would be rendered as part of the runway but someone making an aeronautical themed map could use the runway=displaced_threshold tag to differentiate it and produce a customized rendering.

Before I go on, I am just going to repost a link to the OSM wiki page for Airports as a reference.

Now on to some new material. I was moving west-to-east through Kansas so the first "big deal" airport I came to was the Salina Municipal Airport. Its 17/35 runway is a massive 12,300 feet (3.8 kilometers) long. Any aircraft currently in service could land here with room to spare. Why does a municipal airport in a city of under 50,000 people in the middle of Kansas have such a massive runway? The military, of course! This facility was built at the beginning of World War II and was used to train heavy bomber crews. Then during the cold war it became part of Strategic Air Command operations until it was closed and turned over to the city of Salina in 1965. (source: wikipedia) Now it serves mostly general aviation but is also used by Kansas State University's aviation program which is based at the Salina campus. A few years ago it was used by Steve Fossett as the start and end point of his solo, nonstop flight around the globe. And I got to bill them for the telecommunications services they used while doing it! (My current day job deals a lot with the telecom billing system at KSU)

I promise, I'll get to the mapping in this post, eventually. But I should point out that this is one thing I love about contributing to OSM. Seeing something unusual in aerial imagery can often lead to an hour spent on wikipedia or a fun discussion on IRC involving people from around the world or, occasionally, a field trip to go see what the heck is happening "on the ground" to cause it to show up a certain way in aerial imagery.

Back to Salina: I had two sets of aerial imagery that conflicted slightly about the length of one of the runways. (USGS and Bing) Obviously the images were taken at different times. But which was newer? Bing imagery has metadata that is sent with each tile that can sometimes tell you when it was taken. In JOSM you can display this metadata by right clicking without moving the mouse. A menu pops up with a "Show Tile Info" option. Unfortunately in most of the areas I have worked in, it usually says something like it did in this case: "capture-date: 2/1/1995-3/15/2007." So that's useless. While reading the wikipedia page about the Salina airport, I noticed a link down in the "External links" section pointing to an FAA "airport diagram" in a PDF. I clicked on it... and just like that I had found a new source of data to use while mapping! Since this is federal government data that is intended for public distribution, I believe it is public domain so license isn't an issue. And yes, this is only for airports in the US. My apologies to the rest of the world.

These diagrams are constantly updated by the FAA. They have the basic layout of the airport. Turns out the Salina airport shortened the 12/30 runway for some reason so that answered my aerial imagery question. But the diagram also has information about the taxiways. Unused ones are X'd out and the ones that are still in use have their names specified. That can go into the ref=* tag of the taxiways just like runway numbers. One thing to note however is that these are "diagrams" not "maps." So they can not be relied on for accurate cartography. Buildings are usually just squares and do not represent the true proportions or shape of the building. So use with caution - just like any other external data source.

Here is the Salina airport with its enormous runway:

Other information that can be taken from the FAA PDF files include location of the control tower and runway width which can go in a width=* tag. Looking at the contents of existing width tags, it doesn't look like there is currently any support for specifying units so I guess everything should probably be in meters. (Everything should be in meters anyway. Everywhere. But that's another debate for another blog)

It seems there are two ways to map the control tower. One being aeroway=control_tower and the other (documented on the wiki) being man_made=tower/service=aircraft_control. Currently they seem to have about the same usage (both minuscule - about 50 instances) but I think I personally prefer aeroway=control_tower. That keeps airport features together in the same tag. Also, man_made=tower seems to be more for things like cell phone or radio towers. An aircraft control tower is typically a building that happens to be tall and sometimes skinny.

As I said above, I found this particular FAA diagram via Wikipedia. When I went back to try and find diagrams for other airports I ran into all kinds of old information, broken links and just plain bad design on faa.gov. So to save you the pain, here is a link directly to where you can start a search for airports in your state: http://avn.faa.gov/index.asp?xml=aeronav/applications/d_tpp First you have to select the date range, then click on a state. Then you can search by airport code, city name or airport name. Not every airport has a diagram but certainly all major ones do. Go forth and tag!

Another interesting document from the FAA is Airport Marking Aids and Signs. This is just a document showing exactly what all the little lines/blocks/arrows/signs/etc at an airport mean.

The next major airport I hit was Mid-Continent Airport in Wichita. It already had some pretty good coverage but I added ref=* tags for all the runways and taxiways using the FAA diagrams. The Wichita airport has a few more taxiways than usual because it has both Learjet and Cessna facilities adjacent to the airport and they are all connected by taxiways. Because the previous mapper also used the source=* tag, I knew it was based on Yahoo imagery so I also improved the level of detail according to the high resolution Bing imagery available here.

Two new features I identified at this airport: parallel runways and terminal buildings. If runways are named based on magnetic heading then what happens for parallel runways? Turns out this is handled pretty simply. Just tack "R" and "L" to the end of it based on which one is to the right and which one is to the left as you are approaching the runway to land. So one runway is named 1L/19R and the other 1R/19L.

Terminal buildings aren't too hard either. In this case, there is a main airport building and two wings that extend to the east and west. These wings are where the gates are. Since I have been to this airport before, I know they are simply referred to "East Concourse" and "West Concourse." The previous mapper had done it as a single huge polygon with a building=yes tag. I decided to split it up int 3 parts so that I could name the concourses. I added an aeroway=terminal tag to the two concourses. Technically I believe there is a difference between a "terminal" and a "concourse" but the distinction seems a little hazy in common usage and depending on airport design. OSM doesn't seem to have anything mapped as a concourse so I just stuck with terminal. Anyway, here is the finished product:

The couple of times I have embarked on bigger projects like this on OSM, I tend to start out mapping basics. As I go I find more details to map and by the end I'm doing some serious micro-mapping and worrying about sub-meter offsets in aerial imagery. In this case, these tendencies were worsened by the increasingly good resolution of aerial imagery as I moved into the urban areas of eastern Kansas. The last airport I touched was Kansas City International airport. Yes, it is in Missouri. But it is in the part that was robbed from Kansas by the pesky meandering of the Missouri river. Also, it was within the simple square bounding box I used to download airport nodes from the XAPI.

Again, a lot of features were already basically there but were traced from old Yahoo imagery. Having flown out of this airport multiple times, I was able to use my own local knowledge to augment the aerial imagery. For example, the original imagery had a good bit of radial distortion and it was hard to distinguish the terminal buildings from the large overhang that covers the sidewalk in front of them so the buildings were lopsided and not uniform like I know them to be in reality. Here is a "before" shot I got from OpenCycleMap before it had a chance to re-render:

Not horrible at this zoom level but when viewing it in JOSM with z21 aerial imagery, it looked a lot worse. I found this airport to be kind of fun to work on and ended up adding more and more detail over a period of a couple of days. With large operations like this and high resolution imagery, you can pretty much add as much detail as you want, especially if you have some local knowledge to add to the imagery. So here is what I did with the terminals:

I used the architecture to my advantage and placed nodes in the columns of the building where straight segments connect at an angle to make a curved building. Here you can clearly see the large overhang in front of the building that somewhat confused the previous mapper.

One area that wasn't mapped very well were the long term parking lots north of the airport as well as the buses that take you from long term parking to the terminals. Again, my local knowledge combined with the high resolution aerial imagery let me pick out a lot of features. For example, bus stops, tagged highway=bus_stop and shelter=yes since they are small enclosed huts to protect travelers from the wind that usually blows from some direction around here.

Then of course there are the parking aisles (highway=service/service=parking_aisle) which in this case can also be tagged as oneway=yes since you can clearly see the angled parking stalls. Also, there are dedicated bus service roads leading from the parking lots to the main road to the airport. Access to these roads is restricted to buses only and enforced by lift gates. The roads are tagged tagged highway=service/access=private and the lift gates, barrier=lift_gate/access=private. This is where a restricted bus route enters a parking area through a lift gate:

Then of course there are the parking areas themselves. These were already present but I refined them and added some more tags. Currently they are tagged with this: amenity=parking/parking=surface/fee=yes to indicate that this is a surface (as opposed to underground or garage) parking lot and that a fee is charged for parking here. The result is this map of the long-term parking area:

One useful thing I need to add is either a name or ref tag for the parking areas. I know they are named to correspond with the terminals (A, B, C) and you are supposed to park in the lot that is named the same as the terminal you will be flying out of (for bus routing reasons) but I don't recall which is which off the top of my head. Guess it's time for a trip to some place exotic so I can survey the airport parking lot on the way there.

The short term parking in the center of the terminal buildings can be tagged similarly but with "parking=multi-storey" instead of "surface" to indicate that these are multi-level parking garages. They aren't rendered any differently though. There are also walkways that cross a small bridge from the top of the parking garage to the streets in front of the terminal buildings. Here is a view of one of the terminals:

Notice the "B" in the middle of the parking area. That is the name of this terminal building. This is kind of a bug in the Mapnik rendering software in my opinion. I believe it uses a "center of gravity" algorithm for determining where to place polygon labels. In this case, the center of gravity of the building is outside of the building outline. Oops.

The last bit of micro-mapping involved the grassy areas between the runways/taxiways. The wiki page mentions that they can be mapped as landuse=grass so I thought I'd give it a shot and see how it turned out. The result seems pleasing to the eye however it is a fair a mount of effort. Although I personally find mapping just about anything at zoom level 21 is kind of fun. Like this:

Yes, I found a windsock in aerial imagery. And of course there is a tag for it! aeroway=windsock - and there are currently 280 of them in the database. JOSM even renders a little red/white striped cone on a little stick! However it is not rendered in the default map styles on osm.org.

After adding a few more features like fences where I could clearly make them out (barrier=fence) and gates (barrier=gate) as well as the drainage ditches, (waterway=drain) here is the Kansas City International Airport:

The main thing I'm not really happy about is the road classification. When I started, everything in the area was tagged as highway=residential, as imported from TIGER. I upgraded the major road coming in to the airport to trunk since it is basically built like an interstate. Even some of the roads inside of the security perimeter of the airport had been imported from TIGER. These I changed to highway=service and added access=private. The main circle that goes to all three terminals is marked as tertiary. This all seems fine. The terminal roads themselves are very low speed (I think the sign says 15 mph) but also very wide. There are two 3-lane roads separated by an island in the middle for taxis and hotel buses and such. They seem too big to be called a service road. I left them classified as residential for now but this doesn't seem quite right either. Same with the roads leading to the long term parking lots. Suggestions are of course welcome. :)

One thing I haven't touched on is gates. They can be mapped by tagging a node with aeroway=gate and ref=* to indicate the gate number. I have no source of gate names/locations so for now I ignored them. I was also unsure about how to map them. One airport I looked at had lone gate nodes floating out in the apron, essentially where the airplane would sit to dock at the gate. While this kind of makes sense from the pilot's perspective, I might argue that gates should probably be mapped at the edge of the terminal building or even inside the building to make them of use to passengers wanting to find out where they need to go in the airport to find their gate.

And one last note: there are a good number of adjacent features (mostly apron/terminals/landuse) that share nodes with each other right now in my work at the Kansas City airport. These should probably be converted to use multipolygons as node sharing is often frowned upon for this kind of thing and is indeed somewhat harder to deal with in some instances. That's the way it was mapped when I got there. My edits made the map better. Not perfect. Yet.

And so concludes my great Kansas airport mapping adventure. Thanks for flying!

Airports, Part 1

noreply@blogger.com (Toby Murray) — Fri, 07 Jan 2011 07:47:00 +0000

For the past few days I've gone on an airport binge in OpenStreetMap. I ended up learning a lot about airports. It was fun to start mapping, then come across something different or notice a new detail and realizing that this too can be mapped! Here are my musings on the subject. Feel free to tell me I did it all wrong in the comments :)

The recent import of airport nodes from www.ourairports.com and the subsequent discussion sparked the idea to map easily identifiable runways from aerial imagery. So I downloaded all "aeroway=aerodrome" nodes within Kansas from the XAPI. In addition to the above mentioned import, a lot of airports were imported from USGS GNIS (Geographic Names Information System) back in early 2009. In total there were just under 600.

This being Kansas, a vast majority of those 600 "airports" are nothing more than a flat spot in a field where farmer Joe lands his crop duster. They have such names as "Lundgren Hereford Ranch Airport" or "Flying N Ranch Airport." They should probably be downgraded from "aerodrome" to something else and not rendered at zoom level 10 like they are now. Right now looking at western Kansas at z10 reveals a ridiculous number of airports being rendered. I have yet to see any consensus on a tagging scheme for smaller airports so I mostly just ignored them. Additionally, the aerial imagery I was using (USGS NAIP) wasn't really good enough to reliably pinpoint these grass strips in the middle green fields. It also didn't help that the GNIS information is often off by quite a bit so I wasn't sure if the runway was just blending in or if there really wasn't one in the field I was looking at but rather 2 fields over.

The more interesting airports are the municipal air fields. Most of these have a paved runway, sometimes in addition to a grass one. Even small airports stick out when they have a paved runway:

This is pretty easy to map. One runway, 3 taxiways and an apron. A quick look at the Airport page on the OSM wiki will tell you to use the aeroway=* tag with values of runway, taxiway and apron. However upon closer inspection, there is a little bit more information that can be extracted:

That big number 17 on the pavement must mean something, right? I have known that runways have names for a long time. But I never knew what they meant. Maybe this is already common knowledge to some people but it turns out they are based on nothing more than which direction the runway points. The runway above has a heading of 173 degrees. Round that to the nearest 10 and chop off the zero and you get 17. The exception to this naming rule is that there is no zero. Runways that point within 5 degrees of north are named 36. But since each runway has two ends, it also has two names obviously separated from each other by 180 degrees. So this is actually runway 17/35. This goes in a ref=* tag on the way that represents the runway. One catch: This is based on the magnetic north pole, not the geographic one. So in theory these can change over time as the magnetic pole moves. Luckily down south here magnetic and geographic poles aren't too far apart and the magnetic pole only moves about 0.1 degrees per year. However up in northern Canada they actually do use the geographic pole for runway numbering because the declination is ridiculously large. I'm sure you will thank me for this information next time you are flying in northern Canada.

Update: One of the first readers of this post (petschge on IRC) pointed out that the magnetic pole shift has actually affected at least one major airport: Tampa International

Next, notice the lines extending across the taxiways a couple hundred feet from the edge of the runway. Kind of hard to make out in this picture but they are there. These are the holding positions where airplanes must stop and wait for permission to enter the runway. In OSM these are tagged with aeroway=holding_position. They are not rendered on the default map style but it is still useful information.

Well that's done. The other thing that could be mapped is the buildings and the outline of the airport land. I did map buildings on some airports but not this one. The land area I found difficult to determine from aerial imagery. It was hard to tell if I was looking at an area of mowed grass that belonged to the airport or a field of soybeans so I left the airports as single nodes for now. The final product: (the airport name isn't actually rendered at this zoom level. That "Russell" is from the city boundary around the airport. And I just noticed now that the city boundary probably also defines the outline of the airport. Hindsight!)

Side note: I was surprised and somewhat disturbed by how many cemeteries I saw immediately next to airport runways in rural Kansas. It kind of makes sense from a landuse perspective. You can't put tall buildings next to a runway. Plus, the local residents won't repeatedly call city hall to complain about the noise. But still, it seems kind of disconcerting in a slightly humorous way.

Next up:

Ok, we've got the runway which will be named 17/35 again. The taxiway with the holding position, check. But what's this business with the big arrows and the runway start line not being... well... at the start of the runway?! That's what I asked on IRC. User stevensn had the answer. It is called a "displaced threshold." What's that? To Wikipedia! Turns out a displaced threshold is a section of the runway which can be used for planes to start their takeoff roll but it can not be used to land on. In this case, there is a prison just south of the runway. Presumably it has tall fences or walls which means the glide slope for approach to this runway is too high to safely hit the beginning of the runway.

This setup also makes me conjure up images of a low flying airplane dropping a bungee cord into the prison yard to yank someone out. Probably impractical due to airspeed and g-force limitations but I'm sure Hollywood could stuff it into a movie somehow. If they do, I want royalties! There have however been several successful (and some failed) helicopter jailbreaks. Google it for fun some time.

Anyway, back to mapping. There is no guidance on how to map displaced thresholds on the OSM wiki. Using the TagInfo tool I looked around and noticed that there were about 15 existing uses of "aeroway=displaced_threshold" tags. This seemed pretty logical to me so that's what I used. Now there are almost 40 uses of this tag. Unfortunately the renderers don't know about it so it doesn't show up on the map but as with holding positions, it is still useful information and if enough of them get mapped we might be able to convince the renderers to add support for it. The only real problem is when a taxiway connects to a displaced threshold. On the map it looks like the taxiway just ends in the middle of nowhere. But this is a rendering problem and remember, in OSM we tag for what's on the ground, not the renderer!

And finally (for now) two more new things I learned about:

The yellow chevrons might appear to be similar to the displaced threshold but of course aviation tends to be a very precise business so each splash of paint is precisely regulated to mean something distinct. Going back to the wikipedia page on runways, we see that these are "Blast pads, also known as overrun areas or stopways" and are constructed to take the brunt of the hot air blast when jet engines throttle up for takeoff so that the ground doesn't get eroded. Their construction may not be as robust as the rest of the runway so aircraft are prohibited from using this area for any purpose except in emergencies.

Blast pads/overruns/stopways are also not documented in OSM. Once again, I looked at TagInfo. There were a few that are tagged as aeroway=stopway and a few more that were tagged as aeroway=overrun. So I picked the more popular one. If someone has a compelling argument that it should be stopway or blast_pad I would not object to changing it. The same issue exists with not being rendered as described above for displaced thresholds although it isn't as critical since active taxiways will not connect to this section of the runway.

Lastly, note the yellow X marks to the left of the runway. This isn't too hard to figure out. It means the runway/taxiway is not in use. It could be temporary, due to construction, or (as is the case here) permanent closure with the possibility of obstacles/damage present which would make landing here dangerous. I tended to not map these. There were a few that were already mapped as active runways/taxiways so I retagged them aeroway=runway_disused/taxiway_disused which I also found used in TagInfo.

I am going to cut this post off here before it gets so long that no one will ever read it. Tune in again soon to see part 2. It will focus on some of the bigger airports and some data sources that I found to help map them in more detail.

<ding> This is your Kansas mapper speaking. You are now free to move about the planet but we ask that you leave your GPS device turned on at all times. We hope you enjoy the resulting map. Thank you for choosing OpenStreetMap. Have a nice day.

Update: Here is part 2!

Updated TIGER map

noreply@blogger.com (Toby Murray) — Thu, 16 Dec 2010 04:45:00 +0000

For a full history of this post you may want to read these three posts first: 1, 2 and 3.

When I first presented my nationwide TIGER map at wherecamp5280, Samat mentioned that he was surprised that his county wasn't more heavily edited since he has done a lot of work there. A few days ago we were chatting on IRC (#osm on oftc.net) and he asked a question that made me realize that the map did indeed have a serious flaw in it. Earlier today a discussion came up on the talk-us mailing list that made me notice that the same error exists in the TIGER edited map that MapQuest (re)set up. So I decided to look at correcting it.

The error is that I was only looking at the latest version of the ways. This means that any edits between the original import and the mass edit that expanded street names were not taken into account. Both the import and the abbreviation expansion happened well before I started mapping so I didn't even consider this fact. As Antony Pegg says later in the mailing list thread, it is too expensive to go back and look at all previous versions. Especially since you would have to use a FULL planet file since the regular ones only contain the current version of objects, not a full history.

But we don't really care about the contents of the edits or who performed them... we just care that they happened. The initial import obviously created version 1 of the ways and if nothing else changed, then version 2 was created by balrog-kun in the name expansion edit. If any edits happened between the import and the name edit, then the ways will be on version 3 or higher. Thus any TIGER way with a version higher than 2 must have been edited by someone other than these users, even if one of them was the last to touch a way.

So here is the updated map which takes this into consideration:

Click for bigger version

There are indeed several noticeable differences. For one, Samat's county (Dona Ana, NM) turned two shades lighter! Also, a large chunk of Wyoming came out of the "virtually untouched" classification. The southern tip of Texas as well as the Dallas area lightened up a bit as well. But at the end of the day, there is still a LOT of untouched data out there.

This map still isn't perfect. For example, if someone split an imported way, creating a new version 1 way as well as version 2 of the existing one and then the name change edit happened, the new way would be counted as not having been touched by a human. Also, people deleting the tiger:county tag on ways will obviously throw numbers off a little. But this should be much better at least. I'm hoping this change can be incorporated into the MapQuest version as well.

For giggles I threw the data up on OpenHeatMap as well. I'm not sure what happened to Alaska and Louisiana. For some reason OpenHeatMap didn't recognize the county names. It also did its own data classification so it won't match my map exactly. But you can zoom in and mouse over counties and get a pop-up with the name and percentage of unedited ways so that is kind of neat.

The Data
For your further enjoyment, I am providing the CSV file that was produced as the result of my SQL query. The map is nice and all but some people might want cold hard numbers. I'm thinking some statistical analysis might be interesting. I originally tried to make a graph of the distribution of values but OpenOffice Calc wasn't cooperating that night and I had to pack for a trip so I dropped it.

Click here to download the file.

The first column is the value from the tiger:county tag. Following that is the percentage of unmodified ways in that county followed by the raw numbers of edited, unedited and total ways. The last three columns were used by me to manipulate the data into an acceptable form to join the data to the shapefile I had. I had to split out state code and county name and then expand the state to its full name using a table I generated from the source of a wikipedia page that lists all states and their abbreviations.

I didn't clean it up at all so there are some nonsensical values in there. ArcMap just ignored the values it couldn't match to the shapefile. The first line has a county name of "30 days late" - I have no clue what that is doing in a tiger:county tag. Feel free to do an XAPI request and clean it up! There is also data there for Puerto Rico and D.C. that didn't get matched up. Puerto Rico wasn't in my shapefile and I don't think D.C was correctly matched to the abbreviation.

Enjoy!

Nationwide TIGER Map

noreply@blogger.com (Toby Murray) — Sun, 21 Nov 2010 07:57:00 +0000

Update: You should of course still read this post because it is is awesome but there is a new post with an updated version of the map here.

I presented most of this material this weekend at WhereCamp5280 and promised to put this online soon. I have a few more details and maybe some raw data that I will put in another post after I get back home after Thanksgiving but for now I'll throw up what I have.

Well after some sweat, blood and tears (mostly on the part of my hard drives) I was eventually successful in extracting the data I needed to make a map of the entire US! If you just want the map feel free to skip down but I am going to document my trials and tribulations a little.

First, some things that did not work:

Importing the whole planet file into an apidb using osmosis. I actually did this before I made the original Kansas map but it is part of the workflow I went through to get to the final map. The initial response I got on IRC was that it would take maybe 200 GB and a few days, maybe a week. I thought it would be fun to poke around in the database a bit beyond just this map so I decided to try it. I believe the time/space estimate may be accurate for importing to a pgsql database for rendering however I wanted a full apidb. The result:

Yes, that is 3 weeks. I kept adding disk space to the volume hoping each time that it was almost done. The good news is that ext4's online resizing works great! But after having my heart broken one too many times I went hunting for more information which I probably should have done first. On the mailing list it was pointed out that the size of the main OSM apidb (The Source) is 1.4 TB and while that does include historic data that isn't in the planet file, I realized that I may only half way done so I decided that I needed a different approach.

I tried trimming out a rough bounding box for the US which worked. Then I tried filtering out only nodes used in ways since I am only interested in the imported TIGER ways. I was hoping that a lot of the overhead in creating the apidb had to do with having to hit the node table every time a way was inserted due to foreign key constraints and such. I don't have a pretty graph for this attempt but apparently osmosis builds a data structure in memory for every node before filtering out the unused ones. Or something... The end result is that a computer with 16 GB of memory in it was still not enough to finish the job and failed within minutes with an "Out of Memory" exception from osmosis.

My last ditch effort was just to import the whole US. I didn't use a fancy polygon to filter, just a straight bounding box that included Alaska and Hawaii so I'm sure I got some extra data (like a good chunk of Canada) but this ended up working... after 80 hours and about to 200 GB.

Success!

Click for a bigger version to see those small counties!

The first thing you might notice is that there are some counties with no data. This is because the TIGER ways in those counties, for one reason or another, do not have tiger:county tags. Some were imported by a user other than DaveHansenTiger and they chose not to import all the TIGER attributes. Some were imported by him but just didn't have the tags. It was suggested at WhereCamp that these may have been the first counties imported and that he may have still been fine-tuning the import process. This seems like a plausible theory.

Other than the missing data, the map makes sense for the most part. The west and northeast coasts are more edited than the center of the country. Large cities are easy to pick out. But there are a few individual counties that do stand out here and there. I'm guessing there are more people like me who have taken an interest in their local area out in the middle of nowhere and have made noticeable contributions to the map. The level of activity in Iowa does seem a little odd. I should single out Iowa and take a look at the data to see if I missed a bot or something. But who knows - it might just be a really motivated Iowan.

Anyway, there you have it. As I said above, there will be another post with some details and data at a later time. Right now I am tired and need to get to bed after spending the day gathering GPS traces on the slopes of Winter Park with Steve, Hurricane, Josh and Richard. It was all for the map - I can assure you that no "fun" was had by anyone!

Well maybe a little.

Two more TIGER maps

noreply@blogger.com (Toby Murray) — Sat, 30 Oct 2010 15:36:00 +0000

After my last post there were some requests for me to do additional states. So I imported Colorado and New Mexico since they are kind of one big block. However doing 50 states a couple at a time will be rather cumbersome so I think after this I will try to coax osmosis into doing a more fine-tuned import so that I can import only the things I need but do it for the entire US at once.

I also learned some more quirks about ArcMap. The program is amazing in some ways but user interface is not one of them. I was also unable to figure out how to use two different fields to join my data to the counties shapefile. So that was slightly frustrating but I was able to work around it.

Anyway, until I get around to poking osmosis again,
I give you Colorado:

Colorado has another strong outlier: Teller county with only 35.8% of ways untouched . I didn't look at the data but I believe I know the person responsible for this, in real life even! Hi SpenSmith! Also, as you can see I replaced the "total number of TIGER ways proportional symbol with a few city names for general reference.

On to New Mexico:

Notice the breakdown here. Looking at the data it would seem that either the TIGER data in New Mexico is nearly perfect and requires no editing or New Mexico is a barren wasteland, incapable of sustaining the fragile life force of mappers. There are no outlying counties and the most heavily edited one (Socorro county) is still sitting at almost 90% untouched. And looking at that county closer, I'm guessing it just doesn't have very many roads and all the editing that has been done there is just related to I-25 that runs right through the middle of the county. Of course this is percentages. As far as total number of TIGER ways touched, McKinley and Bernalillo counties win with 877 and 762 respectively.

OpenHeatMap

Lastly, SeveC pointed me at an interesting website that may be of use once I get my nationwide data lined up: http://www.openheatmap.com/

I went ahead and imported the Colorado data and you can view it here:
http://www.openheatmap.com/view.html?map=FoundlingsFiascoesUngodliest

I found a slight bug in that Broomfield county is missing from OpenHeatMap's database. I chatted with Pete (the admin) about it for a few minutes, until he had to go to participate in an interview for thisweekinmaps. It turns out that he is using census data from 2000 for the county boundaries and Broomfield county didn't yet exist in 2000! I also mentioned to him that it would be nice if state borders were drawn on county based maps and he seemed to agree. There is always something to nitpick, right?

TIGER ways in Kansas

noreply@blogger.com (Toby Murray) — Sun, 24 Oct 2010 21:41:00 +0000

Update: I have a second post with two more state maps up here. More to come!
Update 2: Here are two more posts with nationwide maps.

If you are at all familiar with OpenStreetMap, especially in the US then you will know that most of the roads here are imported from the US Census Bureau TIGER data. This data was imported from the current data in 2007. It is fairly complete but the accuracy leaves a lot to be desired in some areas. For example look at this screen shot comparing the TIGER data to reality (USGS aerial imagery in this case). Some roads are off by over 200 meters. And there are even worse spots - this was just one I was able to find quickly.

Example of bad TIGER data

While the data was pretty complete when it was made, it is now several years old and missing many new housing developments and does not reflect instances where roads were moved. The good news is that, this being a census year, there will be new TIGER data available from the Census Bureau starting at the end of the year. It will be more complete and probably more accurate considering that GPS technology has made some advances over the last 10 years.

Well that's great but the question is how do we use the new data to improve OSM? In the software world this would be an ideal situation to use a diff utility to determine what had changed between two versions and apply patches to update the old version. But I am not aware of any geo-spatial diff utilities.

So how about just deleting the old data and re-importing the new stuff? This would obviously be a disaster in areas with active mappers. All the work they have put into correcting existing TIGER ways would be destroyed and replaced with data that is probably better than the old TIGER import but also probably worse than what local mappers have done.

BUT...

I'm in Kansas. There aren't a lot of mappers here, especially in the western part of the state. Mostly because there just aren't a lot of people out there. Some counties have a population density of 6 people per square mile. Oh and the TIGER data was imported in county sized chunks. So how about handling this on a county-by-county basis? Counties that haven't been touched could just be blown away and refreshed with the new data. If only someone could determine how much TIGER data has changed since the initial import, on a per-county basis... Oh wait, I did!

I am taking a cartography class at Kansas State University this semester. After starting to contribute to OSM I decided to take advantage of the employee tuition assistance and learn more about the subject. For $150/semester, why not! 20% of the grade in the class is for a project where he turned us loose to go get our own data and make some kind of interesting map. So I got my data by importing a Kansas extract from the OSM planet file and made this map:

A couple of notes about the map:

I let ArcMap classify the data using natural breaks. There are 105 counties in Kansas. 104 of them still have have 78% of their TIGER data in its original state. One has had 75% of it modified. Yes, this is the county I live in. Yes, most of it was done by me. Yes, this inflates my ego.
The total count doesn't really do much for the map in my opinion but the class project called for a bivariate map, so a bivariate map I made! Obviously counties with bigger cities have more TIGER ways. This is your dose of non-surprise for the day.
My method of detecting changes by "local" mappers is by no means fool-proof. Basically I looked at the user who last changed a given way. If the user was one of 4 users I identified that were definitely not local mappers, I counted that road as unchanged. (technical details below)
Labels would add a lot of clutter to the map. If you want to see which counties are which, I suggest the Counties of Kansas wikipedia page.

So I guess the question is how do we use this information? This will have to be a discussion amongst the OSM-US community. Due to time limitations I only did Kansas but this could certainly be done for other states (particularly the sparsely populated ones) to help local mappers decide what, if anything, to do with the new TIGER data. My suggestion would be to do a fresh import of any counties that have above a certain threshold of unchanged data. Say 95%? In Kansas that would be 59 counties. If a threshold were decided upon the map classifications could be altered to reflect that number.

For counties that HAVE had local activity, perhaps some process could be set up using tiles from the TIGER edited map that MapQuest has provided as a background layer in JOSM or P2. Then pull up the new TIGER data on top of it and compare. Import missing roads or ones that are more accurate in the new data and haven't been touched by local mappers. That is just a thought that popped into my head as I was writing this.

Technical Details

I imported the October 20th planet file into an apidb using osmosis. I originally tried to do the whole planet and intended to do a larger analysis of bot activity on a worldwide basis but after 4 weeks and 350 GB it showed no signs of being anywhere close to finished so I fell back to only importing Kansas. Luckily Kansas is pretty much a big rectangle so I just used a simple bounding box. This finished in about 2 hours.

Here is the final SQL query I came up with to get me all the data I needed in one result set. Keep in mind that I only imported a bounding box around Kansas:

select v as county,
sum(CASE WHEN user_id in (147510,7168,20587,293105)
    THEN 1 ELSE 0 END) as bot_count,
sum(CASE WHEN user_id in (147510,7168,20587,293105)
    THEN 0 ELSE 1 END) as user_count,
count(*) as total_count
from ways, way_tags, changesets
where ways.id = way_tags.id
and ways.changeset_id = changesets.id
and k = 'tiger:county'
and v like '%KS%'
and v not like '%;%'
group by v
order by v

In less SQLish terms: I am primarily looking at the tiger:county tag to group the query by county. It contains values like "Riley, KS" so basically I am looking for any way with a 'tiger:county=*KS*' tag. This excludes the few ways around the Kansas border that are from other states since my bounding box was just a little bigger than the state borders. However I exclude ways that have multiple values in the tag, separated by a semicolon. Typically this would come from two ways in adjacent counties being joined together. This is pretty rare so I ignored them. Once I find those ways I examine the user who last touched the way by joining through the changesets table. If the user ID is one of 4 values then I count the way as not having been modified by a local mapper. Those 4 user IDs belong to the following users:

147510 = woodpeck_fixbot (this is a bot that has performed various automated edits as documented on the OSM wiki)
7168 = DaveHansenTiger (this is the user who did the initial TIGER import)
20587 = balrog-kun (this user did some mass editing of TIGER ways by expanding abbreviations in street names)
293105 = NHD edits (NHD = National Hydrography Dataset. I'm assuming this user probably imported some rivers and ended up splitting some TIGER ways to make bridges or something

Looking at the data now, I should have maybe also excluded NE2. I believe he has done a lot of work on national highway/interstate routes. I don't think there is any reason to re-import ways that are part of those systems since they have since been added to route relations and such. So those should probably be excluded from the data. Hindsight and all that.

Anyway, this query gives me a result like this:

county            | bot_count | user_count | total_count
------------------+-----------+------------+-------------
 Allen, KS        |      3062 |         33 |        3095
 Anderson, KS     |      2879 |         64 |        2943
 Atchison, KS     |      1235 |         37 |        1272
 Barber, KS       |      2173 |         26 |        2199
 Barton, KS       |      1764 |        254 |        2018
 Bourbon, KS      |      3815 |        137 |        3952
 Brown, KS        |       907 |         46 |         953
.
.
.

From there I regex'd it into a CSV file and imported it into ArcMap and joined it to the shapefile of Kansas counties that was provided to us. Easy, right?

The Shiny New Blog

noreply@blogger.com (Toby Murray) — Sat, 23 Oct 2010 05:53:00 +0000

I have avoided creating a blog for years now. I never felt like I had enough important ideas to force upon the world to justify it. That and I assumed I would abandon it at some point and it would become yet another piece of litter along the information superhighway. I'm not convinced that these things have really changed but there have been a few times recently where I did want to post an idea to a more permanent medium than IRC, especially concerning my latest hobby: OpenStreetMap. So here I am, creating a blog on a Friday night.

Who knows where it will go but I do hope to post occasional observations concerning OpenStreetMap as well as some open source topics in general (yes, I use Linux) and maybe an occasional note about my job. I work as a developer for Kansas State University. Java with a heavy dose of SQL (Oracle) if you are curious.

Now, here is a random picture of me at the end of No-Shave November in 2008. If anyone reading this doesn't know me, it is a horrible first impression because I usually shave. But there you have it.