Off the Map - Official Blog of FortiusOne

Dataset of the Day: Transparency and Community

Bill Greer — Tue, 30 Jun 2009 18:48:47 +0000

Since Obama has taken over the White House, there has been a significant emphasis on open and transparent government. Some people claim that Obama hasn’t gone far enough in opening up government information, I’ve seen drastic changes in the way the government is exposing data to the public, Such as Data.gov and Recovery.gov. There are also some serious initiatives through the tech community to develop their own tools to help government become more transparent. There are a variety of Tools available, such as the Sunlight Foundations new Transparency Corps and Stimulus Watch which are providing valuable services, and building up an open government infrastructure.

Many of these tools are used to show in depth views of particular Bills or government projects. One example of this comes from aGeoCommons community member, Joseph Hirl from Agilis Advisory, who made this map to show how members of the House voted on Bill HR 2454, American Clean Energy And Security Act of 2009.

We really are seeing a change in the way government information is presented to the public, and the public’s role in voicing their opinions on that information. If there are any other government transparency tools or websites out there we are forgetting, feel free to mention them in the comments.

Data Set of the Day: Michael Jackson and Farrah Fawcett

Bill Greer — Mon, 29 Jun 2009 15:58:07 +0000

With the passing of Michael Jackson and Farrah Fawcett this past week, we wanted to pay tribute the best way we knew how; with a map. This map shows an index of Google searches for the late celebrities, the higher the number the more searches from that state. Interestingly enough, Searches for Michael Jackson seem to be coming from the most heavily populated states, but Farrah Fawcetts’ fans seem to be located in the southern states. Click the eyeball icon to turn layers on and off:

I’ll leave you with my favorite Michael Jackson video, Enjoy!

Dataset of the Day: Cancer Rates vs. Cancer Risks

Kevin Burke — Thu, 25 Jun 2009 23:20:11 +0000

I recently came across an article in USA Today about a recent study that was conducted by the EPA. The study, 2002 National-Scale Air Toxics Assessment, dealt with air pollution levels across the USA. One interesting feature of the study was the findings on cancer risk estimates. These estimates provide information on which areas across the country have high or low risks of cancer- causing air toxins in their environment.

I decided to load the data into Finder! and Maker! and compare the risk estimates from the EPA with actual cancer incident rates that have been reported by the National Cancer Institute in their State Cancer Profile Pages.

Below are a few interactive maps of some of the data that was loaded from both the EPA and the National Cancer Institute on cancer risks and cancer rates.

Cancer Risk vs. Cancer Rate on the State Level

Cancer Incident Rate (per 100,000) by County

EPA Cancer Risk Estimates by County

I then decided to run correlations between cancer rate vs. cancer risk on both the state level and on the county level. Data for cancer rates is missing for a few states for unknown reasons, so their data has not been included in the analysis. Below are the maps and the results. I want to see if the EPA high cancer risks areas are in fact areas where cancer rates are also high. (click on the image for a larger view)

On the state level you can see the overall correlation is at .29. This shows that the EPA risk levels are not a strong indicator of actual cancer rates by state. Now let’s get a more in-depth look at the data on the county level.

From the county level data we see a few things. The first thing that jumps out is the correlation of .19. With a value so far away from 1 or -1 we see that the risk and the rate do not really correlate with one another on the county level. Places that we would expect to see high cancer risk rates do not always have high actual rates. Look at the chart below that shows the top 5 cancer risk counties and the bottom five cancer risk counties.

It is interesting to see that the cancer rate does not differ much at all between the top five cancer risk counties and the bottom five cancer risk counties.

All in all I found this data and the maps that were created from the data to be very interesting. Investigate on your own in you particulat locale to see how cancer is affecting and could possible affect the community that you live in.

Finder Links:
Cancer Incidence Rates by State, USA, 2000-2005 Average
EPA Cancer Risks Estimates by State, USA, 2002
Cancer Incidence Rates by County, USA, 2000-2005 Average
EPA Cancer Risks Estimates by County, USA, 2002

Dataset of the Day: Kiva and Global Poverty

Bill Greer — Fri, 19 Jun 2009 15:48:45 +0000

For anyone who hasn’t heard about Kiva, Kiva’s mission is to connect people through lending for the sake of alleviating poverty. They have developed a system of micro-lending, allowing anyone to lend small amounts of money to people in need, the money is paid back, and everyone is happy. Its truely a great idea to help those in need, i was curious about trying to highlight or quantify whether or not Kiva was working in the right direction and getting money to those in need so i created this quick little mashup of the featured entrepreneurs from Kiva and Global Poverty Data from the United Nations Development Programme (UNDP). I hope both Kiva and the UNDP continue their good work!

Dataset of the Day: Iran Election

Bill Greer — Tue, 16 Jun 2009 19:47:50 +0000

The Iranian election is clearly the hot topic in the news today, but its also showing the power of social media and the openness of the internet and web based tools once again. Even major mainstream news media outlets such as CNN are relying on Twitter and Facebook to help get information across the borders after Iranian officials shut down access to newspapers and internet. To follow the newsfeeds on Twitter search for the #IranElection tag which will keep up aware of everything as its happening. other interesting conversations are going on about the validity of the election.

We’ve added a map into the social media mix showing the ‘official election results‘ from the Iranian Interior Ministry, will be interested to see where the results change after a possible recount.

make your own maps with this and related election data here. This includes vote counts for the other officials, not to mention the number of valid and invalid votes per district.

If anyone has any data that they would like to see mapped, leave a link in the comments section and we’ll do the work for you, to help get the word out.

A Quick Test Drive of Google Table Fusion

Sean Gorman — Fri, 12 Jun 2009 20:32:47 +0000

Google Labs launched a cool new app yesterday called Google Table Fusion Although, the name is a bit confusing since there is also the Google Earth Fusion Server. Anyways, it is a great step forward into the wide world of structured data for Google. This seems to be becoming a trend following the launch of Google Public Data a little while back. The future of structured data on the Web is probably a whole discussion in itself, as well as what Google’s entrance means, but I’ll keep to just the new app for the sake of brevity.

Google Table allows users to upload up to 100mb .csv or .xls file and then allows filtering, merging and visualization of the data. Each of the basic options has lots of nice functionality. Filter allows you do some basic operators like > “greater than” < "less than" >= “greater than or equal to” < = "less than or equal to" against any single numerical attribute in the table. You can also merge two tables together either in total or by attribute. Lastly, Google Table provides a variety of data visualization options - map, intensity map (i.e. thematic map), line graph, bar chart, pie chart, scatter plot, motion (Gapminder's stuff), and timeline.

With that as a high level overview I’ll dig into geographic capabilities a bit, since we tend to like maps. On the plus side this is the first I’ve seen from Google directly linking structured data with maps. Google Charts has had thematic maps for a while, which is leveraged for the thematic mapping in Google Table. Also Google Table has some georeferencing capabilities that auto-recognizes geographic names and georeferences them on the fly.

As MapperZ points out Google Table has “fast batch geocoding” capabilities- except you can only look at the geocoded files 200 at a time and not get the georeferenced data out of the system. It appears that Google is georeferencing a set on place names and addresses - from what I can gather this includes country names, city names (domestic and international) in addition to doing traditional street address geocoding. The georeferencing and geocoding appears to only work in Google Maps unless it is country names. If you have country names then there is an option to visualize the data in Google Charts. This can cause some issues - for example look at this state GDP data I uploaded. The first pic is data geocoded by clicking on the “Map” option:

For some reason the geocoder missed Washington state - looks like it confused the Washington state with Washington DC.

The problem becomes even more severe with Google Chart’s georeferencing where it thinks the state of Georgia is the country of Georgia since that is the only country name it recognizes:

These are problems all geocoders run into, but I think it is endemic of where Google Table comes up short. It is a bit of an application Frankenstein where it does many things but none of the very well. Several different Google apps glued together, but without the cohesive work-flow and technology to make it work as a platform. The integration and capabilities for data search, metadata, handling data other than .csv, large data visualization,and ease of use will all be keys for success. Also in the negative category, the big disclaimer, any data you upload into Google Table incurs the dreaded section 11 clause:

By submitting, posting or displaying the content you give Google a perpetual, irrevocable, worldwide, royalty-free, and non-exclusive licence to reproduce, adapt, modify, translate, publish, publicly perform, publicly display and distribute any Content which you submit, post or display on or through, the Services.

Criticisms aside it is pre-alpha and a “Labs” project. There are many cool innovations like commenting within data sets. Also I think Google Table is a trend we’ll see more of where the geographic components of data is just another option in a larger system and not a system in and of itself. The overall project points towards the big potential in building a cohesive structured data platform, and it will be fun to see which of the various efforts in the ecosystem wins the day.

Dataset of the Day: The New Digital (TV) Divide

Bill Greer — Fri, 12 Jun 2009 15:25:42 +0000

Time to Switch over to Digital TV from the old analog system. This has been coming for a long time and its finally here. From the official DTV goverment website, “The switch from analog to digital broadcast television is referred to as the digital TV (DTV) transition. In 1996, the U.S. Congress authorized the distribution of an additional broadcast channel to each broadcast TV station so that they could start a digital broadcast channel while simultaneously continuing their analog broadcast channel. Later, Congress set June 12, 2009 as the final date that full power television stations can broadcast analog signals. As of June 13, 2009, full power television stations will only broadcast digital, over-the-air signals. Your local broadcasters may make the transition before then, and some already have.”

So who is ready for the switch and who isn’t? We made a few maps based on the Neilson report showing who was ready and who will be left behind in analog. Check the maps out below.

Click on the Eye Icon to turn layers on and off.

To download the data or view the meta data visit Finder here!

Creating the National Broadband Map for $3.5 Million Instead of $350 Million

Sean Gorman — Tue, 09 Jun 2009 16:47:26 +0000

There has been a bit of hub-bub about the provision in the stimulus (ARRA) plan to map broadband in the United States and the $350 million price tag. Specifically the statute calls for “$350,000,000 shall establish the State Broadband Data and Development Grant program, as authorized by Public Law 110-385 *and* for the development and maintenance of a national broadband inventory map as authorized by division B of this Act.”

In a response to an All Point Blog podcast on the topic Archie Belaney said he thought FortiusOne could accomplish the same task for $3.5 million. With gauntlet on the floor it got me thinking about how you would actually pull it off in an economical fashion. It has been a few years since I’ve been deep into the weeds of broadband mapping, so some of my thoughts may not be completely aligned with the status quo.

One of the big challenges, often missed, is that many broadband providers do not know where their fiber is. Large amounts of data still resides in paper maps - in a 2003 court case involving the Maine Public Utilities Commission an expert witness for Verizon states that standard operating procedure for restoring a failed line is to locate the lead engineer for the region, consult paper maps, manually identify an alternate route, and send technicians to wire jump around the outage. The point being that in many cases there is not a nice clean digital database of broadband deployments and routes.

So to build a national broadband map one of the biggest challenges is data collection and aggregation. Therefore the tools in GeoCommons to upload existing digital data is going to be insufficient. To fill this gap there are two key capabilities required digitizing paper maps and collecting data from the field. To solve both problems I think there is a good bit we can borrow from the OpenStreetMap (OSM) project. Broadband fiber lines all have to follow right-of-ways along streets, railroads, pipelines etc. The late 90’s saw a boom in companies whose business models were just finding new right-of-ways to lay fiber across. Mapping these right-of-ways is no different than the streets, railroads, cycle ways and footpaths mapped in OSM.

Since the goal here is to map where broadband is deployed so we can better gauge “universal access” the accuracy of commercial GPS should more than get the job done. Data collection could easily follow the OSM model and a server could be set up inexpensively and leverage any GPS device for data collection. This covers one use case where the data needs to be collected from the field, but is likely to only be a small percentage of data that needs to be created. The larger project is likely to be digitizing paper maps, which I believe is yet another task that can be easily crowdsourced (even if the crowd is not the public).

A good deal of the data in OSM does not come from GPS data from the field but instead from users tracing roads from satellite imagery. This same approach could be used to easily digitize existing paper maps of broadband deployments. To enable this you could leverage Chippy and Schuyler’s Map Warper to pull paper maps into OSM, allow them to be traced and entered into a common system.

I think the combination of OSM and Map Warper would cover the big data creation and conversion issues. You could use GeoCommons to allow the various stakeholders to contribute their existing spatial data on broadband deployments to the system. You could also allow stakeholders to create maps of broadband coverage identify identify need based gaps by overlaying demographic data. Below is an example of cable broadband lines in Vermont overlaid with population count with income between $10-20k.

When you filter the data to locations with more than 150 people in the $10-20k range you can begin to see the mismatch between broadband service and low income population.

While I’m not sure of the exact cost breakdown for what is proposed above, you could deploy everything to EC2 and probably cover software/hardware for half of the 3.5 million. Then cover the engineering labor needed with the other half. Data collection/migration costs would be the biggest variable with lots of possible options - Americorps, volunteers, offshoring (probably unpopular), or leveraging various job works programs. The good news is that it would not require a very technical workforce to accomplish the tasks. It would be a cool project but sadly we’ll probably spend the whole $350 million and still not have a decent product to show for it. On the upside some of the concepts proposed are already being adopted by a few telcos who are trying to crowdsource their coverage maps. RidgeviewTel created a Web site where the public can drop points for where they can’t receive broadband service.

Regardless of how it happens once the data is aggregated we could make maps like these for the entire country:

Or do calculations of broadband deployment concentrations like this (also NYC):

Dataset of the Day: North Korea Uncovered

margot — Mon, 08 Jun 2009 13:55:19 +0000

Curtis Melvin’s North Korea Uncovered gives the public a glimpse into a country typically shrouded in mystery and regulation. The Google Earth project displays information about anything from food and culture, to concentrations of mining operations and prisoner encampments, and even shows the locations of prisoner’s barracks and guard dining halls. Never before has the public been able to take such a look at North Korea thanks to Melvin’s efforts to harness the power of open-source data and mapping utilities.

We’ve broken down the original KMZ file into several kml files and uploaded them into Finder! to showcase just how much information has been collected about the country on various topics. Here are some samples of maps we’ve made from the data provided by Melvin’s project.

Another great way to view the data is to simply download the data as a csv from a layer in Finder!, sort based on any attribute and then make a map of that data. I made a csv of lighthouses from the transportation file. I sorted based on names with lighthouse in them, then saved as a new csv and uploaded it into Finder!

See what things you can come up with using Finder! and Maker! and enjoy the wealth of data available on Geocommons and uncovering North Korea for yourself.

Please visit Geocommons to view any of the maps/layers discussed above.
Lighthouse map: http://maker.geocommons.com/maps/5784?page=
Lighthouse layer: http://finder.geocommons.com/overlays/13232
Gulag map: http://maker.geocommons.com/maps/5776?page=
Gulag layer: http://finder.geocommons.com/overlays/13208
Agriculture/Horticulture map: http://maker.geocommons.com/maps/5788
Agriculture/Horticulture layer: http://finder.geocommons.com/overlays/13185

Same Sex Marriage and Straight Divorce

margot — Wed, 03 Jun 2009 18:32:50 +0000

With the recent passing of Prop 8 in California, I was inspired to do some research into gay marriage laws in the US as well as divorce statistics for straight couples. Originally I had intended to put together an opinionated piece about gay marriage laws and divorce rates, but in my search I was thwarted by a lack of solid information and references. It is exactly this “lack of” information that I find interesting. Why don’t some states report divorce numbers? Why is it so hard to find good religious statistics? Also, is there any readily available data on the rate of divorce for gay couples? Is the process of collecting the data too difficult or is there just not enough interest? Based on what I did find, here are some simple observations.

The United States has one of the highest divorce rates in the world and is also a nation polarized on the issue of gay marriage.

I thought it might be interesting to look at gay marriage laws on a state by state basis, and used the LA Times online as a great resource. Last week they published a fantastic timeline on gay marriage laws by state from January of 2000 to May of 2009. The timeline uses ten different categories of laws from least rights to most rights for gay marriage and animates how a state (and the nation) has changed over time. I took data from the first and last dates on the map, and mapped a difference for each state’s laws on gay marriage using Maker! What I found was that middle America has become less and less gay-friendly and the coastal states have tended to become more gay-friendly. States with negative numbers have become more strict on gay marriage, while those with positive have become more for gay marriage. Those states with zero values have remained the same for those two dates, but that can mean either for or against gay marriage. When you view the map in Geocommons (link published below) click on individual states for their past and current laws and to see whether or not they have changed at all.

The next thing I thought might be interesting was a state by state straight divorce rate map. Unfortunately, this kind of data is very difficult to find in one place. And many states such as California, Minnesota, Indiana, Louisiana, Georgia and Hawaii have stopped reporting their straight divorce statistics. I’m not sure why this is, but it made mapping divorce rates very difficult. I did find some information at the United Nations Statistics Division page and I’ve simply presented each state’s raw number of straight divorces for the year 2007 excluding the states mentioned above. Again, follow the link to the map below to see exact numbers for the states included.

All of these points have inspired me to dig deeper into these issues. If you have good data about any of these topics and would like to create your own maps, please post a comment with your thoughts and use Finder! and Maker! to see if you can find any correlations between gay marriage, divorce, religion, or whatever you think might be interesting. Leaving aside my thoughts on gay marriage and divorce, I think it is safe to say that divorce is a problem in our country and as a nation we are very polarized on the issue of gay marriage. What are your thoughts?

Here are the links to view these data layers:

http://finder.geocommons.com/overlays/13181
http://finder.geocommons.com/overlays/13182
http://maker.geocommons.com/maps/5750
http://maker.geocommons.com/maps/5751