High Earth Orbit

Data Dissemination to the Haiti Government

2010-02-08T15:09:42Z

In a joint project with the World Bank, USAID, and numerous other partners, there are now 6 TB hard drives on the ground in Haiti with mapping tools and satellite and remote imagery data being shared with the Haitian government. Read more about the project on the FortiusOne blog.

Schuyler Erle and Tom Buckley will be heading down on Tuesday to provide on the ground support between the government agencies and the community.

A tremendous thank you to the numerous individuals and groups that helped and provided tools or data: World Bank, San Diego State University / Calit2, Internet2, Georgetown University, DigitalGlobe, Delta State University, Sahaha, Crisis Mappers, OpenStreetMap, NOAA, Ushahidi, DevelopmentSeed, TelaScience, STAR-TIDES, CrisisCommons, USAID, GeoCommons, OpenSGI, GeoEye.

Grassroots Crisis development organization

2010-01-18T19:21:20Z

On Saturday, CrisisCamp Haiti was a revolutionary step that was indubitably a success. Within 3 days of an idea a small group of people helped coordinate and run a series of CrisisCamp Haiti code-a-thons across 5+ cities, over 400 participants, and at least 20 continuous hours of work. At least 6 projects were started, and many more existing projects added people to their community, taught new skills, and built out new features.

In general, the last week has involved a whirlwind of grassroots organization and development of numerous projects. This change of realtime engagement and response by volunteers and non-traditional organizations through internet has no doubt raised the hackles, or at least the concern, of traditional responders, agencies, and government. There are often voiced considerations of causing confusion, providing technology that will have no use, and lack of organization and hierarchy.

Even within these grassroots participants there are calls for centralization, and building chains of responsibility that are somewhat antithesis to the very mechanism by which the project started and how it acts. Many of these projects formulated from simple ideas, growth through passion, an aligned community, and freedom to explore ideas and vet these within the organization. Over time the best ideas crystalize and become part of the long term project and others spin out to new projects.

It’s about the Mindshare and Multiplied Resources

In the beginning of a Crisis response there is an intense desire for people to engage and provide some type of resource: money, time, guidance, knowledge, contacts. At the same time, there is the alternate side of organizations seeking these vast, and limited, resources. Aid agencies put out SMS shortcodes for donations, PayPal links, matching funds. First responders need time, physical labor, and fortitude. Technology projects seek knowledge, translation, testing, documentation, data, integration.

Perhaps uniquely, technology has the possibility of multiplying any individuals efforts. By providing code, or data, and aggregating that data out, my contribution can feed into numerous other projects – whereas time or money is nominally a single use resource. It can buy water, or work for an hour moving rubble, and that’s all that resource can do for that time.

So a perceived problem is in bifurcation and redundancy of efforts and confusion. This can largely be mitigated by open collaboration, and easily sharing data through interchanges. Projects like the People Finder is slowly converging on this type of solution through the use of PFIF exchange and common aggregation points with API’s.

We’re working on improving the CrisisCommons.org site and wiki in order to track active projects, aggregate similar efforts and point volunteers to project homes to join their individual communities.

Haiti Mapping

2010-01-15T12:56:09Z

The last 2 days have been filled with coordinating various efforts in gathering information and volunteers responding to the massive Haiti earthquakes of January 12. The analysis team at FortiusOne has put together a news dashboard highlighting the event and current response efforts.

There have been several tremendous groups that have actively been contributing data and tools both with remote developers and responders on the ground. CrisisMappers, CrisisCommons, Ushahidi, OpenStreetMap, just to name a few.

Many data providers have been making their data freely available. This is most notable when looking at Mikel’s screenshots of OpenStreetMap before the quake and after volunteers began tracing over historic maps and newer satellite imagery from Digital Globe and GeoEye.

Other efforts:

Ushahidi Haiti is crowd-sourcing reports. You can send a text message to 447624802524, send an email to haiti@ushahidi.com, or send a tweet with the hashtag/s #haiti or #haitiquake.
The CrisisCommons Wiki has a list of available data and organizations
Sahana has a form to list offices and organizations that are working on the ground
GeoCommons search for Haiti has all the datasets and maps that people have contributed for download as Spreadsheet, Shapefile, KML, and more
OpenStreetMap’s Project Haiti has a list of datasets and people tracing data

excited about in 2010

2010-01-07T13:20:02Z

As always, each new year brings a refreshed feeling of excitement. Perhaps its the long holidays and copious amounts of food, family and fun, or seeing a magic new number on the calendar that makes it feel like “The Future!”, or just a desire to take advantage of an allowed re-emergence of self and goal setting. Of course, time isn’t discontinous, so 2010 isn’t disconnected from the current continuum of development and trends – but it’s still worthwhile to take the time to step back and consider where we are and where we’re going.

Mashable and James, amongst many others, have excellent predictions that will and won’t happen in 2010. Generally they are good insight into trends in the geo and mobile space, although I will take up counterpoint to some of his suppositions on File Formats, Interfaces, OpenStreetMap and Augmented Reality.

File Formats and Interfaces

Geo is definitely becoming mainstream – everyone in my family has a PND, uses Google Maps, and are asking about various location sharing applications. In the next year we’ll see geo become part of the assumed infrastructure, like the timestamp on a post or article, the location will be embedded.

I don’t think TAG (Twitter, Apple Google), as James puts it, will be the only location sharing services. They, along with even more used Facebook, will definitely be the general public interface to location query and sharing – but just because of this reason alone they will have to be very generic, leaving room for specialized location based services to still thrive in niches. FourSquare offers ‘gaming’ or Flickr visual media, and others for music, drinking, sight-seeing, and house finding. They will leverage TAG, or at least TG.

Apple is like the Nintendo of consumer technology – more interested in providing an integrated, compelling experience, and privacy, before full open-ness and engaging with the developer or geek. They’ll still have API’s, but not something like OpenSocial, GeoRSS, or FireEagle integration.

The iPhone, and to lesser extent Android, have been revolutionizing mobile devices. They are truly providing windows into the rest of the web of data combined with the real world. It’s natural for geopatial tools to move into these interfaces, but like any good user experience it won’t be the same capabilities you find on a desktop or browser application. The utilities will be specialized for the small screens, finger inputs, and out-and-about tasks.

For file formats, the Shapefile, unfortunately, isn’t near EOL. Too many tools only speak shapefile, and there is numerous legacy data that is still only available in Shapefile. Sites like GeoCommons offer alternate formats for all the data, but that still won’t remove this basic format. Only when there is a truly open, license free, API to File GeoDatabases (FGDB), and every off the shelf tool can talk that API or Spatialite, will Shapefiles begin disappearing out.

GeoRSS and/or KML, on the other hand, will be in every service that does anything Geo. Looking at any iPhone App review that includes KML (or doesn’t) brings up this point. Near enough everyone has Google Earth on their desktop, and Google is making big pushes in the utilization of Google Earth Plugin for in-browser virtual globes.

Visualization Technologies

To date, we’ve been stuck with either Flash or JavaScript DOM magic (and yes, Silverlight is out there too) in order to do data and geospatial visualization in the browser. As I mentioned, Google has been pushing Google Earth Browser, but also more generally they released O3D, a modern incarnation of X3D, providing for more general capabilities for creating 3D browser experiences. VRML lives!

More recently, there has been a resurgence in vector graphics that don’t rely on proprietary technologies or additional plugins. SVG and Canvas support is pretty widely supported except in the infamous Internet Explorer (which I hear is still being used even today). Examples such as ProtoVis, Cartagen and Tom Carden’s experiments definitely demonstrate that SVG is just on the cusp of being able to do a majority of compelling visualizations capabilities.

Another driver for alternative visualization platforms is the drive to mobile device integration. I don’t see Apple allowing Adobe onto the iPhone anytime soon, and even Android doesn’t have support. What types of visualization make sense is still a very open question – but whatever they are will be done with something like SVG.

Geo Data Skirmishes

James suggests that OpenStreetMap “won’t dominate”. While it won’t dominate, I disagree that it won’t continue to be extremely successful.

Google has recently moved to gathering their own data. They still have a long way to go, with many, many errors in roads, areas, addresses, and businesses and they’re using the crowd to help clean it up. Google is in fact proving the crowd-sourced model. It will be successful. Google is doing it with Google’s data, so there is no positive external benefit to that work – so to the industry it just looks like another data provider. However, with this proven model OpenStreetMap will succeed since any effort built into OSM has a positive benefit to anyone else.

However, there is a major difference in the trajectory OpenStreetMap is taking in the United States compared with Europe and other regions. In most other countries, the governments had very draconian licensing and as such OpenStreetMap was creating data from blank areas – starting from scratch, and building a community of volunteers along the way.

By contrast, in the US a vast majority of the data is free, and becoming more available everyday under the new administration. Therefore the US has a broad coverage of decent data without having first built the user community. So the difficulty here is both in building out community, as well as engaging companies that can do the same thing on their own while retaining proprietary rights to the data.

What’s fascinating, and what signals the ultimate long term success of OpenStreetMap, is that US state, local, and federal government agencies themselves are engaging with OpenStreetMap. They are investigating how to put their data directly into OSM, and possibly even re-incorporate updates and modifications back to their own infrastructures. Some are even considering using OSM toolset as their infrastructure. OpenStreetMap is going through some growing pains with respect to licensing, maintenance, and community – but all necessary steps in moving from a small cadre of hackers to a global, public project.

As we see an increase in open government, specifically driven by the US Administration’s directives, as well as other initiatives such as INSPIRE, this embrace and utilization of open platforms, and repositories, for sharing, federation, and syncronization of data will increase.

And as for augmented reality, it won’t be as big as you think… yet.

OpenSearch descriptions for Flickr, GMaps, BBC

2009-12-08T04:23:23Z

Some of the sites I use most don’t support the very nice feature of OpenSearch discovery links. Among these are Flickr and Google Maps – so I first have to navigate to Flickr, then search rather than doing it straight from the browser quick search area.

Fortunately, someone has provided this for me. Just go to http://mvinetwork.co.uk/opensearch/ and click open your OpenSearch discovery and add whatever engines you want!

Hopefully more services that already can support OpenSearch templating add the necessary and simple descriptions and discovery links.

For extra credit, you could extend the OpenSearch definition for GMaps to support location name: http://maps.google.com/maps?q={searchTerms}&hnear={geo:locationString?} – and even Flickr supports (and was the model for) OpenSearch-Geo box search.

Who owns Arunachal Pradesh?

2009-12-04T13:12:23Z

I received an email the other day from a reader of my blog with a very interesting question:

I was looking at a certain area in North East part of India ( State called “Arunachal Pradesh”) which is integral part of India.

Both URL’s have different take. [Google Maps] shows it as Disputed Territory ( with Dash lines) and [Google Ditu] shows it altogether as part of China!

So, that got me unruffled and to question validity of both these sources. How does Ditu differ from Google maps? Whats association between the two and does Ditu has autonomy to change the boundary of the maps as per its wish.

Arunachal Pradesh is a border region between China and India – with 70% of the land being claimed by the Chinese as South Tibet. The border in question was decided in 1914 and called the McMahon Line, but never agreed upon by the Chinese. The Google Ditu vs. Google Map views.

Comparing Google Maps (background) with Google Ditu (foreground tinted red)

Territorial disputes are definitely not a new thing – however what is perhaps alarming is that there are two different representations of reality from the same vendor and data providers. So this is entirely a representational decision that is most likely driven by business and government pressures.

What’s particularly interesting here is that primarily these definitions of boundaries derive from the data providers. You can look in the bottom right corner for who the data providers are. For both versions the providers are the same: TerraMetrics, Mapabc, and Europa Technologies.

So it seems that the cartographic designers at Google Ditu have decided to represent it a certain way. Unfortunately, the map has no additional metadata. As broad consumption of maps increases, there is a commensurate interest in the why and what behind them. Who said these are the boundaries, when were they set, and why are they shown in this language?

And I don’t mean the long reams of unreadable metadata that are the current standards in the geospatial community, I mean human understandable descriptions of the various aspects of the data, while allowing additional discovery to deeper data.

One place that you can look at the data behind the source of the map is in OpenStreetMap. Arunachal Pradesh is shown similar to Google Maps version, and a user could optionally download the data to see the attributes, edit history and sources. Alternatively I can look in GeoCommons for the GADM Admin boundaries of India and see pertinent data on who provided the data, sources, and so on.

Boundary disputes in a bi-directional medium

The representation of boundaries is obviously a very contentious issue in mapping. Maps are perceived, and often do inform, territory. There is a long history of map representation being used to influence, coerce, and force land rights.

Unfortunately, even in a “Web2.0″ world of bi-directional sharing and collaboration, with maps we’re still often forced to accept a particular viewpoint. They have on-the-ground meaning and political impact. A well known example of this were the first “edit wars” in OpenStreetMap with the names of places in Cyprus. The resolution was to by default abide by the on the ground signage, but also store both versions and allow users to provide their own personalized perspective.

Understanding, awareness, and discussion about these issues is the reason for projects like OpenStreetMap or GeoCommons where you can download the information, build and share your own maps that represent your perspective.There

There isn’t an easy answer here – with companies such as Google there are obviously market, and government, forces that direct how to represent contentious issues. The best solution is to offer background, open data, and alternative perspectives. Without a voice, citizens are relegated to discussions by officials they may, or may not, have elected – and no meaningful way to illustrate their interpretation.

Geography Week and GIS Day at UVA

2009-11-17T16:42:09Z

This week is National Geography Awareness Week. Hopefully you’re celebrating it your own way by enjoying a map, thanking a cartographer, or even doing some mapping yourself! It’s clear that mapping and geo have entered the mainstream – everyone is engaging with maps through navigation systems, friend location finders, and virtual globes. The next step is to make people aware of the potential for them to personally engage with place and location for personal interests, business uses, and community building.

This Wednesday, I will be giving the GISDay plenary talk at the University of Virginia in Charlottesville. My talk, “Neogeography: from Tower to Town Hall” will discuss how the movement to broad, public engagement and collaboration, particularly around geographic contexts through web maps, mobile devices, and open data can build stronger communities, improved research, representative government and better livelihoods of people. (link to the UVA Calendar)

Around Washington, DC you can join the OpenStreetMapping party in Bethesda, MD on Saturday, or you can check out the large list other activities at GISVirginia. Whatever you do, spread the word and encourage people to go out and map something.

Want to be a GeoCommons Engineer?

2009-11-11T16:08:18Z

It’s time for the FortiusOne GeoCommons team to expand again – and we’re looking for an incredibly bright, hard working, and team oriented engineer to head up our operations team.

GeoCommons is unique among most web applications – it isn’t just deployed to the public web, but also to intranets, the cloud, and to the field. We have servers running in Jalabad, Afghanistan and Nairobi, Kenya, we help develop technology solutions within the Federal government and Intel, and work with Academia, disaster response, and major corporations.

Are you an engineer who likes playing with new technology and solving hard problems? Do you love writing Linux scripts that can deal with massively horizontally scaled servers or compressing systems to run on USB sticks? Do you have a passion for open data, open-source software, collaborative government, and cutting-edge technologies that help the world? An interest in mapping is obviously a plus.

Ping us through the blog, twitter, LinkedIn, email, or stop by our offices in Arlington VA to chat directly. And no, we don’t need any recruiters.

The need for clear data licenses

2009-10-23T18:10:07Z

There is clearly a movement to openly share data from numerous data sources: governments, organizations, Web sites, individuals, and devices. Users are more easily able to publish data through collaborative sites, or find and download data that they can use to remix, reapply, reuse, and extend. The trajectory of open data sharing and utilization parallels the development of open-source, where the potential magnified impact of open sharing and collaboration yields far great outcomes.

However, unlike the open source world, the legal and cultural frameworks in which to share data have not yet emerged. In code, there are a gamut of well known and widely used licenses: GPL, BSD, MIT, Apache, and more. While each has unique characteristics, their overall meaning and implications are easily understood by developers and comply with business operations that wish to use open-source software. The licenses are each unique, and can sometimes be confusing, yet there is a small vocabulary of regular licenses that developers can easily picked and choose.

In the media world, Creative Commons developed an ingenious mechanism of licenses with clear verbage and branding that makes it readily accessible by nearly anyone. With miscible license options such as “Share-Alike”, “No-Derivatives”, and “Non-Commercial”, media producers and consumers can clearly mark appropriate uses of their works. The impact is most clearly seen on content sharing sites such as Flickr and Slideshare where users can choose from a very small list of licenses to publish information, or search for information under certain terms.

Because of these attractive features of understandability, small set, and branding, Creative Commons is increasingly utilized to openly share data. However, the Creative Commons apply only to creative works: stories, songs, photographs, and other media and as such are not truly valid when applied to databases.

Instead, the current landscape of data licenses are all completely unique, incompatible, and difficult to understand. This situation is further complicated by the existing data business ecosystem that thrives on charging large amounts of money to write, verify, and mix unique data licenses and prescribe legal uses of multiple combined data sources.

The implications are that the situation persists and groups that would like to share, or use, open data are relegated to complex, and expensive, legal counsel, or must accept risk and hope they remain within compliance, or at least outside of notice.

There are currently only two potential solutions currently in development. Creative Commons has developed CC0, (”C, C, Zero”) – which essentially removes all copyright from a work or data. Flickr utilizes the CC0 license for their derived boundary datasets.

The other upcoming option is the Open Database License (ODbL), which is being put forth by the OpenStreetMap community in order to create the equivalent Creative Commons, By-Attribution, Share-Alike (CC-BY-SA) for databases.

However, even these two licenses have problems. CC0 is drastic in that it removes all copyright from the data, and so may not work with anything less than full, global release of data. Alternatively the ODbL is criticized for being “too left”, where there is an unclear potential that any utilization of data such as from OpenStreetMap would subsequently have to be released. This is similar to the GPL licenses, or “viral” licenses.

What is missing is a clear set of data applicable open licenses that would allow anyone to easily demarcate the terms of the data they are releasing, and provide confidence to data consumers that they are in compliance with the data rights. The effect will be to allow data to more easily and justifiably be made available as well as tools to interact with this data. It will also address the many questions around collective, or combined, databases and derivative works, such as when deriving vector data from satellite imagery.

A few weeks ago I spoke on a panel at the OGC Summit on Spatial Law and Policy, which is one effort to build a community of developers, companies, data providers, and legal experts to address just such a need. In particular times of disaster response illuminate the immediacy of clear data sharing, which was the focus of the panel, but also more long term use of such data, and derivative works such as in rebuilding and recovery after an event.

I’ve discussed the pitfalls of licensing and Creative-Commons style modules before, which raised some initial questions. But in the year since then nothing has really changed towards this broader vocabulary. Will Creative Commons merely become the de-facto, if non-applicable, licensing? For example, the Ordnance Survey just released data under Creative Commons-Attribution 2.0 UK. I would be interested to hear more about other efforts that are seeking to create a simple, clear set of data licenses. In addition, how else have you dealt with confusing, and complicated data licensing issues in building new datasets, applications, or use cases?

Temporary Mapping – Solar Decathlon

2009-10-13T12:52:22Z

This week on the DC National Mall there is the 2009 Solar Decathlon. It’s a contest between 20 student groups from around the world that build, on the mall, sustainable, energy efficient, and modern houses. The competition measures their efficiency, quality, resource usage, and design. It’s a one week miny village.

So of course, like any village, it needed to be mapped. I went down Saturday afternoon and captured the locations and names of all the buildings and paths that will be up for the week. These are then loaded into OpenStreetMap with start_date and end_date tags that notify the renderer when the features should be visible. It’s a similar model to how Burning Man is mapped year after year as it walks along the Black Rock Desert.

It’s ephemeral mapping – objects that exist in real place, but just for small slices of time. Important as any other building, yet typically relegated to flyers or verbal descriptions.

The fascinating part of projects like this is that OpenStreetMap allowed me to create a map that was useful and immediate. Within minutes of uploading the data, it was available as rendered tiles, vector data, and downloadable to GPS units and iPhones. People on the mall could immediately view the local map with this new information.

It’s a nice demonstration of how community projects like OpenStreetMap will continue to innovate faster, and more openly, then other ‘crowd-sourcing’ options.

Apple Geo

2009-10-02T15:28:03Z

There was a lot of buzz yesterday around the not-new, but recently renewed interest in, Placebase’s – and more specifically Jaron Waldman’s – joining Apple in their “Geo Team”.

Putting aside the question about whether Apple purchased Placebase, it’s more interesting and worthwhile to consider why Apple is interested in pulling in and working with technologists like Jaron that obviously demonstrate the ability to pull together components and build a compelling, unique mapping stack.

Apple technology has increasingly added location capabilities. Address Book, Mail, and iCal all detect addresses and provide links to maps. iPhoto and Aperture understand coordinate tags and can provide maps as well. CoreLocation on the iPhone, and now in Snow Leopard, allow any developer to get the location of the device via a cascading order of geolocation: GPS, Wifi, IP, etc. Apple themselves developed the “Google Maps” iPhone application – just utilizing the Google API for tiles, location and routing.

More recently, Apple has provided for “lost iPhone” tracking via MobileMe. Enterprising uses and developers have used this for friend and family tracking services.

Looking forward, it’s clear that Apple sees the important potential of location to support and augment almost all of their applications and platforms. Like any good business, the less dependent a company can be on third-party’s for core functionality, the better. Therefore, it makes sense that Apple would investigate ways to own and control this key component.

And beyond pure business and strategy, there is a lot to gain by Apple controlling it’s own location and mapping stack. Apple obviously focuses on providing exquisitely crafted experiences. This should permeate through their maps as well. Look at the maps to your local Apple store for an example of how the cartography can fit into the look and feel of the Apple.com store interface. This same customization can exist throughout their product line. Maps applications and API can provide customized interfaces and styling.

And consider that Apple can also build out a MobileMe friend finding and family tracking service. There is now an inherent trust in Apple tools: easy to use, virus free, great for kids. These translate over to trust in sharing my location through my phone to my private family sharing portal.

So in the end, what this signals is a major shift to provide broad, consumer facing compelling geospatial technologies in a well executed interface. Apple is already responsible for enabling location-based services to cross the chasm, and is inducing the broad emergence of augmented reality. It makes perfect sense for them to ensure they control and can craft the entire experience. I’m personally glad they have someone as expert as Jaron on the team.

OGC Mass Market working group discussion from Darmstadt

2009-09-30T19:18:08Z

In keeping with my history of trying to shed some external perspective on how the OGC works, such as live-blogging the OWS-5 kickoff,
Geospatial search summit, and Google’s libkml anouncement – I thought it would be interesting to cover the OGC Mass Market working group telecon that’s being held in Darmstadt, Germany.

OpenSearch-Geo

Pedro Goncalves presented his work taking the OpenSearch-Geo specification and forming into an OGC acceptable document. Pedro has been doing great work extending OpenSearch-Geo for accessing earth observation data. I also talked about how OGC services could be described within OpenSearch templating over a year ago in Atlanta.

Unfortunately the presentation and document is currently locked away behind OGC’s portal. Hopefully Pedro will release it publicly. In addition, it’s not clear within the OGC how to adopt such as suggested standard. It’s not part of the OGC and must go through various OGC architecture boards and discussions to be accepted potentaill as a whitepaper.

We wrote OpenSearch-Geo at WhereCamp 2007 and since then it has stayed as a draft standard with various uptake across projects. The adoption as a more formalized standard should have a very positive effect on its adoption.

GeoCommons supports 3 OpenSearch description documents, one each for Finder, Maker, and all of GeoCommons.

In the end, Pedro’s paper was accepted as a “discussion paper”. Hopefully we can push this forward at the next Technical Committee meeting in Mountain View in December – where DeWitt Clinton (OpenSearch original author) will hopefully pop in to push it forward.

The rest of the working group meeting discussed a potential GeoSMS format that ITRI from Taiwan is working on.

Unfortunately, we didn’t get to talking about the potential <geomap> HTML element ideas. There will be more discussion about that on the OGC Mass Market email list.

Service market for open data grows

2009-09-17T20:55:21Z

Via @cageyjames I saw the announcement that DeCarta is now providing their services using OpenStreetMap data. It’s tremendous to see more service providers building tools and businesses on top of open data.

It’s this transition from primarily ground-swell supported projects into industry supported efforts that makes them sustainabile. Linux went through the same growth as providers like RedHat and Canonical started putting full-time and dedicated resources to ensuring the continual growth and stability of the projects.

This is perhaps very interesting news to CloudMade, since it sits squarely in their market as well – and also Prioleau, the ~~CEO~~ former-CEO and now advisor of CloudMade was formerly a Vice President of Marketing at deCarta.

CloudMade and DeCarta tiles using OpenStreetMap data

The deCarta tiles show the copyright “CCBYSAODBL”, already including the forthcoming migration to the ODbL terms. There have been many companies that are interested in integrating OpenStreetMap into their products but concerned about the potentially unclear terms or permitted uses of the data that don’t inadvertently expose their proprietary data. Hopefully more moves like deCarta to utilizing OpenStreetMap in commercial products will provide clarity on the benefits and implications to other companies.

End of Summer Events

2009-09-16T20:24:39Z

It has been an incredibly busy and interesting summer in DC and the geo-community. In the past few months I spoke at Reboot11 in Copenhagen, was wowed by the progress of the OpenStreetMap community, tools, and data at State of the Map, and did a little preparatory GIS tête-à-tête at GeoWeb.

This was perhaps most summarized by the experience at Gov2.0 Summit. The conference, held in DC but led by O’Reilly and TechWeb saw the convergence of technorati with government agencies, and beltway consultants. There was Twitter co-founder Jack Dorsey sitting next to Vint Cerf – old and new, talking about government leveraging social and internet tools – and then the White House’s Macon Phillips, Aneesh Chopra and Vivek Kundra talking about revolutions within the agencies themselves.

I was fortunate to share my thoughts and our work with various agencies and their ability to leverage location and geospatial tools as a common collaboration point between citizens and government agencies and municipalities. Besides, it’s always fun to follow Jack Dangermond on stage.

Upcoming Events

Of course, summer isn’t over. Next Monday I am speaking about geospatial search at the EPA Search Summit here in DC. Then heading over to UK to AGI Geocommunity’09 to really discuss the current state, and possible futures of geospatial technologies.

In general, AGI Geocommunity looks like a great lineup of talks. The different perspectives of problems (UK postcodes & MasterMap anyone?) and solutions (OpenStreetMap did emerge from the UK) is very enlightening.

While in England – I just may hop up to Oxford Geek night on Saturday, September 26 and try and foment some more interest in CrisisCommons and CrisisCampUK. Let me know if you’re around – it would be great to meetup.

GeoWeb Standards – Discoverability

2009-08-28T07:58:39Z

We have a rich history of geography, cartography and GIS that is currently tucked away in top drawers, intranets, and repositories that may not stay online when we most need the data. How do we expose these huge troves of data in a way that can be utilized across various domains. The GeoWeb is all part of the same web, semantic, sensor, social, (and interplanetary). So it is vital at the GeoWeb align itself with the web and the multitude of sources and endpoints that the web is reaching into.

There are many possible solutions, and a few that are within easy grasp that we can build our tools to encompass, and develop practices that encourage utilization of these solutions while still moving forward onto better ones as the GeoWeb matures. So we’ll take a few articles to look at specific solutions.

Discoverability

Perhaps the most prevalent issue, and the one that is most easily addressable, is the findability and discovery of geodata on the web. Mano Marks reflected this same sentiment in his blog post on standards.

In thinking about discoverability, there are several primary use cases to consider: Machine crawling, Human discovery, and Tool discovery. Providing data via just a single mechanism means that it doesn’t get utilized and consumed to it’s potential and so somewhere along the chain of utilization it will be a burden to actually incorporate into a workflow.

Think of the machines

Machine crawling is the ability for any spider to walk links, find data, metadata, and formats automatically. It’s what Google, or GeoNetwork does to find and register data sources.

There was recently a discussion on auto-discovery in the GeoWeb suggesting the use of robots.txt, sitemaps, or embedded META tags in HTML pages.

Consider how a spider would get to a site: it follows a link to a geospatial portal from some blog, resource, or directly entered as a good place to get data. It does a GET request on the root homepage, “/” which most likely returns the index.html equivalent. The program then parses through that for links or additional information.

If the spider knows about them, then it may ask for a sitemap.xml or robots.txt. But nothing in the original page request noted that this potentially very complete listing of data was there. This problem is the equivalent of an application having to know that it needs to ask for a GetCapabilities or other method to even discover what is available. Too much implicit knowledge of the specification is required for a program to easily discover new data and services.

What the program does see are these links that can contain information such as a link to a list of available resources. The simplest is a link to the Atom or RSS feed that can simply be a paginated list of all the resources available to the application. Within Atom, there is then the ability to link to various representations of that data in different formats. So applications are able to take the most appropriate format based on what they can consume.

Several years ago I first proposed how KML and GeoRSS could easily support one another via cross-links and with HTML documents. Atom has very nice rel and type attributes that allow for linking to all sorts of different representations. You can even link to OGC services like WMS and WFS using atom links.

Of particular interest here are looking at the currently approved list of Atom link relation types that provide basic semantics for telling you how what this link means. Is it another page? just related? It’s a limited set, but one that covers an approachable majority case for developers to begin using.

For example, mechanisms like OpenSearch, specified in a rel="search", simply notify the application that here is a service that it can query to get at additional resources. And with OpenSearch-Geo, a geoweb crawler can query information within a specific location or bounding area.

Humans need data too

Crawlers are great, they provide a say to pull together information into various other sites and tools to provide customized interfaces to users. However, within any site or tool, how should we expose geodata in a way that humans can easily use for whatever purposes the may have.

Again, links have become a very well understood concept on the Web. That underlined blue line states “beyond me lies an unspecified amount of information about this topic“. However, these links typically imply that they will open another human readable HTML page in the browser. A problem caused by links to media such as geospatial data is that the content behind a link may not be just text, it could be an image, audio, movie, KML, database, or a service. Clicking on that link relies on the browser interpreting the MIME-type (remember the point about how vital mime-types are?) and opening the application the user has specified, or left as default.

So consider what this means for generic media. Clicking open a link to an image probably just opens the image in your browser, or opening a movie loads an embedded video player. Geodata browsers, however, probably doesn’t have the same install base as say, Quicktime. Except perhaps GoogleEarth. The Web has become much more comfortable with clicking on a KML link and seeing Google Earth open up and show the data on a globe.

But something very vital often exists with a link to KML data – a recognizable icon that notifies the user (as they learn) that it is a file that will open in Google Earth, or another KML viewer. This is the same as the very widely used RSS icon.

I discussed this idea before about the geotag icon showing various other formats – and now sites like Data.gov actually show the various data format options.

So what we need for GeoWeb standards are some visual representation to people that they are can clink on this link and open a spatial relational database, or an OGC service, and perhaps have some confidence that there is an application that will provide them a useful way to access the data. (and I’m still waiting for Sean Gillies’ ISO and Dublin Core icons)

Of course, we should also employ emergent interfaces that show users the type of data links that are appropriate for them based on their profile or registered MIME-type handlers.

Man-Machine hybrids

So we have discovery links for machine crawlers to register and harvest geodata, and links for humans to click on to follow to data and within data. However, this can easily become overwhelming to need to click through to every link. Imagine if browsing Flickr through lynx.

Browsers already do a lot to assist users in finding relevant extra pieces of data in a page. RSS autodisovery links show up in URL bars notifying our feed readers that we can subscribe to this page. OpenSearch allows someone to embed this search into their browsers (most of them at least) to easily search the repository later.

The decreasing cost of links

These various approaches for different needs and use cases are all very well aligned. They don’t rely on additional external files that we need to make sure stay up to date or that tools are built to just know that the file can be found at a pre-defined location. Links cost next to nothing, mostly measured in bandwidth sizes, but provide a wealth of accessibility and discovery of geospatial data. Especially data in formats that make sense depending on the tools and use cases for different problems.

Of course, links alone don’t address all the needs of the evolving GeoWeb, they merely provide for the integration of geospatial data with the rest of the web. An important, necessary, but not entirely sufficient first step. We need to consider the actual uses and interfaces of these standards, archival, synchronization, conflation and more.

GeoWeb Standards – Where we need to go

2009-08-28T07:58:34Z

In previous articles on the status of the GeoWeb I highlighted the myriad of options and problems with current GeoWeb standards and interfaces. Overall, it’s clear that the practice of geospatial data publication and sharing in a web oriented way is still very nascent but getting better at the same time it becomes more mainstream. More data is being created and published in web-oriented ways that make it more consumable and usable.

Too often standards and tools are being by domain experts and technologists that lead to overly complex, and irrelevant formats that become a burden and introduce as many problems as they are trying to solve.

What they’re often not considering are the end user experiences. Who are the users, what are they trying to achieve, and how can these formats make for better, and easier utilization of these tools.

Granted, there are expert users. People who really want to make intricately related, projected, spatially and spectrally bounded queries into data and utilize them in advanced analytics engines. But these are not the majority and they’re not what is driving the long-term demand on the GeoWeb (you can use ‘long-tail’ here if you would like). Who are the users that want to engage with this information on a daily basis in their personal lives, businesses, family, safety, governance, and goals.

Grassroots is an option

I’m a very big fan of grassroots organization and emergent structures. The needs tend to grow from real demand, and solutions are built through actual demonstrated benefit and impact. They are agile, evolutionary, and garner broad support amongst users and developers. These are all aspects that are beneficial to achieving standards that meet the needs of end users and provide good experiences.

However, it is not the only solution. Grassroots tends to look at the immediate needs and may not incorporate more distant issues and expected needs. They seek for broad appeal, and “good enough” rather than totally encompassing all potential aspects of all interested domains. Top-down, industry derived, committee driven standards provide more directed needs and objectives that can serve different types of users.

So the solution is a hybrid – where grassroots solutions are encouraged as demonstrators and emergent needs – that are then accepted and supported by more formal organizations.

Conversation is required

But we also need to open up the conversation beyond just technologists and experts. We need to be engaging and understanding users – and not merely from the “how do I sell them more of my coffee”, but “what can I do to make their lives better”? And actually asking and engaging with them in dialogues.

This technique of user stories, and engagement is not new or unused. However it appears to be missing from the GeoWeb standards developments. We’ve been designing standards for ourselves first, and then foisting these upon others. Instead, we need to understand their needs and issues, and then apply our expert knowledge in how to approach solutions properly.

Other articles in the GeoWeb Standards series:

Institute for Enabling Geospatial Scholarship at UVA

2009-08-25T14:22:49Z

This November, I’ll be a faculty member at the UVA Institute for Enabling Geospatial Scholarship along with several other well known geo hackers.

We’ll be holding a series of talks on software tools, data formats, techniques, and scholarship of geospatial data. The institute is accepting applications for attendees until September 1 – so you can apply through the UVA Scholar’s Lab.

Through the generosity of the National Endowment for the Humanities, the Scholars’ Lab will host a three-track Institute for Enabling Geospatial Scholarship at the University of Virginia Library in November 2009 and May 2010. This Institute will bring scholars, cultural heritage professionals, and software developers together to support and develop geospatial projects and methods in the digital humanities. The NEH’s Institutes for Advanced Topics in the Digital Humanities program will support travel and lodging for 40 attendees as well as Institute faculty members. Dedicated funding is available for graduate students as well as faculty attendees.

You can read more on Bethanie’s blog
In addition, to cap off the institute, I will be giving the GIS Day Plenary talk on Wednesday, November 18 in Charlottesville, VA. The event will be open to the public.

It’s truly an honor to be teaching at my alma mater – albeit in quite a different discipline than the one I learned while attending.

SXSW Panel – Time + Social + Location.

2009-08-25T00:51:21Z

Voting is now open for next spring’s South-by-Southwest (SXSW) Interactive conference, and I was kindly asked by Josh Babetski to be on his panel SXSW Panel: Time + Social + Location.

As more devices become location aware, social uses will continue to evolve beyond just who and what, to WHEN. Adding the temporal dimension creates new opportunities for social interaction. Learn about ways to leverage and use technology to add features at the intersection of temporal, social, and location.

The panel is quite a great group – from the inceptive MapQuest that led the way in internet mapping, to the more modern social location networks of Brightkite and the recent Twitter announced location support.

We’re now carrying the equivalent of super-computers in our pockets that have near ubiquitous connectivity, location, media capturing, and sensors. We can easily lookup businesses, cabs, and directions. But what else is next? Is it really just people checking for the closest coffee shop or hotest hookup?

Or do people want to ask “The Big Here” questions: to inquire about their location and context – the history, environment, perception, and culture that surrounds them or guides them through place. Will we have tricorders, or will mobile devices be relegated to toy devices and glorified media viewers?

This panel includes the foremost implementers and leaders in the field that are building real tools that are used by thousands and millions of people around the world. What have we learned, and where are the next steps leading us?

SXSW panels work by people voting. So please vote up the panel. Last year I moderated one of the only location-based panels, but fortunately this year there is a much wider aspect of geospatial discussions and presentations. So while you’re at it, check out the other good panels.

Afghanistan Election Monitoring

2009-08-20T13:14:45Z

Today the democratic elections are occurring in Afghanistan. There is a FrontLine SMS on the ground pulling in data into Alive in Afghanistan, which is running Ushahidi.

At FortiusOne, we’re pulling together demographic, violence, and polling information into a dashboard of maps.

Estimated Number of Voters by Polling Location vs Average Risk Level of Polling Centers, 2009 Afghanistan Presidential Election, Eastern Region of Afghanistan (View full atlas)

You can read more about it on USAID’s Global Development Commons, or how we’re partnering with numerous other projects to build out solutions like this for deploying to the field.

The compelling example here is the combination of demographic information with contextually relevant information (polling locations), realtime feeds of data, and user-generated, on the ground observations. Together they are helping provide a common picture into the complex relationships, consequences, and potential outcomes of the election might look like.

Open Data Standards don’t apply to the Military?

2009-08-12T14:36:56Z

Last night I came upon a new posting at FedBizOps for the US Air Force Academy’s ESRI Software License Renewal. The solicitation is a sole-source justification for license renewal of ESRI software for $25 million USD.

While government procurement makes things like sole-sourcing common as a mechanism to just renew license – it is really the supporting justifications for this sole-sourcing that are disconcerting.

From the solicitiation: “ESRI is the only source that can satisfy the needs of the government for the following reasons.”

The Geospatial Information System section within the 10th Civil Engineering squadron has been using ESRI software since their initial development eight years ago.
All of the past and present mapping and client specification have been developed using ESRI software products.
The Dean of Faculty’s Geography department also uses the ESRI academic site license for teaching all Geographical Information System coursework to the cadets.
Software standardization between the 10th CES, DFEG, and the entire USAFA is extremely critical.
Compatibility allows GIS data sharing between all agencies on the USAFA will continue to support GIS development in the future.
Award of this contract to another contractor would jeopardize the performance of our mission by making all of the existing GIS data non-usable.

Which of those reasons are legitimate to the missions of the government and defense, and which are indicative of a more endemic problem of vendor lockin?

The points of the critical nature of compatibility are very important. Data and information must freely flow between sources, analysts, consumers, observers, and archiving. In addition, there are definitely costs to retraining and maintenance that affects changing or introducing new software.

However, the majority of reasons provided by the solicitation point to legacy decisions, old implementations, academic education of specific vendors (definitely not uncommon), and the bold closing statement that the data from these software packages is not usable in any other tools.

It’s that last particular point that should be the most disturbing to the administration. Apparently all geospatial data being developed and utilized by the USAFA would be unusable without a sole software vendor. This causes concern over broader interoperability with other agencies and organizations, access to important national information, and archivability and retrievability.

The fault here isn’t on ESRI. They offer an interoperability suite that supports OGC, ISO, and other standards that agencies could utilize. The fault lies with the government contracting that justifies this type of reasoning of renewal and continuation because of single-vendor lockin. There is little excuse that open, compatible interfaces should be part of such a large contract.

I wonder when software licenses and interoperability spending will show up in the USASpending IT Dashboard?