It’s been a long (yet worthwhile!) journey, and we couldn’t be happier to continue our efforts by joining OpenGov. Not only do they share our values, mission and ambition to help create the 21st Century connected government we expect and deserve, combining forces with OpenGov will greatly accelerate and amplify our shared ambition. As Zac, Sami and I surmised when we closed the deal: “1 + 1 = 11!”

Our GovTech adventure started all the way back in 2010. Back then, I headed up the Knowledge Engineering Practice of a small consulting firm when the siren call of digital public service beckoned. Five years later — after winning NYCBigApps twice; the NYCpedia and OpenData.city experiments; becoming a CKAN specialist running the Open Data portals of some of America’s largest cities; and launching CivicDashboards last October – Sami and I knew we were getting traction.

We were getting approached by multiple governments interested in our #BuildWithNotFor, open-source, standards-based solution, and we also got the attention of the OpenGov team.

Shortly after successfully partnering on a strategic project early this year, our conversations quickly turned from collaboration to acquisition.

After spending a lot of time (12 hour days were the norm) planning, brainstorming, designing and carousing with the entire OpenGov team at their Silicon Valley headquarters and in New York, we’re already executing on our shared vision.

The future is bright! And the future is OpenGov!

Sami and Joel

P.S.
Don’t forget to check out the FAQ and Zac’s blog post…

But as a company born of open data, we’ve spent the past four years not only deploying but hacking on open data portals of all kinds.  Actually, all our early experiments – from NYCDataWeb, to NYCFacets, to NYCpedia, were attempts at improving data portals – by linking, contextualizing and humanizing data.

Because as Mark Headd points out, we know we can do a lot better.

So instead of railing against the state of open data, we decided to build CivicDashboards.

A Data Portal is just One Step of Many

A data portal does not an open data program make.  If that was the case, then you can just install CKAN Multisite, or even set it up as a blog post and be done with it.

It involves far more – building connections to data silos (ETL); enriching, sanitizing, linking and contextualizing the published data; building knowledge products with actionable information from the data.

And that’s just the tip of the technical iceberg – there’s far harder stuff.  Marshalling the political will to publish open data; building and funding teams to steward open data innovation inside government; getting buy-in from the various custodians of data; empowering data custodians with the incentives, tools and training to proactively contribute data; all while navigating the broken government procurement process that effectively discourages innovation and the requisite experimentation that comes with it.

With CivicDashboards, we aimed to address all these:

Open Data with Open Source

Imagine if the Web was not made available freely.  Would we have all the modern web-based conveniences, businesses and business models we have now?  Or will we still have walled gardens like Prodigy, Compuserve, AOL and Minitel?  Will Wikipedia, Google, Facebook and smartphones be around?  Will the whole concept of “the Cloud” even exists?

We believe that Open Data will only realize its full potential if the base infrastructure that supports it is also Open.  That’s why we chose CKAN as a core component of CivicDashboards.

Already, CKAN powers the largest data portals in the world, and it has a vibrant, active ecosystem, with hundreds of extensions developed by the community, all innovating , collaborating and competing with each other without needing to sign a license agreement, or asking for permission.

Because having an API is just not enough.  Having full open source access gives you the freedom to change everything and anything – the freedom to integrate best-of-breed tools, to use different authentication schemes, to create and share custom schemas, to store datasets in third-party cloud infrastructure, to create API-compatible solutions like DKAN, to address the gaps that Mark pointed out, and yes, to install CKAN on-premise yourself without signing a contract.

To stand on the shoulders of the global CKAN community as we have with CivicDashboards.

GovTech vs CivicTech

We also realized that Ontodia was primarily in the business of GovTech, not CivicTech – the difference being GovTech is mission-critical government infrastructure, and CivicTech, is the subsuming wider field of technology applied to civic problems.

One perennial criticism of civictech is “solutionism” – especially in the realm of hackathons and app challenges.  As a product of one, we know first hand that a lot of governments hesitate to deploy “apps” from these events largely because of sustainability.

Is there a company behind this app?  Who else is using it? Does it conform to our mission-critical requirements? Can we customize it to support all our citizens – will it be multi-lingual, accessible and mobile-friendly? Is it backed by a Service Level Agreement? Will the technology be around next year? Five years from now?  Are there multiple vendors supporting the technology?  Or are we locked in and have to sole-source all follow-up work?

Open Source!?! What is the quality of the code? Who else is contributing to the project? Our developers are busy with production issues and cannot be expected to compile the latest version and manually apply patches. What is my Total Cost of Ownership when I have to hire an expensive development team to maintain our deployment?

These questions have been asked before in another realm of mission-critical open source – Linux.

So similar to what RedHat did for Linux, Ontodia aims to do for CKAN.

#BuildWithNotFor also applies to GovTech vendors

Open Data is a Process.  Ultimately, the success of Open Data is not going to be measured by the number of high quality, linked, machine-readable datasets in a portal, nor the number of Civic apps using that data.

As Stephen Goldsmith and Susan Crawford put it in the conclusion of their book – The Responsive City.

“The real payoff will come when technology changes legacy processes for good to create truly data-smart and responsive cities.”

Great open data programs are only possible when the agencies publishing the data implements processes and enabling mechanisms that ensure the data stays current, relevant and responsive to its constituent’s requirements.  And those constituents include not only citizens and businesses, but other agencies as well, who surprisingly, often do not have ready access to their partner agency data.  

As Anthea Watson Strong puts it on her screed – “Hey Uncle Sam, Eat Your Own Dogfood”.

“The real problem behind our data quality issues, is that the people who have the power to fix the data, don’t have an incentive to understand the problem or improve it. Government officials are lovely people who work hard in under-resourced offices. Although many of them believe deeply in transparency and citizen engagement, these portals tend to generate additional burdens that get in the way of their primary functions. When data is stale or data is inaccurate, someone has to take the time to update it or fix it. It is difficult for any one group to see beyond the limits of their own projects. The real trick is to align incentives. What we actually need, is for Uncle Sam to start dogfooding his own open data.” (highlighting ours)

That’s why we created our “Analytics-as-a-Service.”  Because after helping a client launch a data portal, we wanted to build with, not for our clients as they embark on their open data journey.  

Not because we fancy ourselves as experts with ready-made, off-the-shelf solutions that we want to upsell to them, but because:

• we want to learn about the local issues, so we can become a better partner and help them employ the Data Scientific Method and formulate the questions that can be answered with their data
• we want our clients to answer low hanging questions faster by helping them focus on the hard task of opening data, while we focus on the secondary, though necessary task of data wrangling
• so we can help them create a showcase of open data powered answers, to generate buy-in from all stakeholders.
• so we can build-up each other’s capacity.  For us, to develop the product driven by real-world use cases; for our clients, so they can eventually wrangle data themselves instead of us doing all the wrangling for them.

To accelerate success, we maintain a Solution Template Library – pre-built Solution Templates including Crime Maps, Economic Activity Dashboards, Snowplow Trackers and open source projects like Citygram and Mapzen.

These Solution Templates often include Solutions developed by someone else, cherry-picked from other successful civic tech projects and best-of-breed commercial tools like CartoDB, Accela, Tableau and Necto, solving a similar problem.

And when we do develop or extend a new extension, as we’ve done with our CKAN Discourse Extension, the CitySDK CKAN Connector, and the OpenRefine CKAN connector, we add it to the Library, and when it makes sense, open source it as well so everybody wins.

Answering Questions often require 3rd-party Data

Ontodia has already compiled Key Place Indicators™(KPI) for all 3,000+ counties and 35,000+ townships/municipalities in the US, compiling maps and data from various high-value trusted sources like the U.S. Census, the Department of Labor Statistics, the FBI, etc. going back several years.  Clients can use these KPIs and the data behind it to compute their own KPIs.

And as we’ve done with NYCpedia and data.beta.nyc, we can also wrangle data from other non-governmental public sources – like job postings in Indeed, hyperlocal news from DNAinfo, and curated, released FOILed data from news organizations like WNYC and the New York Times.

And with our Up-to-Data™ subscriptions, our clients can have continuing, up-to-date access to these datastreams.  They can even commission custom feeds that further expands the datastream library.

So this is why we built CivicDashboards…

With the CivicDashboard suite, we offer a different path towards 21st Century government, building on the solid foundation of CKAN – the best open source data portal platform – we added all the other missing pieces to start putting open data to work.

Our novel approach of combining an open source data portal not only with enterprise support, but with performance management, analytics and data subscriptions, allow us the freedom to tap best-of-breed tools/techniques, and to #BuildWithNotFor our clients, partners and the CKAN community, all without artificially locking in our clients to a proprietary technology or the product roadmap of a single vendor.

We call this approach #OpenInfrastructure – Open Data with Open Source with Confidence.

“Paperless Office!?!” Yep! We just moved it all to the hard drive! Image by Alexandre Duret-Lutz

A lot of government data is published as PDFs, which is machine-renderable, not machine-readable.

When it comes to Tim Berners-Lee’s 5 star deployment scheme for Open Data, PDF is at the bottom of the heap.  And as we found out from last year’s World Bank study, hardly anybody reads PDFs!

Still, PDFs are a very useful, portable way of sharing information, so I’m not saying we should not use them.

To the contrary, I say we extend it!  What if we leverage modern PDF features to embed machine-readable data when its referred to in a PDF?  Tabular data is top of mind, but with the ISO PDF/A-3 standard, we can embed any file – JSON, XML, XLSX, etc!

Better yet, if the publisher doesn’t want filesizes to balloon, you can also optionally use a link to the associated data portal/webserver where the data is hosted.

So on this day of Tabula’s 1.0 release, I present our initial PDDF (Portable Document & Data Format) experiment.

Using Tabula to extract the CSVs, I attached the extracted CSVs back using Adobe Acrobat Pro and used Chris Whong’s Get the Data button to link to the CSVs which I also posted on the data portal as XLSX files.

If you want to extract the CSVs directly from the PDF viewer above, click on the 1) the sidebar button, then the 2) attachment button.

We can even give a visual hint to the user that they’re viewing a PDDF, by tweaking PDF.js – the cross-platform PDF-rendering technology embedded in Chrome, Firefox and CKAN.

What do you think?  Is this a pragmatic way for publishers to have their nice, pretty PDFs, and allow us to extract machine-readable data from it too?

Detail of “Blah, blah, blah” by Louise Campbell

I was getting ribbed by some folks that all that talk about open-sourcing ckanext-discourse WAS cheap, because we sat on it for a while.

Well, we finally got around to cleaning up the code, and even improving it a bit after some initial feedback.  https://github.com/ontodia/ckanext-discourse

Check it out in action at talk.beta.nyc and data.beta.nyc!

Open Data + Open Conversations = Community Insights!

P.S.  Hopefully, somebody gets around to creating an optimized CKAN Onebox embed, and a Discourse spectrogram poll type…

Our focus on context here at Ontodia, is not just about making it easy to create maps and promoting conversations around data – its all towards our ultimate goal of Humanizing Open Data.

Behind each dataset, there’s a story.  Building and construction permits tell stories of urban renewal and gentrification.  311 heating complaints tell stories about delinquent landlords and old infrastructure.  And Vision Zero data records the tragedy of preventable accidents that result in fatalities.

Now that WNYC released the data behind Mean Streets, we want to show the value of community-curated opendata – as we believe open data is not just government datasets about us, for us, and paid by us.

It’s also data published and curated by us.

Because WNYC did not just do a visualization with the raw, anonymous Vision Zero data in the City’s portal and stopped there.  They did the hard work of investigative journalism to reveal the story behind each incident.  Was it a child? What was the reporting from the local papers? What did each location look like in Google Street View?

Already, we created several maps just hours after WNYC released the data.  But we felt that those maps still failed to humanize the data.  So today, we’re releasing another map to focus on the Children of Vision Zero 2014.

It’s a shame the Mean Streets project “was designed to run just thru 2014.”  Wouldn’t it be awesome if the City partnered with WNYC and Vision Zero View had WNYC’s Mean Streets enriched data?

When we first started Ontodia, we were really keen on applying pragmatic semantic web technologies to open data.  “Onto” comes from ontology, and “dia”, from encyclopedia.  We thought (and still do) that “Open Data is the killer app of Linked Data and vice-versa.”

NYCpedia was actually our third attempt at doing that (NYCFacets was our second,  and NYCDataWeb, the first).  And on our third attempt, we found what we thought was a practical way to link open data, and that was with LOCATION.

Location, Location, Location!

It’s the mantra of every real estate agent.  You may have the best property, but if its not in the right place, it doesn’t matter!  With just a property’s address, a real estate agent can find out a lot of information culled from hundreds of data sources.  What are the latest sales in the area? Is it near entertainment or shopping?  In the right school district?  Accessible to public transportation?  Low crime rate? Etc, etc.

We found that this also applied to Open Data!  Because when we first attempted to link data through identifiers, we found that it was a Sisyphean task!  The datasets were simply too “dirty” and raw to link across datasources to create true linked data.  And what can a small startup like ours do to minimize, never mind, eliminate the endless data-wrangling that is behind every Big Data project!?

But as we learned with NYCpedia,  just enough context is good enough, and location is the ultimate link across disparate data sources!  I don’t need to know that Building ID Number 1009745, is tax block 579 lot 70,  is 137 Varick Street in Manhattan, NY.

All I need to know is the address, and with it, I can start linking data across systems – from NYC’s Building Identification System, property tax information from it Department of Finance, job listings from Indeed, 311 complaints, Foursquare check-ins, Flickr pictures taken in the area, Zillow property valuations, companies in Crunchbase, etc. etc.

CartoDB

We first discovered CartoDB in 2012 when we met co-founder – Javier de la Torre at NYCBigApps.  Since then, we’ve been using it and love how it made creating beautiful data-driven maps, and telling stories around data so much easier.  We also love that they share the same technologies/values as they democratize geospatial software – using open source software delivered as a service over the cloud, using PostgreSQL – much the same way we’re democratizing open data with OpenData.city.

There’s simply no better “GIS in the Cloud” solution.

So when we embarked on our fourth attempt with OpenData.city, we knew we were going to incorporate it as a fundamental component of the hosted service.

CKAN + CartoDB > Data Portal

We built OpenData.city on CKAN – the leading open source data portal platform, and the technology behind the biggest data portals in the world.  Out of the box, it already has some advanced geospatial capabilities to allow you to preview, search and organize geospatial data.  However, those capabilities were largely limited to viewing geospatial data, and more advanced capabilities required access to specialized GIS software.  There were also constraints in the geospatial file formats it recognized, and there were some performance issues with large files.

But with the Resource Views feature in the upcoming version 2.3 release of CKAN, it created an opportunity for us to integrate CartoDB.  As both are open-source, it allowed us to integrate the two seamlessly.  Taking our CKAN hosting offering from just being a data portal solution – an online catalog for viewing and downloading  datasets; to where data publishers can actually start to contextualize, analyze and tell stories with geospatial data.

And similar to our Discourse integration announced earlier this week, this allows our customers to quickly reap the benefits of opendata.  Here it is in action at data.beta.nyc.

And our CartoDB integration isn’t only skin-deep. Datasets can be set to transparently synchronize with CKAN with full access to the CartoDB platform – do SQL joins across datasets, add layers, customize and stylize map elements, georeference, map wizards, the works!  We even linked any map generated from the portal back to the dataset (and if there’s a related Discourse instance, to related discussion thread as well).

And there’s more!

We’re unveiling a few more things as we ramp up OpenData.city.  Tomorrow, during International Open Data Day and the first day of Code Across, we’re unveiling the updated Open Refine CKAN extension!

Attribution: the “Open Data is Geospatial Data Venn Diagram” was inspired by Andrew Nicklin‘s presentation at GISMO NYC November 18, 2014, “GeoData Startup Showcase.”

OpenData.city is not just about democratizing opendata in terms of affordability.  Its also about  widening the opendata community to something that regular citizens, businesses and students can understand and use in their daily lives.

And as we learned when we built NYCpedia in 2013, its all about creating context.  People don’t care about raw Big Open Data – thousands of datasets in a portal.  They care about Small Data – actionable information you can tell me now.

I’m not saying that data portals are not required.   To the contrary!  We’re saying that data portals should do more to contextualize the data – in terms of location, time, uses and conversations.

Conversational Context

As StackExchange has shown us, people prefer Q&A to reading dense reference material, reading user manuals or going through a knowledgebase.

Because “wisdom is in a crowd!”  A crowd asking the same questions, caring about the same issues.  The answers, as StackOverflow demonstrates, does not necessarily flow from the publisher, most of it actually come from the users!

In NYC’s vibrant CivicTech community focused around BetaNYC for instance, members of the community started to realize that people were asking the same questions.  And sometimes, new members hesitate to participate in the conversation for fear of making a fool of themselves.   Much the same way software developers “lurk” in Q&A forums looking up answers to previously answered questions,  a space was needed where all these discussions from the disparate threads, online and offline (mailing lists, meetups, google groups, townhall meetings, etc.) can be pulled together.

Discourse

Discourse is the hottest thing in forum software at the moment.  In the short time since the open source project was launched early 2013, it has redefined internet forums – whose structure have essentially remained unchanged since the early 90s.  It was appropriately founded by Jeff Atwood – Stack Overflow co-founder.

Discourse is forum software created  using today’s technologies and users in mind.  It’s mobile-responsive, it works through email, it supports just-in-time loading of long conversations, and the list goes on.

Its already running large community forums – twittercommunity.com, discuss.newrelic.com and discourse.mozilla-community.org, to name just a few.

And now talk.beta.nyc.

talk.beta.nyc

BetaNYC is the largest CfA brigade in the country.  When Noel Hidalgo (@noneck) asked late last year if Ontodia is up to spinning up a community data portal, we jumped at the opportunity!

And when he asked if we can integrate Discourse early this year, we immediately got to work creating a CKAN plugin.

BetaNYC was in the process of consolidating its various mailing lists into a Discourse instance, and instead of using ckanext-disqus, which was the only available CKAN commenting plugin at the time, we worked with Volkan Unsal (@picardo), who was leading the consolidation effort, to use Discourse instead.

In the open data portals we use and have spun up, we noticed that hardly anybody was using the Disqus-based system.  We realized that this was because it was seen more as a complaint mechanism, not as a conversational medium.

It wasn’t a way to share expertise; asks somewhat tangential questions; or start discussions that may be informed by a dataset, but could be on a larger topic altogether.

With our CKAN-Discourse integration that we’re now beta-testing at beta.nyc, we hope to change that and enable data-driven conversations.

Starting today, it will now be an available option for OpenData.city customers.  Later this spring, we aim to open-source it and give back to the CKAN community.

More Context Enablers

And this is just the first value-added addition to our CKAN hosting service.  Leading up to Code Across this weekend, we’ll be unveiling new services/plugins every day.  Tomorrow, we’ll go into more detail about another way we’re contextualizing opendata with our CartoDB integration.

attribution: Let’s Talk image courtesy of Flickr User Ron Mader – CC By 2.0

Today, we’re thrilled to announce  that OpenData.city is live!

As a company born out of Open Data, we’ve seen how fast the field has evolved.  When I first entered NYCBigApps back in 2010, NYC’s DataMine was primarily a list of 350 files being served out of a web server – no API, files in various formats, little metadata.

Since then, NYC has become the top Municipal Open Data City in the world – with its Landmark Open Data Law; NYCBigApps – the world’s largest, longest-running municipal Open Data Challenge; a vibrant civic tech community helmed by BetaNYC – the largest civic hacker community, more than 2100 strong; a tech-savvy administration and City Council; CUSP; a CIO, a CTO and a CAO; and now CivicHall.org – a community center dedicated to Civic Tech, managed by the same team that runs Personal Democracy Forum – the annual must-go confab for the Civic Tech community.

Even with all this – we always felt that something essential was still missing.  Even NYCFacets, which won us the NYCBigApps Grand Prize back in 2012, was our early attempt in closing what we still think is a gap in current open data portals – Context.

Context

With NYCFacets in 2012, we attempted to address the lack of context in the absence of searchable metadata, even creating a page-ranking algorithm of sorts to score datasets on freshness and quality using the metadata crawler we built.

When we released NYCpedia in 2013, we were trying to address this lack of context in the datasets themselves.  It was our first attempt in “humanizing” open data by location and time – from a list of hundreds of datasets primarily useful to techies and researchers, to something that my mother-in-law, local policy makers, businesses and citizens can use.

We did this by geotemporally correlating the datasets and cross-referencing it with other data sources (the Census for demographics, Flickr for pictures, Indeed for job listings, DNAinfo for hyperlocal news, etc.), so casual users can go to a neighborhood, and find out things about it.

We got a lot of great feedback and coverage, and a lot of people told us that they wanted something like NYCpedia in their jurisdictions. But at the end of the day, we built a website, not a product.

But today, that’s no longer the case.  Based on all the lessons we’ve learned from the past three years since we started Ontodia – from our experience in NYCBigApps – first as entrants, and since our win, as data partners helping third-party organizations publish data for the challenge; from building NYCpedia; from interviews with various policy-makers and researchers; and from deploying several CKAN data portals for various organizations, we built the PediaCities Open Data platform.

The PediaCities Open Data platform is composed of two major subsystems:

• OpenData.city powered by CKAN – the world’s leading open-source data portal platform, extended and optimized for both SaaS and PaaS, and our
• Key Place Indicators platform

Open Data for All

Beyond having a managed Open Data portal with CKAN, we installed a number of plugins from its large open-source extensions ecosystem to make it more usable for both data consumers and data publishers.  And with our Pro option, we go even further – an analytics dashboard,  API management, enterprise connectors, and an app generator framework.

We also seeded the portal with great data and maps coming from trusted sources (US Census, Dept of Labor Statistics, etc.), pre-sliced for each jurisdiction, updating the data as those sources are changed.   And we’ll do this for all 3,000+ counties and 30,000+ towns and cities in the US.

We also offer a hosts of data-wrangling and geocoding services to help organizations publish great data – from screening datasets for security and privacy, hyperlocal geocoding that recognize local boundaries that constituents care about, and visualization services through us and our partner – CartoDB.

All this managed and overseen by our NOC partner – ExterNetworks, to ensure high availability.

And when we say Open Data for All, we mean Open Data for All… if your jurisdiction has less than 20 datasets, you can keep using the data portal for free!  And  since its powered by CKAN – there’s a No Lock-In Guarantee – that is, after one year,  you can take your CKAN with you if you wish to run it on your own, or with another CKAN provider.

Key Place Indicators

Alongside CKAN, we also developed an Indicators platform that allows you to track local issues you care about.  For this Beta launch, we seeded the platform with ISO 37120 indicators –  the result of a years-long effort by experts from around the world to create a standard way for Cities to track and “measure delivery of services and quality of life.”   Our value-add is that we precompute these indicators – from Data uploaded in our platform, and data that we harvest from various public sources.

And these are just our starter indicators. Soon, we will allow users to create their own indicators which are automatically computed as the underlying data changes.  Want to track vacant/abandoned lots?  You can build an indicator for that.  Do you want to track the density of green areas in a certain place? Yep!  Do you want to create a noise indicator at the zipcode level?  Pedestrian traffic in a given neighborhood? Average fuel prices for a jurisdiction?

It’s all possible – so long as the data and the polygon for the desired jurisdiction is there.

We feel that by having a ready mechanism to measure desired outcomes on top of the aggregated, curated open data, we can incentivize data producers/mappers/sharers/gatherers/story-tellers to publish their data in a way that’s immediately usable.

And by always having the data available behind each indicator or visualization,  that we do our part in accelerating dynamic, live, data-driven decisions, as we go away from trapping data in PDFs that nobody reads and pretty maps and infographics with no data provenance.

End of Year Beta promotion

We’re just getting started.  We’ll be exposing more services over the next few months to accelerate Open Data-driven decisions and innovations.

And we’d like to inform our product choices with your real-world needs.  So if your jurisdiction signs up before the end of the year, we offer a 20% discount.

Rest assured, with our partner ExterNetworks, you’ll be publishing your Open Data using our extended version of CKAN with confidence, while we prioritize indicators and pre-seeding and harvesting data for jurisdictions that sign up.

So you’ll get all the benefits of Open Source – hundreds of CKAN contributors ranging from companies like Ontodia to State governments; no lock-in; the ability to tailor your data portal with our PaaS offering; access to hundreds of CKAN extensions – with the peace of mind of a managed cloud offering running on Amazon Web Services, backed by a Service Level Agreement.

Join us in our journey to democratize Open Data for All!

Last Thursday, I had the great honor of witnessing the passage of the City Record Online Law.  In my book, its as big a milestone as the widely-acclaimed PLUTO dataset release last year.

Because the City Record is the “logfile” of the City.  Its the City’s official daily newspaper and has information about public hearings, agency rule changes, procurement actions, contract awards and City employee salary changes going all the way back to 1873!

It’s a valuable resource that up to now, has only been known to City insiders, even after the City started publishing the PDFs in 2011.

Because as #opendata practitioners know, PDF is 1-star open data. Its “digital paper”, much the same way the first automobiles were called “horseless carriages” – a direct translation of what we had before computers.

Not to diminish the utility of PDFs.  I for one, was a big proponent when they were first introduced back in mid-90s.  Back then, when I first started my career in LIMS software development, a lot of our support problems were due to customers not being able to share their reports with co-workers and customers without having them first install a specific version of a specialized, proprietary report viewer.  I was responsible for implementing our report viewer, and I can still remember the various hair-pulling, DLL Hell customer episodes we had when critical Certificates of Analysis couldn’t be opened by our clients’ customers.

As PDF was truly a “Portable Document Format.” It didn’t require the installation of anything else beyond the now universally available PDF reader.  It was no surprise that it was universally adopted shortly after its introduction.

But Nobody Reads PDFs

In a World Bank study released earlier this summer entitled “Which World Bank reports are widely read?“, World Bank researchers inspected their website traffic data and found that fully one-third of their 2008-2012 PDF reports have never been downloaded at all, another 40% were downloaded fewer than 100 times, and less than 13% were downloaded more than 250 times.

And that’s for a corpus of 1,611 reports spanning four years, which by the researchers’ own reckoning, accounts for about one-quarter of its country services budget, with a mean cost of $180,000 per report.

What more for the City Record, which truly reads like a logfile, and not thoroughly researched scholarly prose written by experts?  Within its drab, black and white, 8-point font, single-spaced, three-column, information-packed pages, the City Record details all major contracts and employee salary adjustments, which last fiscal year alone, amounted to $21.2 billion and $22.2 billion (total payroll) respectively!  That’s 55 percent of the City’s $78.2 billion dollar FY 2014 budget! (source: checkbooknyc.com)

If we had access to City Record machine-readable data, and cross-referenced it with the CheckBook NYC API, could we have detected the CityTime scandal in the making?  When contractors “treated the City like an ATM machine“, inflating the project’s original $63 million 1998 budget to over $600 million by the time they were exposed by Daily News columnist Juan Gonzalez in 2009?

And that’s just for contracts and salaries.  What about all the public hearing notices? Rule changes? And public property dispositions? And all the other notices mandated by the City Charter be published in the City Record?

Just imagine the possibilities. What can we do with this data once its liberated from PDFs?

Can we prototype an “Ombudsbot” – a Machine as Ombudsman, that can automatically flag suspicious activities?  Can we use technologies like Splunk on the liberated City Record data to gain operational intelligence – the Pulse of the City Administration? What about other municipal datasets trapped in information-packed PDFs like Environmental Impact Statements, City Agency Audit Reports and all the other “knowledge products” largely disseminated as PDFs?  Could this be a precedent for unlocking those data-sources as well?  Could we apply these techniques to other PDF-rich, data-poor, 1-star data troves – the World Bank’s PDFs, academic publications, annual reports, etc?

Enter the City Record Online Working Group (CROW)

And we just don’t need to ruminate on  the possibilities.  Because not only was the City Record Online Law passed, Mayor de Blasio also announced the creation of a public-private partnership centered around the release of more than 4,000 archival City Records dating back to 1998, with the City working with six partners – BetaNYC, Citizens Union, Dev Bootcamp, Socrata, Sunlight Foundation and Ontodia to convert the historical archive to machine readable information – taking it beyond just Optical Character Recognition (OCR) to Entity Recognition and Classification.

Its an honor to be part of CROW, and we thank the Mayor, Manhattan Borough President Gale Brewer, Council Member Brad Lander, Council Member Ben Kallos, and Council Member James Vacca for their leadership and this amazing opportunity.  We thank Citizens Union for their pioneering vision and we look forward to working with the Department of Citywide Administrative Services (DCAS), Department of Information Technology & Telecommunications (DoITT),  and the rest of the team.

The time has come for PDFs to be redefined.  PDF stands for “PDF Data-Fication.”

Join us at  http://bit.ly/git-crow.