Socrata Open Data API

International Open Data Hackathon at Socrata

2011-11-18

Socrata is proud to be hosting the Seattle branch of the 2011 International Open Data Hackathon at our offices in Pioneer Square. For the uninitiated, the International Open Data Hackathon is a loosely-connected event of hackathons in different cities around the world with a common goal of getting as many people around the world hacking on open data at the same time. It’s a great chance for the community to connect around some common goals and build some awesome tech.

The Seattle event will be hosted at Socrata’s offices right in downtown Seattle. We’ve got space for maybe 20-30 developers if we spread out around the office, plently of bandwidth, and a great view of Pioneer Square. My personal goal is to hack on some of the awesome restaurant inspection data provided by Chicago, King County, and the City of New York, but I also welcome people to come work on whatever catches their fancy.

We’ll plan on starting up at about 9AM and sticking around until 6PM, but if people want to keep working later or head out to socialize we’re always game. If you’re interested in attending, please sign up on EventBrite so we know to expect you. That’ll also make sure we have enough refreshments to keep you energized throughout the day!

I’ll also be available on the hackathon IRC channel as “metcalf” if people need help with the Socrata APIs

Join Socrata at #OGDcamp!

2011-10-17

Socrata is off to Poland! I’ll be representing Socrata at Open Government Data Camp in Warsaw, Poland on October 20th and 21st. If you have any questions about our APIs, our products, or our data, track me down and I’ll be glad to help out.

From the Community: Access Chicagoland data with Windy.php

2011-10-07

Something amazing is happening in Chicago. The City of Chicago, Cook County, and State of Illinois all are coming together around the cause of Open Data and are publishing awesome datasets to their collective data sites, and a great developer community is coming together around this influx of new data.

Paul Weinstein has put together an awesome PHP library, Windy.php, in order to help PHP developers get started accessing data from the city, county, and state. It wraps up all the complexities of accessing the different datasites into a neat library and even helps you retreive and parse KML files.

For you Ruby hackers, there’s also the windy.rb module, which provides a similar interface for accessing City of Chicago data from Ruby.

Photo from chicagophotogirl’s CC-licensed photo stream

From the community: SODA examples in Python

2011-10-06

We always like seeing the community pitching in and showcasing how powerful SODA can be when combined with meaningful open data. Today I wanted to highlight Metro Chicago Open Data Examples, written by Stephen Newell.

Over the past two months, Stephen has been posting a series of great posts showing how to do various tasks working with open data in Python, and much of it has to do with the Socrata Open Data API. If you wanted to see step-by-step, for example, how to create an inline filter on some data, he’s written about it here. Or, if you need to be able to page through a large amount of data on the API, there’s an example you can use here.

If you have written anything else (sample code, walkthroughs, or otherwise) that might help your fellow civic developers get acclimated with the open data platform, please do write to us and let us know; I’d be happy to feature your work here. For now, though, read up on the genesis of the Metro Chicago Open Data Examples blog and grab the code off of GitHub if you’re at all interested in Python development on the Socrata Open Data API.

Important Changes to the Publisher API: Publishing Workflow

2011-08-25

Update: The rollout of the Publishing Workflow has completed. If you encounter any problems with the new workflow, or if you need help migrating your processes or code, please open a support ticket and we’ll be glad to help you.

Update: The Publishing Workflow rollout has been delayed until 9/20. As before, we’ll provide updates here when the actual deployment has been completed.

Update: The Socrata team will be rolling out the publisher API changes on the evening (Pacific Daylight Time) of Thursday, September 15th. We’ll post another notification when the deployment is complete.

The following is an important announcement for users of the Socrata Open Data Publisher API. If you don’t use the Publisher API to update datasets, it doesn’t pertain to you. But if you’re interested, feel free to read on, because we think it’s pretty cool.

The Socrata team is excited to announce an important update to the Publisher API: The Socrata Publishing Workflow.

Details of the Publishing Workflow have been available for several months, and now that many of our customers have migrated their systems to be prepared for the change, we’re making the final preparations to launch it to our shared production platform.

The Publishing Workflow does change the way that publishers must interact with our platform, so this is an important change to be aware of and prepared for. We plan on rolling out the change during the week of September 12th. Final deployment timing and details will be shared with all publishers before the change is deployed.

For more details, see the documentation on the Publishing Workflow or read on.

What is it?

The Publishing Workflow changes some of the fundamental ways in which datasets work on Socrata. Instead of being a dataset being an always-live representation of your data – where consumers see changes or updates you make in real time – datasets will have two states: Published and Working Copy

A Published dataset is a static, finalized copy of your data intended for sharing either with the public or within your organization. It represents a particular version of your dataset, and thus is locked against editing. In order to make edits to your datasets, you must first create a Working Copy.

A Working Copy is an editable version of your dataset that is distinct from the Published copy. Think of it as an internal draft copy of your dataset that you can collaborate with your coworkers on. You can make whatever changes you want to the Working Copy without those changes being reflected in the Published copy. When you’re done with making your changes, you can “publish” the Working Copy to be the new Published Copy, and your changes will become live. Even better, the old Published Copy becomes a Snapshot, which you can reference as a historical version of the dataset.

What are the benefits of it?

There are many benefits to the Publishing Workflow, but here are a few of our favorites:

Since there is an explicit publishing step, Published copies can be heavily cached, allowing us to improve the performance of datasets both large and small
Since you can make changes to a Working Copy without affecting the Published copy, you have a chance to review updates for correctness, clarity, and completeness before they go live. You can even share them within your organization to get approval before the changes are published. No longer will you have to fear that the changes or updates you’re making will break your dataset!
Since old versions of your dataset are archived as Snapshots, and even accessible via the Consumer API, you can keep historical archives of your dataset around to show how it has changed over time.

How does it impact me? How do I prepare for the change?

The Publishing Workflow will be available both in the online Socrata Publisher tools and also in the Publisher API. For users of the online Publisher tools, all you’ll need to do is click the provided button to create a Working Copy before modifying your dataset.

For users of the Publisher API, there are a few small modifications you’ll need to make to your client. Since you’ll need to create a Working Copy of your dataset, and then publish your changes after you’re done, there are two additional calls you’ll need to make to the /publication API in your workflow, like wrapping database updates in a transaction:

First you’ll need to call POST /api/views/$view_id/publication.json?method=copy to create the Working Copy. The response to this service call will include the metadata and ID of the Working Copy that was created.
Next you’ll make your updates to your dataset as usual, this time substituting the Working Copy for the original dataset.
Finally you’ll call POST /api/views/$working_copy_view_id/publication.json in order to publish the Working Copy as the new Published Copy

That’s it. For more details, see the detailed documentation on the Publishing Workflow.

tl;dr – That was all far too much, give me the short version

Socrata will soon be deploying changes to the Publisher API that will make it possible for you to isolate changes to datasets from the version consumers on your data site will see.

You’ll need to update your Publisher API client by the week of September 12th to take advantage of these changes. For more details, see the documentation on the Publishing Workflow.

Help! I’m not sure how to migrate my code!

If you have additional questions about the Publishing Workflow that aren’t answered by here or in the documentation, please email your questions or concerns to support@socrata.com.

Important change to SODA APIs: HTTPS Everywhere

2011-05-23

In the coming week, we will be enabling a new feature on all Socrata-powered data sites that we call “HTTPS Everywhere”. This frequently-requested feature will change the behavior of our platform to use secure HTTPS connections, just like your bank uses, for all interactions rather than just for the login page. This means that not only will your login be secure, but your entire session will be protected against the possibility of having a malicious third party hijack your session credentials using a technique that has gained some notoriety recently in the news as “sidejacking”. Sidejacking allows a malicious third party on the same network as you to steal your session information and masquerade as you without needing to steal your username and password. Many prominent companies such as Google and Facebook allow HTTPS everywhere as an option – we are going a step further by making this the default for all data sites.

For API users, this means that all interactions with the Socrata Open Data Consumer and Publisher APIs must be performed over secure HTTPS connections. Otherwise, everything will continue to operate the same.

We’ll be rolling out this feature on May 27th at 5PM Pacific Time. If you have any questions, feel free to email us at support@socrata.com or comment below.

Join Socrata at Open Gov West 2011

2011-05-03

Socrata is excited to be taking part in the Open Gov West Cross-Gov Civic Apps Contest on May 14, 2011.

As we’ve done with several of our hackathons in the past, we’re partnering up with our good friends at Tropo for this event. It runs from 9AM to 7PM on Saturday, May 14, and we’re excited to have a great group of government participants from across the Pacific Northwest, including representatives from the Portland, Seattle, King County, and the State of Oregon.

The theme of the event is “cross-government applications”, and Socrata and Tropo will be kicking in prizes for the teams that build the best apps judged by the following criteria:

The top three projects will be awarded prizes based on the following criteria:
Most innovation use of technology.
Problem or public policy imperative addressed.
Reusability of solution or service by multiple governments.
Use of multiple data sets or APIs.
Availability of source code or application to interested governments.

For more details, see the contest rules.

The event is part of the Open Gov West conference itself, so you will need a ticket, but there are developer sponsorships available for those who qualify. Make sure you sign up or order your ticket soon. See you there!

Important changes to policies on authenticated requests

2011-03-15

As a part of our ongoing commitment to constantly re-evaluating the security of our APIs, we’ve made some changes to the requirements for authenticated requests to the Socrata Open Data API.

Note that these changes only affect authenticated requests, so the vast majority of API users are not affected. This change only affects data publishing clients that are making authenticated requests in order to modify data. It does not affect unauthenticated read-only request from users of the Socrata Open Data Consumer API.

The following rules must be followed for all authenticated requests:

You can continue to use Basic Auth, but we’re requiring that all requests that modify data be made over HTTPS.
Authenticated requests are also now required to include an Application Token with every request.

You will also no longer be able to use cookie-based authentication, as this can lead to insecure practices.

Adding an application token to your requests is straightforward:

Register for an app token for your application. If you don’t want information about your application to be public, you can make the application profile private.
Add your app token to your request using either the app_token parameter or using the X-App-Token HTTP header.

For more information, see the article on API authentication. You can also ask questions via the Developer Forum.

Join us at Civic Hack Day Baltimore

2011-02-07

Now that the Socrata-powered OpenBaltimore is officially launched, I’m excited to be joining the gang from Tropo on Saturday, February 12 for Civic Hack Day at the Beehive. It should be a great opportunity for everybody to start getting their hands dirty with Baltimore’s new data site and build some awesome apps using the SODA API. I’m looking forward to interacting with the Baltimore open data community to see what we can create together.

So, if you’re in the Baltimore area and you’re interested in open data, come hack with us!

Making government data accessible by phone using SODA and Tropo

2010-10-28

Update: I’ve updated the script to handle text messaging as well. Details below

In an age of smart phones, iPads, and the mobile web, we often forget that a huge portion of the population doesn’t even have access to the Internet. How do we make sure that no one is denied access to the excellent advances we’re making in open government data? One great option is to make that information accessible via a technology we’ve had for the last 134 years – the telephone.

Earlier this week I was asked to speak at a King County Open Data Workshop and talk to some of Seattle’s civic developers about what they could build using the Socrata Open Data API and data published by King County on their DataKC.org Socrata-powered datasite.

Stretched for time, and wanting to build something compelling, I thought about what “itches” I’d had in the past that could have been scratched with an app built using the SODA API. After finding King County’s What do I do with…? dataset of recycling options in my area, I remembered the first Christmas I hosted in Seattle. Finding myself in the “Evergreen State”, and never having had a “real tree” before, I went down to the local Boy Scout tree sale and bought my first ever live Christmas tree. Not knowing what to do with it, I held onto that tree until almost February, until it was good and brown and crispy. What if King County had an automated hotline I could have called in order to find out where I could take my tree for composting?

And thus, an hour and 100 heavily-commented lines later, the King County Christmas Tree Disposal Line was born:

Built using the great tools from our friends at Tropo, the app is very simple. First, we tell Tropo to answer the phone, greet our user, and accept a five-digit zip code, either entered via the keypad or speech to text:

The lookup_facilities function then builds a filtered query using the SODA API to request matching facilities:

That’s really about it. The Tropo Scripting API needs a URL from which to load the script to run, so I created a simple Rack app hosted on Heroku that simply hosts the christmas.rb file along with a homepage with a cute little tree. If you want to check it out or fork the code for yourself, I’ve pushed the whole shebang up to Github.

If you want to give it a try, simply call (206) 397-0769 and enter a Seattle-area zip code, like 98133. Tropo also supports SMS, IM protocols, and even Twitter, so there is much more left to explore. If you build something cool, make sure you submit your app and leave a comment below. Good luck!

Update: I’ve updated the Tropo script to handle requests via text message as well. Besides some general refactoring, the key was checking the Tropo session’s initialText value to see if the sender sent a text message to start with. The rest of the details can be found in my diff:

Photo from slorp’s CC-licensed Flickr photostream