Greg Tracy

Celebrating National Day of Civic Hacking

2013-06-02T23:56:00.000-05:00

This weekend was National Day of Civic Hacking. An incredibly well organized, distributed and disconnected network of events across the country where people came together to build tools and policies to make their communities better.

I love this type of work. It's the inspiration behind SMSMyBus, the underlying API that powers 40+ transit applications, and our local civic hacking organization, HackingMadison.

Although we didn't get an event organized for Madison, I wanted to contribute something. I had a few hours each morning this weekend to mash my keyboard on a project so I picked something small. There has always been a lot of interest around the data in the transit API so I decided to expose some of it by visualizing the API requests in real-time.

The browser app displays a Google Map and drops red circles on the bus stop locations as API requests come in for those stops. Fun. Simple. Transparent.

http://api.smsmybus.com/map

Life should be a bowl of ice cream

2013-03-24T17:45:00.001-05:00

A public service announcement for my peers... Enjoy your parents. Spend time with them like it's your last.

My dad is visiting this weekend and we're not doing anything important. We watch basketball. Drink beer. Compare career notes. Turn society's debates into math problems. Tour the city like tourists. Run errands. Sit in silence. And eat ice cream.

It's wonderful.

The truth is, these visits are numbered. His mobility is decreasing and his mental acumen is dulling. Enough to make everyone quite sad if we stopped to focus on it. If we stopped to stress on any of the messy, strained bits of families that weave into the fabric of life, we would miss opportunities like this weekend.

But we don't. And that is the secret sauce of the wonderful.

These moments have been unfortunately aided by tragic events. The loss of a sister and daughter and premature mortality that Parkinson's brings changes your lens.

That's the tragedy behind the tragedy. What if you didn't need to have your lens prescribed?

That's your challenge. Look passed the messy stuff. Scoop your dad ice cream. Sit back. And enjoy the simple part of life.

Goodbye Posterous. Hello Blogger.

2012-11-18T10:41:00.000-06:00

I've been hosting my blog on Posterous since I started writing in 2008. Earlier in the year, they sold the business to Twitter with no guarantees that the service would be around long term.

I kept waiting for them to offer a tool to extract/port content and it hasn't arrived. So I decided to scratch an itch and do it myself. I ported my blog to Blogger this weekend and here's how I did it.

Problem scope

The blog content is the easy(ish) part, but if you want to preserve the links to the original content, it can be tricky. Depending on your search engine and social media traffic, those thinks can be valuable.

First, every blogging platform uses its own slug path structure. Second, if your blog is in a subdomain, you have to find a way to preserve that subdomain with the addresses on the new blogging platform. For example, the parking API post has two different URIs.

Posterous :

www.gregtracy.com/adding-parking-to-the-madison-api-homebrew

Blogger :

blog.gregtracy.com/2012/01/adding-parking-to-madison-api-homebrew.html

Migrating content

Content migration is a tedious process. But the current option for getting Posterous content to Blogger is well documented on the web.

Import Posterous content into Wordpress
Export Wordpress to an XML file
Use wordpress2blogger to transform the XML file for Blogger import
Import XML file into Blogger

This flow is documented in detail here. Lifehacker has also documented a process for migrating content using the auto-post feature inside Posterous, but it is tedious.

It doesn't necessarily end there, however. There are two glaring problems that make this process linger on forever. First, the content is poorly formatted so you'll find yourself editing posts in Blogger to clean up formatting. Second, multimedia sometimes doesn't import correctly or references files on Posterous or Wordpress so you need to upload that content to Blogger.

Someone will eventually build a migrator by mashing up the Posterous API with the Blogger API to make this all go away. In a different life, I might have taken the time to do it. But until someone does, the content migration is a little messy.

Building the redirector

This was the piece I was most interested in because it's simple and fun to build.

Use the Posterous API to get a list of all post URI slugs and their post dates
Construct a slug map for Posterous URIs to Blogger URIs utilizing the dates and slugs
Build an app that takes any Posterous URI slug and redirects the page to the Blogger URI
Setup DNS so your old blog points to your app

Posterous listing

I wrote the following nodejs app to grab the full list of public posts from Posterous.

Blogger listing

If you can get access to the Blogger API, which requires a special request to the Blogger team, you can do the same thing I did with Posterous. The challenge is finding the ways Blogger changes slugs. For example, they remove "the" and "a" from most post slugs.

Finding these differences is tedious so I actually recommend making sure you get the mapping correct for your top ten blog posts and then just crowd source the rest. Keep track of the page misses with logging (see below for my solution) and then update the map as you find mistakes.

In the end, the map looks something like this:

{ 'awesome-post-title-wins : { date : '/2012/11/' , blogger_slug : 'awesome-post-title' }

With those details, you can get from any legacy slug - /awesome-post-title-wins - to any Blogger URI - /2012/11/awesome-post-title

Redirector application

I chose to use Google App Engine to host my redirector, but you could use anything. The logic is very simple. Just lookup the inbound request in your slug map and redirect to the Blogger URI. It looks a lot like the following.

My full implementation, including the slug map can be found on github. My implementation also includes a miss tracker which is what I've used for understanding which redirects are failing. I'm persisting the routes and a counter for the number of occurrences to understand what's missing. I can find all of the misses using the datastore viewer in the App Engine dashboard.

The incidentals

There are a couple of miscellaneous resources that you'll need to add to your map file as well.

RSS feed
favicon.ico
robots.txt
Apple touch icons
Posterous "tags" (Blogger "labels")

All of those can be redirected, but creating static files for your favicon and apple touch icons allow you to personalize that content better on your new domain.

Domain mapping

Now that you have an application that redirects content to Blogger, you need to map that application to your old domain. In my case, I updated the DNS for www.gregtracy.com to point to my app. The combination of App Engine and Google Apps makes this very easy.

Inside your App Engine dashboard, select "Application Settings" and use "Add Domain" to map the application to your subdomain. These steps walk you all the way through DNS setup.

That's it. What details did I miss? http://blog.gregtracy.com

Cancer stole one of the good ones today

2012-07-16T20:03:00.000-05:00

Around 1,500 people die from cancer every day.

But today that number feels different. Today, my own sister became a statistic.

Marla Tracy Pak was diagnosed with glioblastoma multiforme last August. Fought an ugly and losing battle for 10 months, and left us today, leaving a big void in the TracyPak Tribe.

I was asked to do a PechKucha talk the day I got the news last August. I called an audible and instead of talking about civic hacking, I talked about human qualities that make families, tribes and communities better. The qualities that my sister had.

I neglected to share that presentation last year, but here are the highlights...

Be kind to every person you interact with and talk to strangers like you’ve met them before.

Have empathy.

Support the things your friends are passionate about regardless of your opinion.

Celebrate nostalgia within your tribe.

Spoil your nieces, nephews and neighbors’ kids.

Converse with others with infinite optimism in your voice.

Send surprise packages to people.

Call an out of town friend or sibling and setup a long-weekend visit just because you can.

Make a choice to relate to people rather than drawing contrasts.

Success in your personal relationships defines who you are. We should all be a little bit more like Marla.

Spam blockers and colors

2012-02-18T16:29:00.000-06:00

The Astronomy Picture of the Day email service I created several years ago has been getting pummeled by spam critters. I was getting up to five signups a day with bogus email addresses. Most of them came from Hotmail, AOL and Yahoo. With the new pricing structure of Google App Engine, it was beginning to cost me in the pocketbook so I had to address it.

The solution I came up with was part spam filter and part fundraiser. I decided to draw more attention to the PayPal donate button by requiring new signups to type the color of the button into one of the form fields. A couple of interesting things happened.

First, the rate of donations didn't change. I've had one modest donation in two weeks.

Second, the spam ended. Overnight. It was fantastic.

Most interestingly, the little test showed fun insight into how we all see colors. When I originally wrote the test, the answer was "yellow". At the very last minute I decided that I should throw in "orange" as a possible answer as well and it was a good thing I did. Since deploying that change 15 days ago, there have been 51 signups - 26 orange and 25 yellow. I don't know why, but his fascinates me. It seems obvious to me that it is yellow. :)

What color do you see?

Adding parking to the Madison API homebrew

2012-01-02T17:35:00.000-06:00

It started with this tweet.

... and then I found myself on the couch during New Year's Eve with a really bad movie. Laptop out... in the wee hours of the New Year, I released a new web service for Madison. It provides a simple JSON wrapper around Madison's recently released data for parking availability on the isthmus. I added it to the list of services available in the SMSMyBus API.

http://smsmybus.com/api/v1/getparking
I was greeted in the morning by a lot of great feedback so I hooked it up to the SMSMyBus interfaces - SMS, xmpp and email - to create an app for it. For example, you can find the percentage of open spaces in downtown lots by texting 'parking' to the service...

... but what I really hope for, is that others take this and build something creative with it.

Although it's great to get so much positive feedback from the Madison community on these little projects, it's really (really) important to recognize that the work I've done with the API is going in the wrong direction in many ways. I'd rather be building the apps on top of services created by Madison. Eventually, this great city of ours needs to stop building websites and start producing the core feeds that I've done. That's the foundation. That is the government as a platform.

It's been two months since I canceled my data plan

2011-11-25T09:33:00.000-06:00

Nearly two months ago, I wrote that I had canceled the data plan on my mobile phone. I'm reporting back that it has been a smashing success. I love being connected... to the people around me rather than the interwebs.

If you're disciplined about keeping the phone in your pocket, good for you. If you're not, this might be an option for you.

The experiment was given an extra boost when two weeks into the experiment I promptly lost my non-smart phone. Brilliant move in retrospect. I avoided the nervous, fidgety grabbing of the phone to look to see if a new message arrived. One might say that I went cold turkey, and it likely made the whole transition easier. So easy, in fact, I often questioned whether I needed any kind of phone at all!

That's not to say that the data plan isn't missed. My expectations were pretty spot on. I miss Google Maps and posting photos for the family. Here are the few notes I've jotted down along the way...

The pot

Let's get it right out there... I miss having reading material when I'm on the pot. If you have a smart phone, you know what I'm talking about.

My calendar

I miss access to my calendar... I haven't missed big meetings, but I have missed those non-vital but still important events that aren't necessarily on my radar every day. I've looked into the Google API to get access via SMS and will try to implement this one.

Twitter

I'd like to find a better desktop tool for Twitter. Now that I see less of it throughout the day, it would be nice to find a tool that helps me catchup on some feeds I don't want to miss.

Email

Email becomes less important - which is good. Everyone talks about the email tax where you can't control inbound email. I've found that by sending less and reading less frequently, I've been able to lower the burden of managing email. It now comes in well controlled bursts - those blocks of time that I dedicate to my inbox. I've also become ruthless with unsubscribe options.

Better focus

I haven't quantified this, but it feels like I have better focus through less distraction. If you're not pulling your phone out all the time to check in on your online life, you've improved your chances of focusing on a specific task whether it's work, a game with the kids or cleaning up around the house.

You all have become more annoying

I'm now way more annoyed by people who choose their phones over me. Whether it's to take a phone call, respond to a text message and check the sports scores, it's nothing short of annoying. I don't know how much I did this to other people two months ago, but I'm glad I don't do it anymore.

Revisiting Google App Engine's pricing changes

2011-11-06T03:40:00.000-06:00

This post revisits my earlier evaluation of Google App Engine's post-preview pricing changes and how it affected my project, SMSMyBus. As I noted, the app was projected to cost between $6 and $7 dollars per day under the new platform pricing.

Since that post, I’ve been rolling out small, incremental changes to optimize the code and combat all of the known issues. I’m thrilled to report that I have the price down to $0/day. And I’m once again impressed by the snappy and reliable App Engine platform.

Looking back on the changes, I can say that I was doing some bad things, some abusive things, and App Engine was making some bad choices as well. But the end result proves that if its developers optimize and do smart things, they are rewarded. App Engine remains a great solution for my transit API service.

Here’s a history of the changes over the last two months that got me down to $0/day...

Platform Configuration

1. Instance allocation
The new pricing model charges applications based on their use of instances (hardware resources where your application is running) rather than CPU utilization. A key to keeping your instance cost down is to simply reduce the number of instances that are spinning. Duh. So I grabbed the instance slider in the application settings and yanked it to the left. This doesn't prevent scaling, it just limits my billing for normal traffic flow.

2. Delete data
App Engine data storage (for your database) costs $0.008/GByte-day. Doesn’t sound too expensive, but I had been storing every single API call I had ever gotten. I thought it would be useful for API developers and for analytics. My drive to $0 outweighed that, however, so I deleted all of the history data and got under the free quota for storage.

Application Configuration

3. Memcached the application's route listings
I was surprised to find that I wasn’t doing this already, but there it was. I have a data structure that maps bus routes and bus stops to scheduling data on the Metro website and it never changes. In some cases - like the static calls from the kiosk clients - I was looking up route listing details in the datastore once every minute!! Fail. I used memcache to keep the common queries in memory and avoid the extra datastore reads.

4. Limit access during off hours
One thing that never changes is when the Metro service is running. There are five+ hours a day where the buses aren’t on the street. But some clients are still asking for data. I stubbed out most of the API during these off hours before the code ever gets close to making a datastore or memcache call.

These four changes brought me down to $0.70 per day. Bam!

Algorithm Changes

5. Asynchronous screen grabs
If you don’t know, behind the API curtain is an ugly screen scraping task that extracts the arrival estimates from the Metro website. So when a client requests arrival data for a stop, the app goes off and requests multiple web pages, machine-reads the information and aggregates all of the results.

The original implementation of the SMS interface did this by creating multiple tasks (one for each route traveling through the respective stop). When a task ran, it stored the results in the datastore. An aggregator task would read those results out of the datastore and piece together the response to the caller.

When the API was created, I couldn’t use background tasks because I had to respond with results in the same HTTP context. That’s when I discovered the great feature, asynchronous url fetch. This essentially let me grab all of the different Metro web pages at the same time. But when I implemented this, I continued to use the datastore as the mechanism for storing and retrieving results. This was just lazy. Under the old pricing, I wasn’t incented to change it other then the fact that it was a bit slow.
Under the new pricing model, this solution was very expensive. The API is continuously running this aggregation algorithm - constantly writing and reading to the datastore for model instances that have a lifespan of under a minute!

I rolled out a change that removed the use of the datastore and instead sorted the aggregated results in memory. This had a dramatic effect on my API quota for datastore reads and writes as well as overall performance and latency for my users. Especially the write operations, where you get penalized by an order of magnitude for this type of behavior because index updates work against your API quota as well.

6. Dogfood
After optimizing the API, I realized that the original SMSMyBus apps (SMS, chat, email and phone interfaces for the Metro) were now the long pole. Those apps were implemented before the API existed so they weren’t benefiting from the API optimizations. Solution... re-implement to use the SMSMyBus API.

It should have been done long ago simply as a validation exercise of the API methods. Credit to the eligence and simplicity of the API - this port was simple and only took a couple of hours.

These two changes brought me down to $0.10/day. Badda-bing.

AppStats

7. Run Appstats on all application interfaces
The last stop on the optimization train was Appstats. A truly great tool in the App Engine toolbox. In just a matter of minutes, you can find the hidden datastore operations that are dragging you down. In my case, it led me to one area that wasn’t being memcached at all. And it revealed an area that was simply using the memcache incorrectly! Love this tool...

This change brought me down $0.00/day. Winning.

Results
App Engine remains a great platform for developers that don’t abuse it and take the time to optimize their applications.

The SMSMyBus API now serves over 6,000 transit requests per day. It’s fast, reliable and flat out fun to use. I’m as proud as ever that I brought this to Madison.

Next step... find a way to fund my SMS users. :)

I just canceled my data plan

2011-10-02T21:55:00.000-05:00

... and here are some of the good things that I expect to happen:

I'll be more engaged when I'm with my kids

I'll have more money in pocket each month

I'll spend more time writing

I'll spend more time reading books

I'll text more with my family

I'll think more about the things needing thinking about

I'll talk to more strangers when I'm waiting in lines

I'll build more Twilio apps that give me access to apps via SMS

I'll be unplugged from work

And here are the bad things that I expect will happen:

When I'm traveling, I'll get lost more without the benefit of Google Maps in my pocket

I'll post fewer mobile photos that I think others would enjoy

I'll miss out on serendipitous information flow I get on Twitter

My Twilio bill will go up with more SMS apps getting built

I'll be unplugged from work (yes, that is double-edged)

It's only been two hours but it feels good so far. Text me in two weeks and ask me how I feel then. :)

Twilio + Google Docs meets MadCamp

2011-09-11T11:17:00.000-05:00

We had another terrific Barcamp event (dubbed "MadCamp") this year. Supported by great sponsors, a great community of learners willing to share their time, and the tireless effort of Phillip Crawford to pull it all together; it was a proud day for Madison.

Urban Land Interest provided a great venue right on the Capital Square. But it wasn't without its challenges. MadCamp sessions were spread out across three floors - with six simultaneous sessions. Attendees had the challenge of getting back to the session board on the first floor and reviewing the options, finding the next room and still use the fifteen minute break to network with other attendees. To limit the burden of foot travel, we did two things.

First, we put the session board online using Google Docs and projected it on a giant wall on the first floor. This not only enabled people to view it from anywhere on their own devices, but it allowed multiple people to edit it at the same time.

Second, now that the session board was online we were able to setup a notification system that could read from Google Docs via the Google Data API. The MadCamp Notifier was born. The notifier would use the Twilio SMS API to send every (subscribed) attendee a list of the next set of sessions every hour that included the session title as well as the room number.

Steven Faulkner had the great idea of including a feedback interface so attendees could text in feedback for the individual presenters at the end of a session. A difficult habit to create in a condensed amount of time, but some folks did provide some thoughtful feedback throughout the day.

So I think there's a product here somewhere. A better way to organize and connect with others at an event. A free idea! Who's going to take it? If you want a head start, here's the MadCamp code... http://github.com/gtracy/madcamp-notifier

Incidentally, both Twilio and Google were sponsors of MadCamp so in many ways, this app was our thank you note.

Photo Credit: Pete Prodoehl

Madison Transit API snags a giant new user

2011-09-10T18:34:00.000-05:00

Over a year ago, I smiled when I found SMSMyBus getting repurposed in the form of a kiosk at the Mother Fool's Coffeehouse. I wrote then about the personal satisfaction of seeing a pet project growing into something bigger.

Well, last week, that satisfaction blossomed into gratification when I learned that the University of Wisconsin had included the SMSMyBus API in the latest version of their app released for the 2011 academic year. Now students using the app can get real time bus arrival estimates throughout the city rather than relying on the fixed schedule right from their iPhone and Android devices.

Super proud.

Mostly because this is exactly the kind of application that can get the attention of the Metro and the City of Madison as an example of what can be accomplished if more public data becomes accessible in an easy to consume (programmable APIs) interface.

Dear Mayor, please take note. We can achieve more, do so at a lower cost, and generate more economic development if we work together.

App Engine's place as a developer playground

2011-09-05T10:28:00.000-05:00

The Google App Engine developer community is a hot mess this week over the new pricing plan for the platform. And for good reason. Many developers are seeing their hosting expenses going up by as much as 500%.

If you're looking for a post that is trashing the App Engine team, you can move along. You won't find it here. These guys are smart and considerate. If you spend any time interacting with them on StackOverflow, email or in person at Google IO, you understand this. In fact, just using the platform for a project you can appreciate their outlook and passion for their product and users. That's not to say they don't have room to improve. But enough with the negativity already!

Effects of new pricing on my projects

There have been a lot of people posting about their apps and revealing the effects of the new pricing on them. I wanted to do the same as a reference point. Note that my use of App Engine has primarily been for personal projects. Some have web front-ends, some have SMS interfaces, some are just based on background tasks and others come and go while I experiment with ideas or calendar events. I still think most of these experiments are well suited for App Engine, but I need to take a hard look at the more successful apps to figure out a long-term strategy because they are not scaling well with the new pricing plan.

I'll share two examples - both philanthropic projects - comparing the effects of the new pricing.

Astronomy Picture of the Day

This app had originally been written in Perl as a grad student and was hosted at the University of Wisconsin. I decided to port the application as the vehicle for learning App Engine and Python so it was the first app I ever wrote on the platform. It's primarily a background app. Every afternoon it runs a job that scrapes the contents of the APOD site and packages it into an email and sends it off to all subscribers. There's a simple web frontend that lets anyone sign up. There are currently 1900+ subscribers.

The app is free to run on the platform today and will cost $0.19/day - or $0.03 per user per year - after the price changes. 100% of those costs can be attributed to the use of the Mail API.

My only complaint about this app is that the change seems extreme. Going from 2,000 free emails to 100 feels like an attempt to curb the spamming community. And for the charity projects like this, all of the good net citizens are the losers with this change.

SMSMyBus

This app was originally built to provide a better interface for the Madison Metro bus service. It provides real-time arrival times for buses via SMS, chat, email and phone. But then it blossomed into a full-featured API for the Metro for other developers.

The app costs $0.01/day to operate today (excluding the SMS interface). It is estimated to cost $6.79/day after the price change. $2,478/year. Yah. That's a whopping 67,800% increase. Shebang.

The root cause of essentially all the cost can be attributed to the main API call that returns arrival times at a particular bus stop - getarrivals - and some of the clients call this repeatedly (like every two minutes). It is also where the confusion starts for me with respect to the new pricing.

Frontend instance hours

Frontend instance hours is projected to be $5.68/day, 84% of the bill. This represents the platform's transition from billing for CPU usage to billing for the contention of instance usage. I get it that they need to do this. They were using the wrong resource metric for monitoring before.

But how do I go from a $0.00 cost for resource consumption to $5.68/day?!? That kind of increment just feels insane. How about $0 to $0.50? Or $0 to $1?

Datastore writes

Datastore writes is projected to be $1.00/day, 14% of the bill. This is harder for me to resolve for a couple of reasons. First, I can't find any cost under the current pricing plan for these operations even though the app's profile is fairly consistent. So I struggle, conceptually, with how this has suddenly become an issue for the app.

Second, $1/day equates to 1M writes/day in the datastore and I simply can't figure out where all of those writes are coming from. My back of the napkin math shows 40,000 writes. I'm totally baffled by this projection.

The rest of it

The rest of the projected cost is a combination of storage and datastore read operations. I can eliminate the former if I simply store less data I wanted to use for analytics. It saves me money, but in the end, ignoring some of the data hurts the developers that use the API.

Optimizing

Now it's my job to go in and take another stab at optimizing the code and start with the getarrivals API call. I thought I had good habits with this so I was a little embarrassed when I found an obvious hole in the query path for route listings. There's a fairly repetitive query that was not being memcached - oops! Now fixed.

The second thing I'm experimenting with is the application's instance configuration. By default, I was letting the platform's scheduler determine my load patterns and create new instances whenever necessary. But I've made two changes. First, I took the scheduler out of 'auto' mode and set the maximum number of idle instances to one, and I've cranked up the minimum latency for the pending request queue to 250ms. In theory, each of these changes should drive the cost down because I should be using less frontend instance time throughout the day.

Let's see what happens! As I do my part with optimization, I'd like to see the App Engine do their part and move to the middle as well. :)

What to do next

I'm guessing that the App Engine platform simply priced things wrong the first time. I think the concept of platform as a service that exploits existing Google infrastructure was a smart, but geeky idea that was poorly modeled or had bad assumptions about its use/abuse. Ironically, the idea didn't scale well and they've been forced to admit that early assumptions on how to price it were just wrong.

The good part about this move... developers are forced to take a deep dive into optimization. Something i've written about before and have been doing again since the clock started on the pricing changes. This not only makes for a better platform for Google and project sharing the resources, but it makes for a better net as a whole. Faster is better.

The bad part...

Developers will be forced to dead pool worthy projects that don't have a business model.

Developers will be forced to port apps to other platforms. That could be a painful pill to swallow for developers when they aren't money making projects

Developers may be sacrificing analytics to avoid datastore bloat and access charges.

What the App Engine team should do about it

Provide better pricing structures for philanthropic and open source projects. App Engine is a great platform for these things and it provides a great playground for developers to support important projects at a low cost while also learning about a platform they can adopt for larger, commercial project down the road. They've hinted at this but will they do it? - http://code.google.com/appengine/kb/postpreviewpricing.html#special_programs_...

Provide more runway for optimization. A couple of weeks to get the sleeves rolled up and optimize their apps just isn't enough time.

Provide better analysis tools to highlight problems

Take baby steps. Must they really take these giant leaps in pricing?

Roll out Python 2.7 to support concurrent requests in Python projects.

In the process of writing this post I found some great resources...

A nice FAQ from App Engine on the new pricing plan - http://code.google.com/appengine/kb/postpreviewpricing.html

A nice resource to help you understand how to manage your app's instance utilization; the real cost bottleneck - http://code.google.com/appengine/docs/adminconsole/performancesettings.html

An excellent post from another developer tackling this problem - http://point7.wordpress.com/2011/09/03/the-amazing-story-of-appengine-and-the-two-orders-of-magnitude/

Madison Transit API Homebrew

2011-09-03T09:24:00.000-05:00

A post on my Madison Transit API homebrew is long overdue. It was an incredibly fun project and when you see how it has enabled others, you realize how much more compelling it is then the simple SMS app I had originally created.
The API has been out for a long time, but there was very little usage early on. I'm not sure what sparked the flame but in the last few months or so there has been a lot of interest and lot of activity from developers. It dawned on me that I've yet to write about it. And that's a shame because the API is actually the most clever piece of the whole SMSMyBus effort.
In the beginning
In the beginning, there was a very specific goal of creating an SMS app that delivered real-time arrival estimates. That was it. It was in response to my own need - as a commuter - for such an app. It was also going to be an entry into the Twilio developer contest that was run the week they released the SMS API publicly.
I knew going in that the Metro didn't provide a formal API or even an XML feed, but I'd done enough HTML screen scraping before to know how to overcome it. Little did I know how tedious that would become with the Metro data. So what started out as a simple texting application turned into a day of cursing the poorly formatted HTML I found on my screen. And every minute of the way I could be heard asking the same question, "Why doesn't the Metro have an API?!?"
When I finished, I swore that I would never let anyone else suffer the pain that I endured so decided to abstract the work I did for the texting app and provide access to my data so other developers could get to the transit data through a standard web service interface.
Why it's clever
The ugly work of screen scraping was now done and buried in the implementation of the SMS application. But by creating a couple of other web handlers, I could easily make the same scheduling data accessible via a web service call.
For example - http://www.smsmybus.com/api/v1/getarrivals?key=nomar&stopID=1101 - will return real-time arrival estimates for routes traveling through the stop at Main and Carrol on the square. That's the way every developer wants to query data from a web service.
With a little more scraping, I could also find the physical location of every stop (latitude and longitude) so developers could query the location of stops by a stop ID and could also query all nearby stops based on a lat/long coordinate. In the end, there was a pretty impressive looking transit API - simple to use and understand for new developers. And it didn't matter if it was ugly in its technique of screen scraping inside the implementation. It had a clean interface for the developers.
Supported API methods include...
getarrivals - Get real-time arrival estimates for any stop ID with the option of filtering by route ID.
getroutes - Get a list of every route in the Metro system.
getstops - Get the details for every stop a route travels through.
getnearbystops - Get the details for all stops within a given distance of a latitude/longitude.
getstoplocation - Get the location details for a stop given a sop ID.
getvehicles (not implemented) - Get details for a particular vehicle on the road.
getservicebulletins (not implemented) - Get a list of service bulletins in the Metro system.
Why I Built It
As I noted, a big reason for building the API was simply to save the next person the trouble of building up a new set of screen scraping tricks to get the data. But I also knew that most people that have attempted this have stopped at the schedule data. I don't know anything that has gone the extra step of getting the route and stop location data as well.More importantly, I wanted to find a way to contribute to the larger mission of opening up a public dataset to make it more accessible. Open data systems are important for lots of reasons, but most importantly, it allows communities to operate more efficiently. And nowhere is that more evident than in transit. Every city, state, and national government in the world should be on a path towards open data right now.
Unfortunately, Madison has been slow to find its course. My dream was that by creating the API and recruiting some more developers to use it, we'd have enough applications to take back to the city to say, "Look! If you open more data, developers will do great things!"
I also did it for the intellectual challenging of build an API. I've been an API consumer for a long time, but had never constructed one myself. It's an exercise I encourage any API consumer to go through. You'll have a whole new perspective for what it takes to build, document and support an API.
Sample Applications
In theory, all of the original interfaces - SMS, google chat, email, and phone - could have been re-implemented using this API, but I didn't go back and do that. But lots of new apps have been built. For example, the bus kiosk displays hanging in a few local businesses use the API to visualize arrival estimates for nearby stops.
Larry Walker used the API to build a Chumby app and also used it to display arrival estimates on a small LED display on an Arduino. He also built a mobile browser app to more easily access arrival estimates. And I used the API to build a Google gadget so you can get scheduling data in the sidebar of gmail. The attached gallery shows off some of these examples.
Go get started building your own Madison transit app! http://www.smsmybus.com/api/

Who needs a phone? The magic of the Twilio Client

2011-08-08T02:05:00.000-05:00

Twilio sucked me into one of their contests this weekend. The rules were simple - build an application that leveraged their new Twilio Client, which enables anyone to build phone and SMS interfaces into the browser.

It's a beautiful extension of their existing APIs and really does get you to ask the question, "Why do I need a separate phone device to communicate with others?"

I didn't have a killer idea or business idea to build on top of this so I decided to do something simple but make it available to anyone to use and extend. I built a browser-based phone that anyone can download and deploy to Google App Engine themselves. All you need is a Twilio account.

The phone has four features. Once you've logged into the application, you can do the following...

Call anyone using a dial pad right in the browser

Receive calls in the browser when people call your Twilio number

Receive calls from friends that login to the app in their browsers

Record voicemail if you aren't logged in to answer (and send you an SMS to let you know)

Go give me a call - http://phone.gregtracy.com

It was super fun to build and even more fun to use. If you'd like to get your own you can find the project on github - http://github.com/gtracy/phonefree-twilio. It's intended to be a turn-key solution for App Engine. Just deploy and go.

There are lots of opportunities to extend this if you're interested. I'd like to implement voicemail and SMS. I'd like to put a design on this so it looks nice. :) I'd also like to try implementing conferencing so you could have party lines right inside the browser. What else?

Conan's commencement speech is really for us old folks

2011-06-15T23:31:00.000-05:00

If you haven't seen it yet, find 25 minutes in your day and watch Conan O'Brien's Dartmouth Commencement speech from last week.
It's as funny as you might expect. But it's also raw - you just need to stay with it until the end. Conan lays out his passion for comedy and how it took failure for him to reinvent himself. It's brilliant. And it's very likely advice that means nothing to these twenty two year-olds. It's a speech that is written for us 30/40 year-olds. The folks that have had enough jobs to know what we don't want, but may still be searching for our true passions and goals.

My TEDx talk on Civic Entrepreneurship

2011-05-04T16:06:00.000-05:00

I spoke at TEDxMadtown in March, and it looks like they just posted the videos. In this talk, I reveal the great life lesson I learned in 2010 - giving back comes in all forms.

I think the only thing more nerve racking than getting up on a TEDx stage is watching yourself on a TEDx stage. :)

http://www.youtube.com/watch?v=gCfoqhPcMqg

Do you pave roads or construct walls?

2011-03-23T08:28:00.000-05:00

I had a discouraging conference call recently. The kind of call that makes you want to run for the hills, give up on your ideological dreams, and resign yourself to living within the system and never trying to make a difference.

The problem was that I ran into a wall builder. Not a single wall which could be scaled, but a system that was designed to continuously build walls meant to prevent progress.

Inevitably, people get tasked with "looking out for the best interest of the organization", but where these individuals fail is in the execution of that mission. They forget about the big picture. The best interest of any organization is not to protect themselves. The goal is to make progress. Build the best product, delight customers, advance research, or make new discoveries. The individuals need to be shepherds but too often act as gatekeepers locking everyone inside and refusing to stick their neck out.

The very best shepherds are the ones that have the most creativity. They find solutions in the face of restrictions. In essence, they become adept at scaling walls and overcoming obstacles. They don't use words like "can't" and "won't". They articulate concerns and encourage solutions. And perhaps most importantly, when presented with a problem, the shepherds understand it is more important to listen then to talk.

Locking the gate and throwing the key away is the easy way out. When you block progress or maintain the status quo, you fail. And you piss a bunch of people off along the way.

Do you pave the way for others? Offer help and advice? Or do you just build barriers?

SMSMyBus by the numbers - a recap of the first year

2011-02-19T11:00:00.000-06:00

It was about this time a year ago that I built the Madison transit app that delivers real-time arrivals estimates for Metro buses via text messages. Little did I realize that it would turn into such a fun adventure.

I took a look at the logs for the past 52 weeks. Here's SMSMyBus by the numbers...

Total number of users : 138
Total number of requests : 2900
Number of unique stops requested : 318
Most popular stop requested : 1878, Jenifer & Ingersoll

Below is a histogram of the requests each week, and the shape is curious. I've never marketed the service so it's only traveled by word of mouth. The cold weather likely led to the recent jump but I can't explain the general lumpiness throughout.

I also plotted all of the requests on a map using size to indicate the number of requests for each stop. If you're interested, you can explore the map more at http://smsmybus.com/labs/map

Total cost to me? It was a little more than $75 - all of it going towards costs associated with the SMS messaging (Twilio). With the rising popularity, I'd love your donations. Just press the 'donate' button on the site. :)

Imagine a world where social media was used to find solutions. Ahhh.

2011-02-15T09:44:00.000-06:00

If you're in Madison and listening to the social webisphere today, you can't help but notice the massive organization going on for a rally at the Capital later today to protest Gov. Walker's proposal to end collective bargaining rights for the majority of public employees in the state of Wisconsin.

I wonder what would happen if the governor - or any public official for that matter - stopped and tried to solve their hardest problems with the help of these same social media tools. Could the same level of organization seen with the protestors be achieved if the tools were used to organize a solution rather than a protest?

There are so many examples of Facebook, Twitter and other tools being used to raise awareness (see Egypt). But how about if everyone stopped picking sides and used the same mediums to propose solutions?

Civic Entrepreneurship, the Pecha Kucha talk caught on video

2011-02-07T23:00:00.000-06:00

youtube http://www.youtube.com/watch?v=3P2cJOum7KI

This is how you don't give someone feedback

2011-02-06T09:00:00.000-06:00

I build stuff all the time. Sometimes it's just for me but most of the time, if I think the work I've done would be useful to others I open it up and share with others. For free. And a few of these projects have been very well received.

One such project I've shared is Ringerous. A simple tool that lets you blog to your Posterous site via a telephone call. It was the very first Twilio app I ever did after I discovered that lovely API. I use Ringerous on my own site and my family uses it on a private site. Very few people - 26 to be exact - have signed up to use it. It was never a company or marketed and it has its own share of flaws. Most notably those related to authentication in the Posterous API.

But it's there for others to enjoy. Like all the other projects I've shared, I try really hard to support it when I get questions. I've even added some features for a couple of users. The support part is really the most fun because I usually learn a lot from those interactions. Who doesn't want to make a product that delights people?

Then I got a trifecta of emails from a disgruntled person last week. I'm doing the only thing I know how to do. I'm sharing them. Because as much as people get pissed about poor customer service from AT&T, the TSA or the local cable company, there is such a thing as a poor customer. I won't say the world doesn't need them, but I will say that I don't need them.

Next time you want to "provide feedback", here's a playbook for doing it badly...

Email #1 at 4:53pm : User is clearly running into a sub-optimal implementation (on my part) for discovering the posterous site, but also fails to read the instructions on the site)...

Ringerous is SHIT.
I've tried *every way I know* to get it to work with my posterous, and it fails EVERY TIME.
"Oops..configuration problems" can basically suck it.

Email #2 at 5:02pm : User must have a bad monitor or a funky browser theme because I can't figure out how "background-color: white;" turns into pink. I want to know. Sort of. But I can't get over "is SHIT" and "suck it" in the first email to want to dig in.

1. change the colors. tiny grey writing on that godawful pink(????) background is *impossible* to read under even the best of circumstances, so putting your "helpful" examples of what to enter into the fields in a way that people can't actually read is.....sub-optimal, at best.
2. "oops...configuration problems" has *got* to be the shittiest, *least* informative error-message I've ever seen in my life. How about maybe having the error message list what was *wrong* with what I entered, instead of making me end up with a photosensitive MIGRAINE from straining to read the bloodydamn, invisible grey-on-pink print, hmmmm?
3. Better yet, how about making whatever setup you have over on *your* side be able to deal with the posterous host-name OR your posterous site-address? That would even eliminate the need for the godawful tiny, unreadable GREY PRINT that I mentioned earlier.
Three words: BAD INTERFACE DESIGN.

Email #3 at 5:16pm : Apparently not content with the first two assaults, he came back with more... Snarky? I think he rounded that corner in line one of the first email. I'm mildly amused that I might have caused a migraine given the flogging I've taken at this point, but know deep down that migraines are likely a part of his daily life independent of Ringerous.

Not to sound snarky here, but:
1. Ringerous (or something like it) *could* very very useful for the blind or visually-impaired in particular. Unfortunately, because you decided to use really tiny fonts, and a REALLY bad color-choice (gray on pink? Seriously????) I now have a photosensitive migraine from straining to look at your godawful mess.
Please study up on how to make this *more* readable for the folks who would actually benefit most from it, okay? Between that, and the *totally cryptically unusable* error-messages, I wouldn't be surprised if a proportion of your potential users just wander away in disgust after trying -- vainly -- to get somewhere with the thing.
Seriously: change the pallete and add some actual helpfulness to the error-messages, or at least make the "example" lettering bigger, and NOT grey on pink.) :(
Haven't actually tried to phone in yet, but given your web-based signup interface, I have my doubts. (Plus, I'm waiting for the damn migraine to go away, thanks to you!!!!) :(

To kill the suspense... yes, he did phone in and successfully posted to his blog. Unfortunetly, there were no fireworks in that post. That makes sense, it was his blog after all. He's saves his best stuff for his service providers. That gave me an idea. I should really setup a Ringerous account for Ringerous. That way people could provide feedback directly over the phone. Ahhh... a project for another time.

My response was short and simple.

I have very little motivation to help you. Your tone is laughable. There are a few nuggets in there that I can take away from this, but I'd actually prefer that you didn't use the service.

It's worth pointing out that there is no pink on that page. So you might need to check your monitor.

Your loyal free service provider

Launched, 2010

2011-01-18T09:34:00.000-06:00

I launched an application at Asthmapolis last week. It's not complete and it probably took longer than it should have. But it's launched - a simple feat worth celebrating.

I was reminded of this when I ran across Seth Godin's recent "#YearInReview What did you ship in 2010?" post. It was a great reminder of one of the best lessons I've learned being a part of some young companies over the last few years - don't settle for ideas or talk of doing things. Instead, go out and experiment, build and put your ideas to work.

In that spirit, I'm going to start celebrating every launch this year. And to match Seth's list, the following is my 2010 launch list. Some worked. Most failed. Some were good ideas with bad execution. Some were terrible ideas but it wasn't clear until after it was tried.

Here's to celebrating doing...

Launch of SMSMyBus, a mobile transit notification app for Madison

Launched Frinook with my daughter so she and her friends could swap books more efficiently

Launched a web service so my neighbors could swap ice skates

Launched the Asthmapolis mobile application

Built spellitfor.us, a tool to help my kids practice their spelling

Helped to re-launch Barcamp Madison

I trained for a marathon and failed

I did an open water swim around the Hog Island lighthouse

Launched the award winning SMS app for Asthmapolis

I launched a notification system for Shorewood swim parents to keep tabs on their kids' swimming results

I gave a PechaKucha talk

Launched the Madison Twilio meetup

Launched a neighborhood block party

"To live a creative life, we must lose our fear of being wrong." -- Joseph Chilton Pearce

Quantifying myself - a 2010 recap

2010-12-31T11:39:00.000-06:00

Back in March when I got started on my marathon training, I signed up for a service called dailymile to track all of my workouts. Even if my marathon aspirations failed, dailymile proved to be a fantastic tool and started a larger trend for me this year. Since starting with dailymile, I've begun quantifying all sorts of useful things in a quest to understand myself better. Partly because my memory is so bad and partly because I just like data. In some cases, I don't yet know when or where the usefulness will come, but in a world of infinite storage it's easy to store all kinds of information.
Here's a summary of what I've been collecting...
Exercise
The year started with dailymile, which I used to track my running, swimming, cycling and bike commutes to work. Dead simple data entry with a social/community element. I will say that I'd trade in the social aspect for a mobile app. :) MyTracks proved to be a great app on Android even though it didn't integrate with dailymile.
When the snow came, my outdoor activity essentially ended and so did my visits to dailymile. But in November, I joined some friends for the 100 Day Challenge. A friendly competition to see who could amass more points (via exercise activities). The idea is simple, you receive points for a variety of exercises - pushups, situps, walking, running, etc. The points are maintained on a Google spreadsheet so everyone knows where the others stand. There's even a little Twilio app that sends out a text message in the morning with the daily standings. I'm not even sure there is a prize at the end, but I can say that the simple fact that my numbers are viewed by others is an incredible motivator for me.
Most recently, I've started carrying around a FitBit to track my general activity level throughout the day. I'm doing this primarily as a research mission for Asthmapolis, but I've enjoyed seeing the statistics each day. And the entire process is passive. I litterally don't do anything other than snap the device onto my pants pocket in the morning.
Task Management
I've been using Remember The Milk for a couple of years to track To-dos - mostly macro-level items like "fix kitchen window" or "make yard signs for melsgreengarden". But for most of this year, I've started using it as my idea board. I've been recording every idea I've had regardless of whether or not it will actually turn into a To-do. Things like "build a Twitter app that let's one query baseball statistics". Likely never to be built by me... but I have an impressive list of good, bad and ugly ideas. Now I just need to find a better place to store them.
At the end of the year, I started tracking my project time with far more granularity. I've often wanted to know how long it would take me to build something like Frinook with my kids, or a more accurate number for the number of hours I'm spending on Asthmapolis related projects. I'm now on a path to figure some of that out. I'm using a service called Toggl which makes this process very very easy and painless.
Given the patchiness of this new habit, I don't have results for an entire year, but here is a sample of the raw numbers for what I've tracked...

Ran, biked, and swam 486 total miles
106 total (dailymile) workouts (1 every 3.5 days isn't bad!)
135 donuts burned
7 pounds burned
2,870 pushups
3,105 situps
Walked 424 flights of stairs
262 tasks completed on Remember The Milk

In addition to logging more things in 2011, I'd like to figure out a new way to aggregate all of it so it isn't spread across so many services. I'm envisioning a system like Nimbits and perhaps implementing a number of APIs to import the data. But I'm pretty sure I don't have the time to implement that so I'm still searching...
The pace of innovation in web design and the proliferation of native mobile applications is making data collection easier and more fun. The tracking tools have become so easy and ubiquitous, you shouldn't stop and ask yourself, "What will I do with this data?". You should be saying to yourself, "I can't wait until I figure out how to use all of this great data!"
What else should I be tracking in 2011?

Boing Boing's startup advice

2010-12-02T10:45:00.000-06:00

I was reading the recent Fast Company article on Boing Boing when I ran into this gem of a qoute from founder David Pescovitz...

Anything that we do, first and foremost, is to please ourselves.

The message of the article pins their success on this core belief. And while circumstances may change with customers and investors over time, David's quote should be the mantra of every startup while they get going and look for direction. Smart people do good things not because they conform, but because they have the vision of something different.

Trying to figure out if you're right is hard enough when you're doing something new. There is no sense in clouding the picture by basing your decisions on perception. It's far easier - and more fun! - to stick with what you believe when navigating the backwoods of innovation.

Civic Entrepreneurship - a PechaKucha talk

2010-11-19T08:41:00.000-06:00

This was the talk I gave at last night's PechaKucha in Madison prior to HTHH. It's the story of my epiphany relative to leverage my technical skills as a way to give back to my community.

via slideshare.net