Chris L Keller...

Automating crowdsourced maps with apps scripts

2013-05-15T12:00:00-07:00

Back in November, on my second day at KPCC, I collaborated with Kim Bui on a crowd-sourced map that displayed the experience that voters had at the polls.

The map – powered using a Google Form to submit data to a spreadsheet which was then uploaded to Fusion Tables – required some hand-holding on my part. OK alot. I had to:

Make sure the address was geocoded properly.
Make sure we added the correct marker based on the experience.
Sync the data to Fusion Tables using a handy script.
Make sure the map markers appear correctly.

But it was my second day, and I didn’t have much on my plate, though I know it could go better.

Some six months later I’ve become more comfortable here, and more comfortable with Google Apps Scripts thanks to some great examples from others. So for a map for Tuesday’s election that will allow voters to submit their experience at the polls I’ve been able to automate a lot of the process. I’ll still hold its hand – cause that only makes sense – but a lot of the manual tweaking should be unnecessary.

Here are some of the scripts I’m using.

Do something when a form is submitted

    function onFormSubmit(e) {
        var emailToAddress = 'ckeller@scpr.org';
        var submissionTimeStamp = e.values[0];
        var submissionPollingPlace = e.values[1]; 
        var submissionExperience = e.values[6];
        var submissionNarrative = e.values[2];
        var submissionFullName = e.values[3];
        var emailSubject = 'Voter issues map submission from' + submissionPollingPlace;
        var emailBody = submissionTimeStamp + '\n' +   
            'Voting at ' + submissionPollingPlace + 
            ' was ' + submissionExperience + '.\n' + submissionNarrative +
            ' -- submitted by' + submissionFullName;
        MailApp.sendEmail(emailToAddress, emailSubject, emailBody);

        // run the data processing functions
        runSpreadsheetFunctions();
    };

I’m still using a Google form to populate a spreadsheet, but I’m using an a script to trigger some additional functions when the submit button is pressed. This one – cribbed from an Andy Boyle walkthrough – will send an email with tje information that was just submitted. The script sets each column to a varible; e.values[0] in this case represents the first column in the spreadsheet…

I’ve also piggybacked a generic runSpreadsheetFunctions() function, so that when a form is submitted, a whole host of actions take place. Among them:

Geocode a user-submitted address

    function geocodeAddressFromCell(){
        var sss = SpreadsheetApp.openById(spreadsheetID);
        var ss = sss.getSheetByName(workingSheetID);
        var lastRow = ss.getLastRow();
        var addressCellData = ss.getRange(lastRow, 2).getValue();
        var results = Maps.newGeocoder().geocode(addressCellData);
        if (results.status == 'OK') {
            var bestResult = results.results[0];
            var lat = bestResult.geometry.location.lat;
            var lng = bestResult.geometry.location.lng;
            var latLng = lat + ',' + lng;
            var addressTargetCellData = ss.getRange(lastRow, 9).setValue(latLng);
        } else {
            Logger.log(results.status);
            return '0,0';
        }  
    };

This script will find a column in the last row of the spreadsheet – var addressCellData = ss.getRange(lastRow, 2).getValue(); – and run that value through Google’s geocoder and output a latitude/longitude pair – var latLng = lat + ',' + lng;. That latitude/longitude pair is assigned to a variable and added to “geocoded_address” column in that last row – var addressTargetCellData = ss.getRange(lastRow, 9).setValue(latLng);

Add a function to a cell

    function evaluateVotingExperience(){
        var sss = SpreadsheetApp.openById(spreadsheetID);
        var ss = sss.getSheetByName(workingSheetID);    
        var projectTypeColumn = "G";
        var projectMarkerIdColumn = "H";
        var lastRow = ss.getLastRow();
        var projectTypeCell = projectTypeColumn + lastRow;
        var projectMarkerIdCell = projectMarkerIdColumn + lastRow;
        var projectMarkerIdFormula = "=IF(" + projectTypeCell + "=\"Negative\", \"large_red\", IF(" + projectTypeCell + "=\"Positive\", \"large_green\"))";
        var ssCellToGetFormula = ss.getRange(projectMarkerIdCell);
        ssCellToGetFormula.setFormula(projectMarkerIdFormula);
    };

This script is similar to the geocodeAddressFromCell() function in that it finds a value in a specific column in the last row of a spreadsheet. But instead of geocoding the value, it will insert an IF formula into the cell to evaluate whether the voter’s experience was positive of negative and add a map marker type depending on the result.

Then of course there is the series of functions that allows you to sync a Google spreadsheet with a Fusion Table.

Tie it all together

The update script is incorporated into runSpreadsheetFunctions(), and which is run when a form is submitted, so the Fusion Table is updated automatically when a voter clicks the submit button.

To be safe, I set a “timeout” to make sure the previous functions have run before the Fusion Table is updated.

    // run functions that will process the data
    function runSpreadsheetFunctions(){

        geocodeAddressFromCell();
        Utilities.sleep(1000);

        evaluateVotingExperience();
        Utilities.sleep(1000);

        updateFusion();
        Utilities.sleep(1000);
    };

Link: Merging Datasets with Common Columns in Google Refine

2013-04-28T12:38:00-07:00

It’s an often encountered situation, but one that can be a pain to address – merging data from two sources around a common column. Here’s a way of doing it in Google Refine…

via OUseful.Info

Link: Meetings Are Toxic

2013-04-28T12:18:00-07:00

A NICAR thread about methods of project management and product development led me to some posts on Agile development which led me to find this gem. Like any piece of knowledge that could be deemed as inherent in some people, it’s always good to be reminded of proper meeting rules.

For those times when you absolutely must have a meeting (this should be a rare event), stick to these simple rules:

Set a 30 minute timer. When it rings, meeting’s over. Period.

Invite as few people as possible.

Never have a meeting without a clear agenda.

via Getting Real

Fresh from Posterous

2013-02-16T09:42:00-08:00

This is my first post using Octopress, which comes after folks finally decided to pull the plug on Posterous. It was announced yesterday that it will be shutting down on April 30, some 13 months after Twitter acquired the platform, or rather the talent that built the platform.

As I wrote in my last post on Posterous, I enoyed using it because it stayed out of my way… I didn’t feel the need to fiddle with the layout, or adjust the admin or mess around with plugins. It let me just write posts… Maybe not as many as I’d like to, but posts nonetheless. And I stuck with it to a certain degree.

I spent some time considering going back to Tumblr, and I found this promising platform called Snipt. Heck I even installed WordPress for spell.

But all in all I like the idea of Octopress, of something that will continue to stay out of my way and just let me write, of something that will lets write in markdown which is then compiled into static HTML.

Time will tell.

I can say Posterous served me well at a certain point in my life, a time when I was learning to learn, code, program and teach/show others, and I’ll have an archive of that. Now it’s time to find something new.

I did take some steps to prepare for a day I figured was coming.

I moved my projects page away from Posterous and organized it bit.
I grabbed an export of all of my content that Posterous housed and moved it to an archive. ~~Now I have some time to figure out how to deal with links that will be broken~~.
I started to move some learning resources and tutorials to Github and Gists.

Last Posterous post... time to find a new method...

2013-02-15T00:00:00-08:00

Looks like folks finally decided to pull the plug on Posterous. It will be shutting down on April 30, some 13 months after Twitter acquired the platform, or rather the talent that built the platform.

I enjoyed using Posterous because it stayed out of my way… I didn’t feel the need to fiddle with the layout, or adjust the admin or mess around with plugins. It let me just write posts… Maybe not as many as I’d like to, but posts nonetheless. And I stuck with it to a certain degree.

Sadly, along with the blogging platform being gone, seems any Google juice I accumulated for things like Fusion Tables walkthroughs and examples will be gone as well… or at least pointing to dead links.

But then that might be for the better. I can say Posterous served me well at a certain point in my life, a time when I was learning to learn, code, program and teach/show others, and I’ll have an archive of that. Now it’s time to find something new.

I did take some steps to prepare for a day I figured was coming.

I formatted my projects page a bit so it’s a little more organized.
I grabbed an export of all of my content that Posterous housed. The Posterous backup made is really easy for me, and I think they way the structured the export should be a model for all content platform companies when they shutdown – ahem, Everyblock.
I started to move some learning resources and tutorials to Github and Gists.
And I started to research some other potential platforms. Wordpress is always there, but it’s too easy for me to get bogged down in how the site looks. Again, more writing and less fuss. Dave Stanton & Robert Hernandez just turned me on to Snipt, which looks promising as well.

Ever want to be a fly on the wall while interesting ideas and topics are being discussed?

2013-02-09T00:00:00-08:00

On Wednesday, thankfully before breaking news shifted my focus entirely, I had the pleasure of listening to some great minds in the area of technology and journalism share their thoughts on everything from data visualization to the pitfalls of throwing everything on a map.

O’Reilly’s Alex Howard hosted a Google Hangout on the topic of data journalism. Present were O’Reilly’s Mac Slocum, USA Today’s Anthony DeBarros, NPR’s Brian Boyer, Dave Stanton from SmartMedia Creative, Michelle Minkoff from The Associated Press, Tariq Kokhar from the World Bank, Simon Rogers from The Guardian and myself.

Some thoughts after a couple months with tabletop and handlebars

2013-02-02T00:00:00-08:00

tl;dr - Learning how I can use tabletop.js & handlebars.js together has opened up some really cool custom mini-CMS possibilities and ways to manage data projects.

But thanks to the experiments of others, I have learned that flat json files can be better to use in production, especially for small one-off projects.

While I’d like to eventually write a script to take the tabletop object and write it to a flat file, I made some adjustments to some python code so I can take a csv, create a flat json file from it and drop that file into a handlebars function with minimal effort.

While it works for me, I’m curious to know if it works for others…

Blog Post

Repo

I’m happy to say that I still have access to some gray days since moving to Southern California, and on a recent one I set about streamlining some project structures, code snippets & templates for two code libraries that I’ve been using more and more since coming to Southern California Public Radio: tabletop.js and handlebars.js.

I’m only now getting back to the write up and posting code.

Overview

tabletop.js allows you to use a Google spreadsheet as a data backend. By simply publishing the spreadsheet and adding its key to a tabletop javascript function you can retrieve an an array of objects to be used on a webpage or in a web application. The library was released back in February – right around the time of NICAR – thanks to work by WNYC and Balance Media.

handlebars.js is a templating library – much like mustache.js – that “provides the power necessary to let you build semantic templates” based on data that is formatted as – get this – javascript objects. Using an example from the handlebar.js website, the library allows you to do things like this…

<div class="entry">
    <h1></h1>
    <div class="body">
        
    </div>
</div>

… where and represents information stored in an object like this:

var context = {title: "My New Post", body: "This is my first post!"}

Practical Applications

Learning how these libraries can work with each other has been a lot of fun, especially in the context of a newly-created position at a new-to-me news organization where I have had a role to play in building a foundation for the presentation of data projects.

For instance, using handlebars.js ability to load precompiled handlebars templates from file we’ve been able to construct a basic project wrapper that can live outside CMS, minimizes the amount of code that needs to be changed at any given time and allows us to manage a single header and one footer file. DRY FTW.

I’ve also been able to use the combination of these libraries to come to having a custom mini-CMS that can run some of these projects, which can then be updated by anyone that can use a spreadsheet.

For instance, one of my first collaborations was with our business and economy reporter Matt DeBord on this project for Explaining the Monthly Jobs Report.

This project combines tabletop.js with handlebars.js – and some highcharts.js for good measure – to show the differences between “Initial,” “Revised,” and Final monthly jobs numbers released by the U.S. Bureau of Labor Statistics.

While my “design” judgement and sensibilities continue to evolve, for my money, the magic is in the fact that Matt can update this presentation for our audience by opening a Google spreadsheet and adding the new numbers. Lightweight, custom mini-CMS!

Tasneem Raja, the Interactive Editor at Mother Jones, has written a solid explainer on how the magazine has leveraged tabletop.js and a Google spreadhseet for collaborative projects such as their You Might Be the 47 Percent If… scoop.

Some of the projects I’ve worked on over the past couple months that use tabletop.js and handlebars.js include a presentation on the purchasing of Capital Appreciation Bonds by local school districts, a “By The Numbers” look at the impact of the so-called fiscal cliff on Emergency Unemployment Compensation benefits and a comparison of California and federal gun laws

But a post on NewsBeast Labs I found while writing this pointing to a post from Jeremy Singer-Vine – appropriately titled “Why I love Tabletop.js but don’t use it in production” – and some words from Mark Boas on an Open News call a couple weeks ago reminded me of potential obstacles and issues with relying too much on tabletop.js for projects.

And a tweet ~~this week~~ a few weeks back from Aaron Williams, and one today from Matt Waite, reminded me I have to finish this post you are reading.

All of the issues that Jeremy listed are real, and Aaron’s experience is real. Thinking about the projects I listed above, none of them really require the kind of dynamic data display that tabletop.js can pull off. It’s a nice feature to have available on the monthly jobs report, but it isn’t crucial, and certainly isn’t required on the others.

A flat file very much would work just fine.

And so I went about finding a way to create a flat json file from a Google spreadsheet that I could use in a handlebars.js template.

My Workaround

I already had Andy Boyle’s python script from a year ago that creates json from a csv. And I noticed in the comments section on Andy’s post that Christopher Groskopf’s csvkit has a csvjson utility that can also convert a csv to a json file.

After flirting with the idea of adding a custom argument to the csvjson utility to allow me to specify a key, I adjusted Andy’s script to create a file that looks like this…

{"objects": [{"Source": "National Employment Law Project", "DataOrder": "1", "SourceLink": "http://www.nelp.org/", "Data": "12.29.2012", "Title": "The last day anyone will receive benefits from the Emergency Unemployment Compensation program unless Congress acts to renew it."}, {"Source": "Congressional Budget Office", "DataOrder": "2", "SourceLink": "", "Data": "$30,000,000,000", "Title": "Estimated cost to renew the Emergency Unemployment Compensation program through the end of 2013."}, {"Source": "National Employment Law Project", "DataOrder": "3", "SourceLink": "http://www.nelp.org/", "Data": "400,000", "Title": "Estimated number of Californians receiving benefits from the Emergency Unemployment Compensation program, which is set to expire Jan. 2."}, {"Source": "National Employment Law Project", "DataOrder": "4", "SourceLink": "http://www.nelp.org/", "Data": "2,100,000", "Title": "Estimated number of Americans receiving benefits under the Emergency Unemployment Compensation program that would lose their unemployment benefits come January if Congress doesn\u2019t act."}]

… which is something that allows me to plug my new file name into a variable in this script and fly.

Snags

Now, admittedly it works because most of my handlebars work thus far has used simple data structures – largely an object that contains an array of objects – but my workflow might fall apart quickly should I need to access multiple data streams, etc. (UPDATE: This has already happened!)

One thing I have noticed through this experiment is tabletop.js strips uppercase letters and underscores from object keys, while the csv to json version doesn’t. So that’s a thing as it’s the difference between a handlebars.js template that uses…

//what tabletop brings

and

// created from a csv to json method

Another thing I’ve noticed is the python script that converts the csv to json renders everything as a string, which has caused some issues when using something like highcharts, which requires is data objects to be integers. So I continue to explore a way to fix this.

Here's a simple script to generate a boilerplate FOIA-letter type thing that I cobbled together...

2013-01-29T00:00:00-08:00

This is a super simple shell script that takes boilerplate FOIA language and creates a markdown file that can be addressed to a particular agency.

It’s tailored to California, but you can adjust it to your liking.

You will still have to add the address, city, state and zip code of the agency, and the type of data being requested, but this is a step up from having a file that I copy and paste from.

The script takes a filename as an argument. That filename will have the date prepended to it so you can see at a glance which were sent out when.

I haven’t decided yet to allow for the address and other potential variables to be passed into the script. At certain points you want to allow for mistakes, and most folks are bound to make mistakes when entering information in on the command line.

#!/bin/bash

    # date variable
    DATE="$(date +"%m-%d-%y")"

    # input for newfile
    NAME_OF_FOIA_LETTER="$1"

    # name for newfile
    OUTPUT_FILE="$DATE-$NAME_OF_FOIA_LETTER.md"

    # create markdown FOIA
    echo "Creating $NAME_OF_FOIA_LETTER FOIA letter"
    cat <<EOF >$OUTPUT_FILE
    **$DATE**

    ADDRESS  $line
    STREET  $line
    CITY, STATE, ZIP CODE  $line

    **RE**: Public records request for NAME OF RECORD

    **To Whom It May Concern**:

    Under the California Public Records Act (Government Code Section 6250 et seq.), I ask to obtain a copy of the following records or data sets which I understand to be held by your agency:

    >*DATA TO BE REQUESTED HERE*

    I ask for a determination on this request within 10 days of receipt, or sooner should you make a determination without review of the record(s) in question.

    If you determine that any or all or the information if exempt, I would like to know if the exemption is discretionary, if it's necessary to exercise discretion in this case and a signed notification citing the legal authorities used to determine the records are exempt.

    Should some -- but not all of the information -- be exempt from disclosure -- and you intend to withhold it -- I ask that you redact it for the time being and make the rest available as requested.

    Please notify me of any duplication costs that will exceed \$10.00. If I can provide any clarification or answer any questions, my contact information is below.

    Thanks in advance and take care.

    Sincerely:

    Chris Keller  $line
    Data Journalist/News Applications Developer  $line
    Southern California Public Radio  $line
    **Web**: scpr.org  $line
    **Email**: ckeller@scpr.org  $line
    **Desk**: (626) 583-5214  $line

    EOF

    open $OUTPUT_FILE

Link: "TLDR" is unnecessary

2013-01-13T00:00:00-08:00

If you summarize your idea in your first sentence, writing “TLDR” is superfluous.

via douglastarr.com

As a guy who has taken to adding a tl;dr nugget on the top of walkthroughs reading the above makes me want to whack myself upside the head… But maybe what I have been trying to do has another name… Maybe something along the lines of, “You Will Accomplish”

Display data from a flat JSON file on a Handlebars.js template file rendered with AJAX

2013-01-12T00:00:00-08:00

tl;dr: The README on the repo below gives a bit of a walkthrough and demonstration of the script I’ve been using to display a flat data file on Handlebars templates rendered with AJAX, which has allowed me to learn to deploy interactive data projects fairly fast.

Blog Post

Demo Page
Repo

Overview

Handlebars.js is a templating library – much like mustache.js – that “provides the power necessary to let you build semantic templates” based on data that is formatted as – get this – javascript objects. Using an example from the handlebar.js website, the library allows you to do things like this…

<div class="entry">
    <h1></h1>
    <div class="body">
        
    </div>
</div>

… where and represents information stored in an object like this:

var context = {title: "My New Post", body: "This is my first post!"}

There are some really good resources out there for those who want to start using the Handlebars JavaScript template library. Here are some links:

Handlebars site
Handlebars GitHubRepo
NetTuts Handlebars Walkthrough
Three-part series on using Handlebars: Part one; Part two; Part three

I’d like to demonstrate a bit of the script I’ve been using to display a flat data file on Handlebars templates render with AJAX, and give a couple practical applications for using Handlebars in a newsroom environment in order to deploy interactive projects fairly fast.

Walkthrough

Coming across Handlebars.js after learning the basics of django templating, I really wanted a way to mimic some of that functionality and store Handlebars templates in reusable, decoupled files that could be shared across projects.

Thankfully this function based on code from here helps me to do exactly that.

// render handlebars templates via ajax
function getTemplateAjax(path, callback) {
    var source, template;
    jqueryNoConflict.ajax({
        url: path,
        success: function (data) {
            source = data;
            template = Handlebars.compile(source);
            if (callback) callback(template);
        }
    });
}

I then abstract out a function to display the compiled template, passing in the name of the template, the css selector I am targeting and the data I want to display.

// function to compile handlebars template
    function renderHandlebarsTemplate(withTemplate,inElement,withData){
        getTemplateAjax(withTemplate, function(template) {
            jqueryNoConflict(inElement).html(template(withData));
        })
    };

I can then call it like this, where dataDetailsTemplate.handlebars is the name of my template, and #data-details is the css selector I am targeting, and data is what I want to display.

renderHandlebarsTemplate('dataDetailsTemplate.handlebars', '#data-details', data);

Let’s go through the full data-script.js file, because there’s a lot in there that I’ve kind of picked up over the last several months.

I don’t really have an idea if it is “correct” to programmers out there, but I know that it works and doesn’t throw me errors.

In learning to use jQuery in the context of my previous CMS – which used several jQuery libraries – I found it just made sense to use a no conflict variable and it’s something I’ve just stuck with:

var jqueryNoConflict = jQuery;

When the DOM is ready I call the retriveData() function which kind of starts the whole ball rolling:

//begin main function
    jqueryNoConflict(document).ready(function(){
        retriveData();
    });
    //end main function

retriveData() looks for my flat JSON file, which set to a variable. It then uses jQuery’s getJSON method to pull the data and run it through a function called renderDataVisualsTemplate(). This is the function that will render my Handlebars template to the page with data in it.

// grab data
    function retriveData() {
        var dataSource = 'working-data-file.json';
        jqueryNoConflict.getJSON(dataSource, renderDataVisualsTemplate);
    };

renderDataVisualsTemplate() gets an argument that represents my the data from my flat JSON file. This function runs my base handlebars template function using the name of my template (dataDetailsTemplate.handlebars), the css selector where I will inject my template (#data-details) and the data I will fill it with (data).

// render compiled handlebars template
    function renderDataVisualsTemplate(data){
        handlebarsDebugHelper();
        renderHandlebarsTemplate('dataDetailsTemplate.handlebars', '#data-details', data);
    };

After that, I have my function to pull my Handlebars template from an external file and compile it. I’ve also included a Handlebars debugger, a “helper” function shows information about the data I am trying to work with.

The base handlebars template function looks like this, and takes three parameters: the name of the template, the css selector and the data object:

// function to compile handlebars template
    function renderHandlebarsTemplate(withTemplate,inElement,withData){
        getTemplateAjax(withTemplate, function(template) {
            jqueryNoConflict(inElement).html(template(withData));
        })
    };

Let’s take a look at the flat JSON file I am using to hold the data that will be rendered to the page. It’s structured as it is in the Handlebars walkthrough.

{"objects": [{"Source": "National Employment Law Project", "DataOrder": "1", "SourceLink": "http://www.nelp.org/", "Data": "12.29.2012", "Title": "The last day anyone will receive benefits from the Emergency Unemployment Compensation program unless Congress acts to renew it."}, {"Source": "Congressional Budget Office", "DataOrder": "2", "SourceLink": "", "Data": "$30,000,000,000", "Title": "Estimated cost to renew the Emergency Unemployment Compensation program through the end of 2013."}]}

To render the data, the Handlebars template is structured just as it would be if it was inline on the index.html page, save for wrapping it in a script tag.

<div>
        
        <h2>Flat file data displayed on a handlebars.js template loaded with ajax</h2>
        
            <p>: <strong></strong><br />
            -- </p>
        
    </div>

In this case, I’m asking that every instance of an object

be rendered to the page and structured in a certain way.

<p>: <strong></strong><br />
    -- </p>

My HTML page isn’t any special, other than have a div that will have all kinds of data injected into it thanks to Handlebars.

<div id="data-details"></div>

Practical Applications?

Your mileage might vary, but I’ve found several practical applications of Handlebars.js just by looking at my needs.

For instance, I came from shops that used a CMS where I could add html, css and JavaScript to a CMS “asset” which was then wrapped by the site header, rail and footer. Here at SCPR, I’ve been lucky enough to have mentors who wanted to and helped to create something similar.

This project is on a custom structure that lies outside of the CMS. The header and footer are each Handlebars templates, external files that I add to each new project. If I need to change a link in the footer I change it in one place and it’s changed on all project pages using the template. Same goes for the header.

You could easily recreate something similar. Let’s say your template structure is something like:

<body>
        <div id="data-header"></div>
            <div id="data-container">
                <div class="row-fluid">
                    <div class="span4">
                        <div id="data-details"></div>
                    </div>
                    <div class="span8">
                        <div id="data-visuals"></div>
                    </div>
                </div>
            </div>
        <div id="data-footer"></div>
    </body>

You can probably spot some candidates for possible Handlebars templates now; data-header, data-details, data-visuals and data footer all make sense, where data-header and data-footer could be used on all projects.

Or say you want to quickly create a table to display some information. Using the data file from my earlier example, I can create a Handlebars template to do just that:

<table class="table">
        <tbody>
            <tr>
                
                    <td></td>
                    <td></td>
                    <td></td>
                
            </tr>
        </tbody>
    </table>

Wrap up

As an intermediate beginner to the world of web development, and entering my fifth year of being an “online guy” in a newsroom, I’ve found Handlebars to be a lot of fun. To increase that fun, there are all kinds of add ons and helper functions that you can use. swag.js might be the most fun thus far.

See while Handlebars tried to keep logic out of the templates – and I respect that – there some things I’d like to be able to do, and swag.js allows for some of that. For instance, if in my table example I wanted the title to be in all-caps I can do this with swag:

Or if I want to evaluate the data in my flat JSON file, I can run comparisons on my Handlebars templates.

        
            <td>The Source is </td>
        
            <td>No Source</td>

What I learned back at BuildMadison and in making CitySolver...

2012-12-31T00:00:00-08:00

Back in September I had the chance to participate in an event called Build Madison, a weekend event that brought tinkerers, hackers and idea people together to build things.

At the event, myself and Paul Klemstine and another took up the challenge of building on a pitch to come up with a way to submit quality of life issues – graffiti, potholes, sidewalks in need of repair – to the city of Madison from a mobile device. Pictures were a must, as was the ability to add a user’s location.

The Build Madison pitch also positioned this project as something that would tie in with the city’s existing ”Report a Problem” infrastructure.

Paul and I developed had a good time hacking together a project we called City Solver… It could aspire to be more – a foundation for an open-source See, Click, Fix or a tool for smaller municipalities – or less and left to lessons learned, of which there were many that apply to many things I find myself involved in on a daily basis.

From different angles and perspectives keep asking, “Is this a problem that needs to be solved?”

The original pitch wanted to tie an application into the city of Madison “Report a Problem” system. Needless to say that is an exponentially difficult task, and one not easily tackled by a hackathon team with 24 hours to work with.

And nevermind that the city of Madison has a comprehensive website that city residents can use to report specific issues and wasn’t looking to extend the system in the manner the pitch stated.

Finally, I knew that Open Data was on the horizon in the city, as the city council had recently passed an ordinance that would let information start following for developers to use.

In the end, our project ended up with a couple prizes from the event, and I had the chance to meet with city officials on two occassions to discuss CitySolver courtesy of Eve Galanter, a former city council member who pitched the project.

The conversation were fruitful as we were able to gain some insight into future city plans in this “Report a Problem” arena, some of which include mobile access and user feedback that we had in mind.

And as soon as the Open Data starts following, real problems might be made available to develop around.
Know who else has “solved” or attempted to “solve” this problem.

Had in been better in touch with the work being done by Code For America, I would have known that developers have open-sourced similar crowd-sourcing applications and made code available that we could have built upon.

That said, as an intermediate beginner, building something myself was very fufilling and helped increase my confidence about what I was capable of learning.
Spend more time on knowing the following:
- Who are your users?
- What are their needs?
- What can we do for them?
Some advice from Code For America’s Kevin Curry helped focus ideas on this project further, when he suggested the following after I posted to a Google Group:

The key element of your plan is residents working together to close their own tickets. That’s significant. I think you should base everything off of that and make it totally obvious this is not a tool for reporting problems about the city that only the city can fix.

To Kevin’s point, after thinking more about this project as something “long-term” I was able to come up with the following:
- City residents have the power, know-how and – I believe – a want to work together to solve some common problems.
- That said, that power and ability is limited when a distinction is made between a resident’s fence that is in disrepair, a street with potholes and a malfunctioning street light.
- Success in building a coalition-based platform might best come if based around a singular idea.
- Privacy is a real concern and offers a very real hurdle to overcome when considering the idea of neighbors reporting on issues with neighbors.
That last one is a biggie, hence even if everything at this point is coming up roses, would anyone use this thing?

There are privacy aspects to keep in mind, and very real hurdles to overcome when considering the idea of neighbors reporting on issues with neighbors. For instance, “Thank you, I’d like to repair my broken fence, but I’m without work and my wife is sick and I haven’t made the time, so thank you for posting a photo of it online with your brutish comment.”

All that said, what follows are the bullet points from the BuildMadison presentation.

Build Madison Presentation Bullet Points

City Solver is a multi-device platform that allows everyday citizens to submit information about quality of life issues and problems they feel a municipality could or should address.

Whether using a web app, a native Android app, or their laptop, desktop computer or tablet, city residents could send images and descriptions of these issues and opportunities.

A rating system is Phase One of furthering the platform to allow neighborhood residents to band together to solve issues or to clean up the neighborhood.

Features

Native Android app pulls geolocation and user information to auto-populate data fields, and sends photo and photo location to a database.
Thumbs up and thumbs down ranking of issues to gauge interest from others.
Interface thats works on devices and browsers.
Photo uploads work on Android and laptop, desktop web browsers. Photo uploads will be possible for users on iOS 6 Safari.

Wishlist

Coalition building. Sign up to band together to fix this problem.
Use geolocation of an issue to query against aldermanic districts to add more context, or partner with OpenBallotBox?
Allow user to submit information via SMS
Use email to submit information.
Full screen map with geolocation to find issues near you.
Filtering & Search.
Admin view of database to assign issues, mark as resolved or not feasible and communicate back with user if needed.

Quick snippets from building off LA's most dangerous intersections map

2012-12-30T00:00:00-08:00

tl;dr - Google offers a Streetview static image API, which will render a streetview image – if it exists – when given a latitude/longitude pair. Using this feature seemed like a good add on this map app of ”dangerous intersections,” which is on a new responsive project template developed by SCPR’s Sean Dillingham. We also found a code snippet that will keep the centerpoint of the map when the viewport is resized.

Back in October – before I started at Southern California Public Radio – Kim Bui began asking the audience what they viewed as the area’s most dangerous intersections.

To date, the map has pulled in more than 140 submissions from across the Los Angeles Metropolitian area, including more than 100 in the maps first week of existence.

Kim and I were able to breathe some new life into the presentation when – a couple weeks back – the city of Los Angeles began to take steps to make 53 intersections it identified as “dangerous,” safer for pedestrians.

I took the newspeg and new information as an opporunity to move the map away from a Fusion Tables embedded iframe map to a “new for SCPR in the last month” project template that is responsive. While we’ve been able to kind of get an idea if the audience and the city are on the same page when it comes to the dangerous intersections, I think we’re still exploring the possibilities here.

Looking at the map now, I think adding the ability to guide the user through the city’s ranking of intersection would be a helpful enhancement for the user, where through a table or a slider or simply links in a div. I also need to get better at where and how information is presented. Most of my data map mashups – indeed most of all data map mashups I guess – follow a similar pattern, so there’s also some room to play and learn there. And I’d like to create a re-usable style for the Google form on which we gather user submissions.

A couple fun experiments with this map did lead to some learning however. For instance, I didn’t realize Google offered a Streetview static image API, which will render a streetview image – if it exists – when given a latitude/longitude pair. On advice from Eric Zassenhaus we added this to the map presentation, so that when a marker is clicked, an image approximate to that intersection is returned. Not all work as well as others, but more is added than subtracted I suppose.

Using the API is pretty straightforward; it’s just an encoded URL string inside an image tag, which makes it easy to manipulate and added data from user input or from a dataset.

When combined with a Fusion Tables layer click event in the Maps API like this:

google.maps.event.addListener(cityCrosswalkLayer, 'click', function(e) {
        jqueryNoConflict('#toBeRemoved').html(
            '<img src=\"http://maps.googleapis.com/maps/api/streetview?size=300x300&location=' + e.row['Lat'].value + ',%20' + e.row['Long'].value + '&fov=90&heading=235&pitch=10&sensor=false\" />');
    });

You’ll get something like this when the user clicks a map marker:

This presentation also includes a short little code snippet I found that will true the map’s centerpoint as the window is resized. Minor stuff I know, because with responsive design the centerpoint is set when the page loads and screen size is determined, but should a user flip from landscape to portrait – or vice versa – this can keep a thing from becoming a thing.

Including this is ultra simple. The function is simply:

// function to maintain center point of map
function calculateCenter(){
    center = map.getCenter();
};

The function can then be called in the same function that creates the map:

google.maps.event.addDomListener(map, 'idle', function() {
  calculateCenter();
});

google.maps.event.addDomListener(window, 'resize', function() {
  map.setCenter(centerLosAngeles);
});

In this case, centerLosAngeles is a variable for my starting centerpoint specified in my map options.

Link: Sunlight from the Command Line

2012-12-30T00:00:00-08:00

Are you as big of a fan of Paul Tagliamonte as I am? If so, then you are well aware of python-sunlight, his awesome, comprehensive Python API client for Sunlight’s APIs. The latest release includes a command line interface, or CLI, so you can interact with the Sunlight APIs directly from the shell.

via sunlightlabs.com

As a beginner I see so much advice in this simple statement...

2012-12-30T00:00:00-08:00

Code linting is your first unit test!
— Eric Sorenson (@dubharmonic) December 30, 2012

Mother Jones, Slate open up a couple stunning datasets related to gun violence

2012-12-29T00:00:00-08:00

In the wake of mass shootings this year, and the journalism that followed, Mother Jones has opened up a dataset of mass shootings in the U.S. between 1982 and 2012. Included are a spreadsheet of the incidents, and a spreadsheet of the weapons used. You can read more about it here.

Meanwhile, Slate has partnered with the Twitter account @GunDeaths to create “an interactive, crowdsourced tally of the toll firearms have taken since Dec. 14” – the date of the mass shooting in Newtown, Conn. that left 26 dead.

Of course, this data is incomplete. Not all reports get caught by @GunDeaths’ news alerts or his followers. Suicides, which are estimated to make up as much as 60 percent of gun deaths, typically go unreported. Nevertheless, we at Slate want to assemble this data as best we can.

Slate too has opened up the data and made it available for download.

Link: Anyone can do it. Data journalism is the new punk

2012-12-14T00:00:00-08:00

1) This is a dataset

2) Here’s another

3) Here are some free tools

NOW BE A DATA JOURNALIST

OK, it lacks a certain 1976 grittiness, but the theory is there. You don’t have to be a developer or a coder to be a data journalist.

via guardian.co.uk

When I first read this back in May I thought I was reminded of a Greg Linch post, or “three-chord song for journalists,” a phrase I used in a project pitched during last summer’s Knight-Mozilla Journalism Learning Lab.

I certainly didn’t think for one second that I was less than three months away from being approached about, interviewed for and offered a position as a data journalist.

Reading it now, and in combination with this thought-starter from Heather Billings and the resulting conversation…

When a guy’s dev skills fall short, does he have “I don’t deserve this job” thoughts? I’ve only observed it in myself+female coder friends.
— Heather Billings (@hbillings) December 14, 2012

…I am more convinced than ever that while I have learned that I can learn how to find solutions to some elementary things, in order to “graduate” into someone who practices a “craft” in this field I will need to double down on time and practice and time and practice.

Link: John Keefe on Designing a Course in Data Journalism

2012-11-26T00:00:00-08:00

When asked what such a course might look like, I point folks to Brian Boyer’s theoretical and amazing Hacker Journalism 101. Journalism needs more journalists who can code, and this is a great way to get there. …
In addition to programming, here are some of my favorite topics for classes, readings or workshops:

Finding data for your stories

Finding stories in your data

How to tell one story well

All data is dirty … and what to do about that

Basic stats

Percentage points for journalists

Mapmaking made easy

Lying and truthing with easily-made maps

When maps shouldn’t be maps

Basic chartbuilding

Lying and truthing with charts and graphs

Did I mention programming?

via johnkeefe.net

It’s up to the individual to find what works for them, but that said, what worked for me was to first learn code to visualize data followed by learn code to gather data.

Now I need to learn code to analyze data while refining and bettering my skills in the first two areas, especially developing a better eye for visualization and when not to use the hammers I’ve acquired…

Using csvkit and PostgreSQL to mimic a bit of Microsoft Access

2012-11-25T00:00:00-08:00

Over the last couple weeks I had the first real chance to take a really large dataset and explore it using tools like csvkit and PostgreSQL.

Needless to say I learned a lot, and with that learning came a bit of backtracking. But something I have learned to do as a matter of course now is to document every step of a project – whether coding or writing. The extra effort not only comes in handy when creating a workflow, but offers plenty of advice and ideas for others who are also learning. What follows are an edited/revised version of the notes from my scratchpad file.

This dataset came in 15 text files – five main files and two sets of five text files related to the main set through a common ID. In total there were some 125,000 rows. I also received a “column key” describing the name of each field, the data types for each filed and what designations stood for.

Without access to Access, and after several attempts to use LibreOffice Base to create three tables in a database and JOIN them based on common ID – crash, crash, crash – I sucked it up and learned how to use csvkit and PostgreSQL. Boy am I glad I did. And boy am I thankful for Christopher Groskopf’s csvkit and his imagination about what is possible, and the ability to progam it.

Thinking about it now reminds me of something Mark Ng said duing #NICAR12 regarding Chris: “He’s from the future”

As I mentioned, I attempted to document each step, query, failure, success and learning moment, so while this is very much a how to, it is also very long, winding and potentially confusing to some. That said, if you like journo-tech-nerdery – or what to offer some tips – please enjoy.

Getting started with some tools

csvkit is a command-line toolset to explore CSV files. It can be installed into a virtual environment which isolates it from key system packages:

sudo pip install virtualenv
virtualenv venv
source venv/bin/activate
pip install csvkit

It can also be installed globally:

sudo pip install csvkit

If you’re beginning on the command line, on the suggestion of Joe Germuska, this is a great resource to learn from.

Preparing the data for exploration

After importing into Google Docs failed – the dataset was too larger – I opened one of the main text files in LibreOffice to see what I was working with.
The file came without headers so I pulled the first few rows into a new CSV file, imported that into a Google spreadsheet – easier to type in – and used the accompanying key to enter in column information.
I then exported the header row as new file, opened each of the data files in a text editor, pasted the header to the top of the file and saved as CSV. I repeated this step for all 15 files. (There’s a time savings to be found here!)
I fired up csvkit and ran csvclean on each of the 15 files to look for syntax errors or bad rows.
I tried to use csvstack to combine the files, but I ran into errors. That I couldn’t decipher. So instead I manually combined each of the CSV files into one large CSV file with one row of headers. (More time savings to be found!)
I dropped the three combined CSV files into my “data_inbox” and fired up a script based on one authored by Jeff Severns Guntzel that:
- Makes a backup of the data files.
- Creates a new project directory with a specified folder structure.
- Audits the CSV using csvkit’s csvstat function and creates a data-abstract file.
- Moves all of the files into the new project directory.

Then I opened up my data-abstract-file to see what I had. Here’s an example of what I was able to see at a glance:

47. type_of_collision
     <type 'unicode'>
     Nulls: False
     Unique values: 10
     5 most frequent values:
         B:  50090
         C:  33952
         E:  16700
         D:  13184
         F:  3913
     Max length: 1

In this case, B, C, E, D and F correspond to a specific value in the column key. Thanks to some magic from Chris and Jeff, at a quick glance I was able to decipher values in every column of the data file and pass information on to the editor for this story.

You can use a csvkit function to do this on an individual column as well, substituting the column_number and file_name in the example command below.

csvstat -c **column_number** --freq **file_name**

Making the deep dive

Once I learned we wanted to focus on a specific type of record I used csvkit’s csvgrep (search) function to search the rows for that value and output the results to a new file which was only about 4,000 odd rows or so.

csvgrep -c 48 -m "A" **file_name** > **output_file_name**

I then double-checked my work to make sure I had the expected number of rows.

csvstat **output_file_name**

The number of rows matched up, but something had changed that would impact my later efforts which I’ll come back to.

Primary in my mind was figuring out a way to JOIN the three datasets together based on their common ID. This is the part where I googled for open source access and LibreOffice Base How-To and struggled to make it work as the application continued to crash when I tried to import the larger secondary file.

Creating django models and relating the data that way came to mind, and then it dawned me that I have mySQL and postgreSql installed in my local development environment and I might as well use them for what they’re supposed to be used for.

Tools revisited

Some kind folks at Heroku have created a one-click installer for PostgreSQL on a Mac. I had gone through the Homebrew route, and while I eventually made it work, the one-click installer worked really well. Just had to open my .bashprofile and set the correct $PATH.

There is great PostgreSQL documentation that walks through the basic use, and I also installed pgAdmin III which is a GUI to explore the databases, which is good for a noob like me.

To get started creating a new database in my local PostgreSQL install I:

Created a new user.
Created a new database for the user.
And then logged into the database I made using the PostgreSQL command line application.
Along the way I learned some additional commands in the PostgreSQL command line application.

Activate PostgreSQL

psql

List databases

\list

See if table has been created

\dt

Drop table

DROP TABLE **name of table**

Quit psql

\q

I then connected pgAdmin III to my local PostgreSQL instance by specifying the user and leaving the server address blank.

And then the fun began…

Using PostgreSQL like Access

So the goal of the project from the beginning was to take the three data sets and JOIN them together based on the common ID. Forgoing Access and using PostgreSQL meant I had to craft CREATE TABLE statements. Little did I know at the time, someone had already thought of this and developed a crafty solution.

But I’ll come back to that as well. To create the statements I grabbed the header rows from the three files, added data types and crafted a CREATE TABLE statement. So still in the the PostgreSQL command line application I added something like this following:

CREATE TABLE **table_name** (case_id integer, special_information_1 text, special_information_2 text, special_information_3 text);

pgAdmin III told me it wanted each table to have a primary key before I could edit them, so I spent some time figuring out how to do that on a table using something called a sequence.

ALTER TABLE **table_name** ADD COLUMN record_id serial PRIMARY KEY;

Then I wanted to upload my CSV files into the PostgreSQL Database table. I found a couple methods – I liked this one.

COPY **table_name** '**path_to_file/**file_name**' DELIMITERS ',' CSV;

Then I learned an important fact:

“If you don’t have access to the server, you can use psql’s \COPY command which is very similar to COPY FROM but works with local files. See the manual for details.”

\COPY **table_name** '**path_to_file/**file_name**' DELIMITERS ',' CSV;

And then that didn’t work either. I later learned another important fact: that if your csv file has a header row, well you need to tell PostgreSQL that, hence this command:

\COPY **table_name** '**path_to_file/**file_name**' DELIMITERS ',' CSV HEADER;

That worked… because the CSV HEADER designation told PostgreSQL that the CSV file had a header row.

And then I learned that csvkit – wonder of wonders – can not only generate CREATE TABLE statements from a CSV, but it can freaking create a database and upload a CSV to a new table in that database with a simple command. Heck, it can take a whole folder of CSV files and create tables from them. Back to Mark Ng: “From the Future.”

Using csvkit with PostgreSQL (1)

So in learning that I can use several csvkit to do what I had done above, I set out to create the project again using only the utility. My notes are also more streamlined as I have a better idea of how to achieve my end result.

With PostgreSQL up and running, you can use csvkit to create a database.

createdb **name_of_database**

From there, you can use csvkit’s csvsql command to generate a CREATE TABLE statement based on your CSV file.

csvsql -i postgresql  **file_name**

Running the above command in the terminal left me with something like this that I can use in the PostgreSQL application:

CREATE TABLE "**name_of_table**" (
        case_id INTEGER NOT NULL,
        special_information_1 VARCHAR(1) NOT NULL,
        special_information_2 VARCHAR(1) NOT NULL,
        special_information_3 VARCHAR(1) NOT NULL,
        other_associated_factor_1 VARCHAR(1) NOT NULL,
        other_associated_factor_2 VARCHAR(1) NOT NULL,
);

But you can also use csvsql to create the database and import data from a CSV directly into PostgreSQL. Make sure your virtual environment has psycopg2 or MySQL-python installed depending on your choice of database. Use pip install psycopg2 or pip install MySQL-python

createdb *name_of_database*
csvsql --db postgresql:///*name_of_database* --insert *name_of_file*
csvsql --db postgresql:///*name_of_database* --insert *name_of_file*
csvsql --db postgresql:///*name_of_database* --insert *name_of_file*

And best of all, you can use csvsql to create tables for an entire folder of CSV files and create PostgreSQL tables based on those files.

createdb *name_of_database*
csvsql --db postgresql:///*name-of-database* --insert *path-to-file*/*name-of-file*.csv

Log into PostgreSQL application from the command line and add primary keys to the tables after they are created per pgAdmin III.

psql *name_of_database*
ALTER TABLE *name_of_table1* ADD COLUMN record_id serial PRIMARY KEY;
ALTER TABLE *name_of_table2* ADD COLUMN record_id serial PRIMARY KEY;
ALTER TABLE *name_of_table3* ADD COLUMN record_id serial PRIMARY KEY;

Learn to create a join on two tables using case_id. Great visual guide is here.

SELECT * FROM *name_of_table1*; (4662 rows)
SELECT * FROM *name_of_table2*; (95716 rows)
SELECT * FROM *name_of_table3*; (257333 rows)
SELECT name_of_table1.*, name_of_table2.* FROM name_of_table1, name_of_table2 WHERE name_of_table1.case_id = name_of_table2.case_id; (5906 rows)

Export to a new file, I’ll call it joined_new_file.csv.

Load file as new table

csvsql --db postgresql:///*name-of-database* --insert *joined_new_file.csv*

Test various JOIN methods to get proper matching of records. This case records in Table 1 could have several corresponding records in Table 2. So I went with a LEFT OUTER JOIN.

It was about this time that I learned that the data type of a column in my table had changed from to . I was able to trace the changce back to something that happened when I used csvstat.

I filtered the file to give me all the rows with a single value. In this case it was the letter “A”. When I created a new table based on that filter I got the . Some helpful advice from Adam Hooper on the NICAR listserv gave me the how’s and why’s.

In CSV, every value is a string–no exceptions. csvkit “intelligently” guesses at a column type by looking at every string value and seeing if they all follow the same pattern.

For instance, if every value looks like “12:23:31”, it’ll pick a time format.

If you have 200,000 rows, and one row has a value that isn’t a time format, csvkit will (correctly) call it a “string”. But if you remove the row that isn’t formatted like a time, then csvkit will (correctly) call the column “time” in the resulting CSV.

I suppose you’ve read this far and gone, “yes, but why would it think ‘A’ is a datetime?” And the answer is … (wait for it) … “A” is a time value! “A” is short for “AM”, so it gets parsed as 12:00 a.m. on the morning the program is executed. (The same trick doesn’t work for “P”, because of a bug in the “dateutil” library csvkit depends on.)

And since every value in column 48 is now ‘A’, every value is a datetime – meaning the entire column is a datetime column. (This wasn’t the case before, because there were non-“A” values in column 48, so csvkit didn’t interpret it as a datetime.)

And for my purposes I didn’t really need column 48 to be the correct data type. In this case, the data was all of the same record. But because it had changed on me I began to wonder what else might have changed on me.

Regardless, I figured I’d give it another go just to streamline things a bit.

Using csvkit with PostgreSQL (2)

Run csvstat to make sure data types are intact.
Use csvkit to create PostgreSQL database and tables from an entire folder of CSVs.
Query working_accidents.csv for rows in which motor_vehicle_involved_with = non-collision and save as working_non_collisions.csv.
Generate CREATE statement for working_non_collisions.csv to upload into PostgreSQL.
Add data from csv into the working_non_collisions table. The CSV HEADER designation says its a CSV file with a header row.
Query for the related records on working_non_collisions and working_party tables to get expected output.
Run INNER JOIN to see of same result occurs.
Export as accidents_non_collisions_and_parties.csv. Open in LibreOffice and delete the duplicate case_id column.
Generate CREATE statement for accidents_non_collisions_and_parties.csv to upload into PostgreSQL. In this instance there was a duplicate column for case_id that happened when I ran the select statement. Also changed motor_vehicle_involved_with to VARCHAR(1) and oaf_violation_code VARCHAR(4).
Add data from csv into the accidents_non_collisions_and_parties table. The CSV HEADER designation says its a CSV file with a header row.
Query on connections between accident & parties and victims. But then I realized this ignores accidents in which there was no victim.
Test various JOIN methods to see different results. I want all records from accidents_non_collisions_and_parties and any that match in working_victim, which I learned is a LEFT OUTER JOIN.
Export as joined_accidents_non_collisions.csv. Open in LibreOffice and delete the duplicate case_id columns.
Doublecheck to see that motor_vehicle_involved_with column hasn’t changed to date-time type.
Generate CREATE statement for working_non_collisions.csv to upload into PostgreSQL.
Run various queries from within pgAdmin III

California dreams... they're a bit crazy...

2012-11-16T00:00:00-08:00

Been having the craziest dreams since moving out west…

So last night in my dreams they were releasing Halo 27 or something like that, and all day people were getting ready for it, including our ex-landlords who were packing up their house.

And as a part of it video game release they had a live TV special to celebrate it, and some guy in a suit was talking about the video game and then he opened this huge door and from far away you saw this figure that looked like the character in the Lord of the Rings: The Two Towers – the guy running towards the Deeping Wall who blows it up with a torch – and he actually runs through the TV into real-life. He took out my dad and my uncle and suddenly there’s like millions of this guy running around the world.

Next thing I know I’m out in the street, and most people are dressed up in camo and they have weapons and somehow I have a weapon and a guy from high school sees me and shouts – he’s got a weapon – and I must have missed something by not playing the game because if you didn’t see the secret message at the end of Halo 26 you weren’t going to be prepared and couldn’t be after the fact.

So I eventually surrender my weapon and then I wander around. Eventually I call some guy I used to work with to ask what happens next and he said he was at home getting ready for the electric sky show and I hung up bewildered. And then the electric sky show started and I said holy crap… and then I woke up.

What?

Fantastic thoughts to live -- and work -- for

2012-10-09T00:00:00-07:00

Working longer is not working harder. Go home.
40 hours includes the commute.
Resist the urge to call a meeting.
Consensus is bullshit. Do what’s best for the company.
We will not ship shit.
If it’s hard to use, it hurts the product, the company, and our reputation.
You will do customer service for the products you create.
We will work in public. Our code, our designs, and our ideas are not our business. We are our business. Our products are our business. Customers are our business.
People are expensive. We will strive to keep individuals working for the customer, and not on the business.
Efficiency is everything. We will remove obstructions.
We will not celebrate. Every day will be awesome, or we stop what we’re doing and make it awesome. Cake is for gazelles, and we are lions.
We will enjoy our work, our products, and our lives. We will make work awesome.
We will be lean. We will NOT ignore problems, and we will swarm when issues arise.
You are an adult. You will manage your own machine and tools.

Via stefankendall.com: Core Values.