Kind of reminds me of the scene in Adventures in Babysitting after the subway fight when Keith Coogan’s Brad needs “just one stitch.”
]]>But something happened around the middle of April. The script pulled in 389 tweets on April 16, but the following week only caught 100. Same for the week after.
I did some hand-holding for a few weeks, but more and more of the same occurred.
#wjchat-transcript news: @ChrisLKeller & I are aware we’re a couple weeks behind. Pipes running the transcripts are a little clogged. Soon!
— P. Kim Bui (@kimbui) June 20, 2014
The script — which was using the Tweepy library — was missing the first hour or so of tweets, and leaving the archive incomplete. Time for a refactor.
I’m not blaming Tweepy but I wanted to see what else was out there. I tried making a go with requests and oauth but I don’t think I’m ready yet. But the twitter library seems to work well.
Tell you the truth: the library didn’t matter. It comes down to how tweets are returned by the search API — descending order — the maximum number of results per page — 100 — and this thing called the max_id.
The max_id is what amounts to “paging” it seems for the Twitter API.
The solution to the issue described above is to use a technique for working with streams of data called cursoring. Instead of reading a timeline relative to the top of the list (which changes frequently), an application should read the timeline relative to the IDs of Tweets it has already processed. This is achieved through the use of the max_id request parameter.
So the best I could get in a single search after April 16 was 100. And once I had 100 I had no means of moving on to the next page. Thankfully tonight I figured out pretty fast how the max_id parameter can work.
The script is based on the #wjchat script, and takes a hashtag and a search start date as arguments and outputs the results as a csv. It requires the following packages…
pip install python-dateutil==2.1
pip install pytz==2013b
pip install twitter==1.14.3
Putting it all together
And so I should probably get down to the job of linking to some of these things that I’ve been holding on to in Evernote, etc. with the idea of someday offering thoughts about others’ thoughts.
via Charlie Kindel: Don’t Make Your Team Say No To You
I’m an idea guy. Ideas come to me a mile a minute. At that point in my career I didn’t realize how disruptive it was to my team that I was spouting these ideas to the team while they were executing on the current plan. In my head, I was just talking about potentialities for the future; by telling the team about all the cool things we could do in the future, I was showing “vision”.
What I found out later, when talking to people who had been on that team, was they viewed me as a “randomizer” they needed to control. In other words, the team spent time and energy MANAGING THE MANAGER. I forced them, regularly, to say “No” to ME.
If you are a leader of an early-stage venture, you need to figure out a way to “vent” your ideas that has NO impact on your team. Here are some tactics I’ve used and seen others use that might help you do this.
via Erik Paulson: The next AppleTV, and the future of newspapers
At their core, local TV news and newspapers are fundamentally reporters. However, I believe that it would be easier for a newspaper staff to put together the content for a TV broadcast than a TV station could put together the content for an issue of a newspaper.
via Mark Surman: Mozillians as inventors
]]>Admittedly, we don’t know how to do this at scale yet. The recent MoJo learning lab reached only 60 people, and was too closely tied to whether one became a fellow or not. But we did catch a glimpse of what might be possible through MoJo events and discussions that happened trough the challenge cycle. The idea of inventing new web things for the newsroom galvanized people, got them sharing ideas. It had people teaching and mentoring each other even if they didn’t know it.
]]>Imagine your newsroom has been pumping out articles about the papal election, yet it turns out that the article readers are clicking on is one about the civil war in Syria. Thank God, you can finally stop writing about the goddamned pope because people don’t care anyway. You can commission a new piece on Syria, perhaps even tailored to people’s exact search terms so you’re not just writing about what they care about, you’re answering their questions too. I think that’s incredibly useful information to have, information you can act on, right now.
Except I’ve never seen a news organization that has a workflow that would allow them to routinely respond to their readers’ behavior right now. Content farms seem to be the only content producers with that capability.
In addition to serving as the assignment editor for the South Suburban edition of The Times, I was the edition’s primary crime reporter — partly out of necessity, partly out of enjoyment and partly out of the one-man-band syndrome I suffer from.
Each day I would add new cases, determine which cases had upcoming court hearings and update older information. Every. Day.
They were meticulous. And inefficient and repetitive.
Naive to guiding lights out there in the world that could show me a better method — one that would allow me to glean insights from this mountain of unstructured data I had collected, or at least lend structure to it — I did what worked.
To this day I think about what I could have done, and now what you — the web developer, technologist or data analyst — could do in a similar situation. Better yet, what could you do every day in a newsroom as part of the Knight-Mozilla Fellowhip program.
The short answer: solve problems and show folks new ways of dealing with repetition and inefficiencies.
Fact: there are mountains of stories, charts, graphs, news applications and much more sitting with every reporter, writer and editor in every single news organization in this world. Some are mundane. But some could change the way society thinks, lives and plays.
Here’s another: that information will sit there for any number of reasons, but I’m inclined to think that lack of imagination is not one of them. There are few too people to demonstrate a better, more efficient method. Far too few people around to develop a plan. And that’s where your help matters.
I’m kind of ashamed that my curiosity never led me to a solution that was better than adding the name of the person arrested, their date of birth, the charges against them and other information to a text file.
I say that because I’ve always loved systems. Whether as a dishwasher, the newsstand manager at a Barnes & Noble or in my first jobs as a journalist at newspapers in very small Wisconsin communities, to me the foundation for how the job is done was as important as doing the job itself.
Certainly I was oblivious to other ways. And many other journalists — who write about the inefficiencies of government or health care or the courts and then do their jobs which are part of a larger system of repetition and routine — are as well.
For me, it took learning “what it means to be a developer”, and thanks to the helping hands and support of others — way too many people to link off to — I have learned lessons that are more important than any code I have written.
These lessons have very little to do with classes, functions and syntax and everything to do with helping to reinforce the core mission of journalism: hold those in power accountable, help people make sense of the world around them and celebrate their place in it.
These lessons are directly applicable to the job descriptions that reporters, editors and web producers have at news organizations large and small where there are obstacles just waiting for solutions of the technical or automated variety.
The solutions could be as “simple” as extracting information from PDFs in a manner that doesn’t require four hours of copying and pasting. Maybe they are as “complex” as combining four or five daily processes that have been duct-taped together through Google spreadsheets, emails, meetings and in-person conversations.
Or maybe there’s simply someone out there with a bunch of text files, waiting for another way they didn’t know they wanted.
Convinced? Become one of the five people who will spend 10 months combining code, data and journalism as a Knight-Mozilla Fellow. Go and apply to become a 2014 fellow today.
Not convinced? Then read what these folks — to whom I offer eternal thank yous — have to say about creating code in newsrooms in the service of journalism. Then just fill out the application. You have nothing to lose and everything to gain.
<select id="searchString" onchange="zoomToAddress(this.value);">
<option value="">--Select All--</option>
<option value="43.138092,-89.747988">Luckenbooth Cafe</option>
<option value="43.017218,-89.831479">Aunt Mary's Hooterville Inn</option>
</select>
So we want to convert the above strings to numbers that the Maps API can use. We’ll place the following inside a function called zoomToAddress.
First we get the value from the option menu as a string and set it to a variable.
var searchString = document.getElementById('searchString').value;
Then we set a variable to find the position of the comma in the string.
var commaPos = searchString.indexOf(',');
Use parseFloat to conver the string to a number while parsing out the Latitude value.
var coordinatesLat = parseFloat(searchString.substring(0, commaPos));
And we do the same for the Longitude.
var coordinatesLong = parseFloat(searchString.substring(commaPos + 1, searchString.length));
Those variables are combined into a new google maps LatLng pair.
var centerPoint = new google.maps.LatLng(coordinatesLat, coordinatesLong);
Then we can use the setCenter() method from the Maps API.
map.setCenter(centerPoint);
]]>I’m not big on offering advice when it comes to learning how to code. Everyone learns differently and has different goals; my experience isn’t necessarily instructive. But I seem to be getting asked the same question more and more often: someone wants to be able to make cool things for the web, and they don’t know how to get started. Here are some thoughts on how to keep your head on straight while you’re trying to learn. Take them all with a big grain of salt.
We successfully powered the map using a bunch of Google tools — a form, a spreadsheet and a Fusion Table — along with some duct tape and some manual labor.
Some six months later I’ve become more comfortable here, and more comfortable with Google Apps Scripts thanks to some great examples from others. So as I re-purposed the idea for an election back in May, I wanted to see if there wasn’t a certain level of automation that I could add to the project.
I didn’t want automation for the sake of automation, but rather to eliminate the repetitive and inefficient parts of the process, while still ensuring accuracy — cause that only makes sense in journalism.
How It Was Done – Nov. 2012
The duct tape was necessary because even though forms and spreadsheets and Fusion Tables all exist within Google’s Drive application, the level of integration between a form and a spreadsheet and a spreadsheet and a Fusion Table remains basic.
Forms can send data to spreadsheets. Fusion Tables can be created from spreadsheets. But a straight data pipeline from a submitted form to a Fusion Table doesn’t explicitly exist. There are methods and scripts that can send form data to a Fusion Table, but in this method once the form and table are created you’re locked in; you can’t add additional fields on the form or table.
The need to perform simple calculations, comparisons and geocoding of submitted data meant there was a level of manual labor.
While Google’s geocoder will do the best it can to find the latitude and longitude of a location, it will only what you tell it to do. So I wanted to make sure the address was geocoded properly before it appeared on the map. And because we wanted different colored markers to distinguish between voting experiences, we need to perform a comparison on the submitted data.
And besides, it was my second day at a new job and surely didn’t want one of my first projects to flop.
How It Was Done – May 2013
I still used a Google form to populate a spreadsheet, but tied a lot of the automation to a function that runs when the form’s submit button is pressed.
This one — cribbed from an Andy Boyle walkthrough — will send an email with the information that was just submitted.
The script sets each column to a variable; e.values[0] in this case represents the first column in the spreadsheet…
function onFormSubmit(e) {
var emailToAddress = 'ckeller@scpr.org';
var submissionTimeStamp = e.values[0];
var submissionPollingPlace = e.values[1];
var submissionExperience = e.values[6];
var submissionNarrative = e.values[2];
var submissionFullName = e.values[3];
var emailSubject = 'Voter issues map submission from' + submissionPollingPlace;
var emailBody = submissionTimeStamp + '\n' +
'Voting at ' + submissionPollingPlace +
' was ' + submissionExperience + '.\n' + submissionNarrative +
' -- submitted by' + submissionFullName;
MailApp.sendEmail(emailToAddress, emailSubject, emailBody);
// run the data processing functions
runSpreadsheetFunctions();
};
I’ve piggybacked a runSpreadsheetFunctions() function inside this piece of code. I don’t know if it’s right and proper, but I want a whole bunch of actions to take place in succession when a form is submitted.
One is a function to run a user-submitted address through the geocoder and add the latitude and longitude of that address to a separate spreadsheet cell:
function geocodeAddressFromCell(){
var sss = SpreadsheetApp.openById(spreadsheetID);
var ss = sss.getSheetByName(workingSheetID);
var lastRow = ss.getLastRow();
var addressCellData = ss.getRange(lastRow, 2).getValue();
var results = Maps.newGeocoder().geocode(addressCellData);
if (results.status == 'OK') {
var bestResult = results.results[0];
var lat = bestResult.geometry.location.lat;
var lng = bestResult.geometry.location.lng;
var latLng = lat + ',' + lng;
var addressTargetCellData = ss.getRange(lastRow, 9).setValue(latLng);
} else {
Logger.log(results.status);
return '0,0';
}
};
This script will open a spreadsheet based on ID and sheet name, find the last row on the specified sheet and find a value in a specific column
var addressCellData = ss.getRange(lastRow, 2).getValue();
and run that value through Google’s geocoder and output a latitude/longitude pair
var latLng = lat + ',' + lng;
That latitude/longitude pair is assigned to a variable and added to “geocoded_address” column in that last row
var addressTargetCellData = ss.getRange(lastRow, 9).setValue(latLng);
While we’re at it, let’s create a function that will allow us use of one map marker if the voter experience is positive, and another if the experience is negative. We’ll do this by adding an =IF() spreadsheet formula to a cell.
function evaluateVotingExperience(){
var sss = SpreadsheetApp.openById(spreadsheetID);
var ss = sss.getSheetByName(workingSheetID);
var voterExperienceColumn = "G";
var voterMarkerIdColumn = "H";
var lastRow = ss.getLastRow();
var voterExperienceCell = voterExperienceColumn + lastRow;
var voterMarkerIdCell = voterMarkerIdColumn + lastRow;
var voterExperienceFormula = "=IF(" + voterExperienceCell + "=\"Negative\", \"large_red\", IF(" + voterExperienceCell + "=\"Positive\", \"large_green\"))";
var ssCellToGetFormula = ss.getRange(voterMarkerIdCell);
ssCellToGetFormula.setFormula(voterExperienceFormula);
};
Again, we’ll open a spreadsheet based on ID and sheet name, and use our columns and rows to whittle our way down to the cell that holds the value we want to evaluate, and the cell that will hold the result of that evaluation, which will be a Fusion Tables marker type.
var voterExperienceColumn = "G";
var voterMarkerIdColumn = "H";
var lastRow = ss.getLastRow();
var voterExperienceCell = voterExperienceColumn + lastRow;
var voterMarkerIdCell = voterMarkerIdColumn + lastRow;
We’ll create an IF/ELSE formula for our evaluation
var voterExperienceFormula = "=IF(" + voterExperienceCell + "=\"Negative\", \"large_red\", IF(" + voterExperienceCell + "=\"Positive\", \"large_green\"))";
and add that formula to the cell that will hold the result.
var ssCellToGetFormula = ss.getRange(voterMarkerIdCell);
ssCellToGetFormula.setFormula(voterExperienceFormula);
Finally, I added an update script, a series of functions that allows you to trigger a sync between a Google spreadsheet and a Fusion Table.
The update script is incorporated into runSpreadsheetFunctions(), and which is run when a form is submitted, so the Fusion Table is updated automatically when a voter clicks the submit button.
To be safe, I set a “timeout” to make sure the previous functions have run before the Fusion Table is updated.
// run functions that will process the data
function runSpreadsheetFunctions(){
geocodeAddressFromCell();
Utilities.sleep(1000);
evaluateVotingExperience();
Utilities.sleep(1000);
updateFusion();
Utilities.sleep(1000);
};
The full script — minus API keys and such — is below.
Tie it all together
I know what you’re probably thinking at this point: that’s a heckuva lot of work and scripting to do something you could do with PHP, MySQL and a bit of Javascript.
You. Are. Right.
If I had to do it over again I might consider other approaches to this kind of project. But I think I might be inclined to take this approach again.
]]>It’s an often encountered situation, but one that can be a pain to address – merging data from two sources around a common column. Here’s a way of doing it in Google Refine…
A NICAR thread about methods of project management and product development led me to some posts on Agile development which led me to find this gem. Like any piece of knowledge that could be deemed as inherent in some people, it’s always good to be reminded of proper meeting rules.
]]>For those times when you absolutely must have a meeting (this should be a rare event), stick to these simple rules:
- Set a 30 minute timer. When it rings, meeting’s over. Period.
- Invite as few people as possible.
- Never have a meeting without a clear agenda.
]]>Back in 2008, I wrote a blog post welcoming the local civic info site Everyblock to the world. Today, just over five years later, Everyblock has announced that their parent company, NBC News, has shuttered the site. Twitter has been awash with people digesting the news and I suspect that this blog post, written while battling a nasty winter cold, isn’t the first nor will it be the last. That’s what happens when you create something with the impact that Everyblock had: People care when it’s gone.
Kind of reminds me of the scene in Adventures in Babysitting after the subway fight when Keith Coogan’s Brad needs “just one stitch.”
]]>As I wrote in my last post on Posterous, I enoyed using it because it stayed out of my way… I didn’t feel the need to fiddle with the layout, or adjust the admin or mess around with plugins. It let me just write posts… Maybe not as many as I’d like to, but posts nonetheless. And I stuck with it to a certain degree.
I spent some time considering going back to Tumblr, and I found this promising platform called Snipt. Heck I even installed WordPress for spell.
But all in all I like the idea of Octopress, of something that will continue to stay out of my way and just let me write, of something that will lets write in markdown which is then compiled into static HTML.
Time will tell.
I can say Posterous served me well at a certain point in my life, a time when I was learning to learn, code, program and teach/show others, and I’ll have an archive of that. Now it’s time to find something new.
I did take some steps to prepare for a day I figured was coming.
I moved my projects page away from Posterous and organized it bit.
I grabbed an export of all of my content that Posterous housed and moved it to an archive. Now I have some time to figure out how to deal with links that will be broken.
I started to move some learning resources and tutorials to Github and Gists.
I enjoyed using Posterous because it stayed out of my way… I didn’t feel the need to fiddle with the layout, or adjust the admin or mess around with plugins. It let me just write posts… Maybe not as many as I’d like to, but posts nonetheless. And I stuck with it to a certain degree.
Sadly, along with the blogging platform being gone, seems any Google juice I accumulated for things like Fusion Tables walkthroughs and examples will be gone as well… or at least pointing to dead links.
But then that might be for the better. I can say Posterous served me well at a certain point in my life, a time when I was learning to learn, code, program and teach/show others, and I’ll have an archive of that. Now it’s time to find something new.
I did take some steps to prepare for a day I figured was coming.
I formatted my projects page a bit so it’s a little more organized.
I grabbed an export of all of my content that Posterous housed. The Posterous backup made is really easy for me, and I think they way the structured the export should be a model for all content platform companies when they shutdown – ahem, Everyblock.
I started to move some learning resources and tutorials to Github and Gists.
And I started to research some other potential platforms. Wordpress is always there, but it’s too easy for me to get bogged down in how the site looks. Again, more writing and less fuss. Dave Stanton & Robert Hernandez just turned me on to Snipt, which looks promising as well.
O’Reilly’s Alex Howard hosted a Google Hangout on the topic of data journalism. Present were O’Reilly’s Mac Slocum, USA Today’s Anthony DeBarros, NPR’s Brian Boyer, Dave Stanton from SmartMedia Creative, Michelle Minkoff from The Associated Press, Tariq Kokhar from the World Bank, Simon Rogers from The Guardian and myself.
]]>tl;dr - Learning how I can use tabletop.js & handlebars.js together has opened up some really cool custom mini-CMS possibilities and ways to manage data projects.
But thanks to the experiments of others, I have learned that flat json files can be better to use in production, especially for small one-off projects.
While I’d like to eventually write a script to take the tabletop object and write it to a flat file, I made some adjustments to some python code so I can take a csv, create a flat json file from it and drop that file into a handlebars function with minimal effort.
While it works for me, I’m curious to know if it works for others…
I’m happy to say that I still have access to some gray days since moving to Southern California, and on a recent one I set about streamlining some project structures, code snippets & templates for two code libraries that I’ve been using more and more since coming to Southern California Public Radio: tabletop.js and handlebars.js.
I’m only now getting back to the write up and posting code.
tabletop.js allows you to use a Google spreadsheet as a data backend. By simply publishing the spreadsheet and adding its key to a tabletop javascript function you can retrieve an an array of objects to be used on a webpage or in a web application. The library was released back in February – right around the time of NICAR – thanks to work by WNYC and Balance Media.
handlebars.js is a templating library – much like mustache.js – that “provides the power necessary to let you build semantic templates” based on data that is formatted as – get this – javascript objects. Using an example from the handlebar.js website, the library allows you to do things like this…
<div class="entry">
<h1></h1>
<div class="body">
</div>
</div>… where and represents information stored in an object like this:
var context = {title: "My New Post", body: "This is my first post!"}Learning how these libraries can work with each other has been a lot of fun, especially in the context of a newly-created position at a new-to-me news organization where I have had a role to play in building a foundation for the presentation of data projects.
For instance, using handlebars.js ability to load precompiled handlebars templates from file we’ve been able to construct a basic project wrapper that can live outside CMS, minimizes the amount of code that needs to be changed at any given time and allows us to manage a single header and one footer file. DRY FTW.
I’ve also been able to use the combination of these libraries to come to having a custom mini-CMS that can run some of these projects, which can then be updated by anyone that can use a spreadsheet.
For instance, one of my first collaborations was with our business and economy reporter Matt DeBord on this project for Explaining the Monthly Jobs Report.
This project combines tabletop.js with handlebars.js – and some highcharts.js for good measure – to show the differences between “Initial,” “Revised,” and Final monthly jobs numbers released by the U.S. Bureau of Labor Statistics.
While my “design” judgement and sensibilities continue to evolve, for my money, the magic is in the fact that Matt can update this presentation for our audience by opening a Google spreadsheet and adding the new numbers. Lightweight, custom mini-CMS!
Tasneem Raja, the Interactive Editor at Mother Jones, has written a solid explainer on how the magazine has leveraged tabletop.js and a Google spreadhseet for collaborative projects such as their You Might Be the 47 Percent If… scoop.
Some of the projects I’ve worked on over the past couple months that use tabletop.js and handlebars.js include a presentation on the purchasing of Capital Appreciation Bonds by local school districts, a “By The Numbers” look at the impact of the so-called fiscal cliff on Emergency Unemployment Compensation benefits and a comparison of California and federal gun laws
But a post on NewsBeast Labs I found while writing this pointing to a post from Jeremy Singer-Vine – appropriately titled “Why I love Tabletop.js but don’t use it in production” – and some words from Mark Boas on an Open News call a couple weeks ago reminded me of potential obstacles and issues with relying too much on tabletop.js for projects.
And a tweet this week a few weeks back from Aaron Williams, and one today from Matt Waite, reminded me I have to finish this post you are reading.
All of the issues that Jeremy listed are real, and Aaron’s experience is real. Thinking about the projects I listed above, none of them really require the kind of dynamic data display that tabletop.js can pull off. It’s a nice feature to have available on the monthly jobs report, but it isn’t crucial, and certainly isn’t required on the others.
A flat file very much would work just fine.
And so I went about finding a way to create a flat json file from a Google spreadsheet that I could use in a handlebars.js template.
I already had Andy Boyle’s python script from a year ago that creates json from a csv. And I noticed in the comments section on Andy’s post that Christopher Groskopf’s csvkit has a csvjson utility that can also convert a csv to a json file.
After flirting with the idea of adding a custom argument to the csvjson utility to allow me to specify a key, I adjusted Andy’s script to create a file that looks like this…
{"objects": [{"Source": "National Employment Law Project", "DataOrder": "1", "SourceLink": "http://www.nelp.org/", "Data": "12.29.2012", "Title": "The last day anyone will receive benefits from the Emergency Unemployment Compensation program unless Congress acts to renew it."}, {"Source": "Congressional Budget Office", "DataOrder": "2", "SourceLink": "", "Data": "$30,000,000,000", "Title": "Estimated cost to renew the Emergency Unemployment Compensation program through the end of 2013."}, {"Source": "National Employment Law Project", "DataOrder": "3", "SourceLink": "http://www.nelp.org/", "Data": "400,000", "Title": "Estimated number of Californians receiving benefits from the Emergency Unemployment Compensation program, which is set to expire Jan. 2."}, {"Source": "National Employment Law Project", "DataOrder": "4", "SourceLink": "http://www.nelp.org/", "Data": "2,100,000", "Title": "Estimated number of Americans receiving benefits under the Emergency Unemployment Compensation program that would lose their unemployment benefits come January if Congress doesn\u2019t act."}]… which is something that allows me to plug my new file name into a variable in this script and fly.
Now, admittedly it works because most of my handlebars work thus far has used simple data structures – largely an object that contains an array of objects – but my workflow might fall apart quickly should I need to access multiple data streams, etc. (UPDATE: This has already happened!)
One thing I have noticed through this experiment is tabletop.js strips uppercase letters and underscores from object keys, while the csv to json version doesn’t. So that’s a thing as it’s the difference between a handlebars.js template that uses…
//what tabletop brings
and
// created from a csv to json method
Another thing I’ve noticed is the python script that converts the csv to json renders everything as a string, which has caused some issues when using something like highcharts, which requires is data objects to be integers. So I continue to explore a way to fix this.
]]>It’s tailored to California, but you can adjust it to your liking.
You will still have to add the address, city, state and zip code of the agency, and the type of data being requested, but this is a step up from having a file that I copy and paste from.
The script takes a filename as an argument. That filename will have the date prepended to it so you can see at a glance which were sent out when.
I haven’t decided yet to allow for the address and other potential variables to be passed into the script. At certain points you want to allow for mistakes, and most folks are bound to make mistakes when entering information in on the command line.
#!/bin/bash
# date variable
DATE="$(date +"%m-%d-%y")"
# input for newfile
NAME_OF_FOIA_LETTER="$1"
# name for newfile
OUTPUT_FILE="$DATE-$NAME_OF_FOIA_LETTER.md"
# create markdown FOIA
echo "Creating $NAME_OF_FOIA_LETTER FOIA letter"
cat <<EOF >$OUTPUT_FILE
**$DATE**
ADDRESS $line
STREET $line
CITY, STATE, ZIP CODE $line
**RE**: Public records request for NAME OF RECORD
**To Whom It May Concern**:
Under the California Public Records Act (Government Code Section 6250 et seq.), I ask to obtain a copy of the following records or data sets which I understand to be held by your agency:
>*DATA TO BE REQUESTED HERE*
I ask for a determination on this request within 10 days of receipt, or sooner should you make a determination without review of the record(s) in question.
If you determine that any or all or the information if exempt, I would like to know if the exemption is discretionary, if it's necessary to exercise discretion in this case and a signed notification citing the legal authorities used to determine the records are exempt.
Should some -- but not all of the information -- be exempt from disclosure -- and you intend to withhold it -- I ask that you redact it for the time being and make the rest available as requested.
Please notify me of any duplication costs that will exceed \$10.00. If I can provide any clarification or answer any questions, my contact information is below.
Thanks in advance and take care.
Sincerely:
Chris Keller $line
Data Journalist/News Applications Developer $line
Southern California Public Radio $line
**Web**: scpr.org $line
**Email**: ckeller@scpr.org $line
**Desk**: (626) 583-5214 $line
EOF
open $OUTPUT_FILEIf you summarize your idea in your first sentence, writing “TLDR” is superfluous.
As a guy who has taken to adding a tl;dr nugget on the top of walkthroughs reading the above makes me want to whack myself upside the head… But maybe what I have been trying to do has another name… Maybe something along the lines of, “You Will Accomplish”
]]>tl;dr: The README on the repo below gives a bit of a walkthrough and demonstration of the script I’ve been using to display a flat data file on Handlebars templates rendered with AJAX, which has allowed me to learn to deploy interactive data projects fairly fast.
Handlebars.js is a templating library – much like mustache.js – that “provides the power necessary to let you build semantic templates” based on data that is formatted as – get this – javascript objects. Using an example from the handlebar.js website, the library allows you to do things like this…
<div class="entry">
<h1></h1>
<div class="body">
</div>
</div>… where and represents information stored in an object like this:
var context = {title: "My New Post", body: "This is my first post!"}There are some really good resources out there for those who want to start using the Handlebars JavaScript template library. Here are some links:
I’d like to demonstrate a bit of the script I’ve been using to display a flat data file on Handlebars templates render with AJAX, and give a couple practical applications for using Handlebars in a newsroom environment in order to deploy interactive projects fairly fast.
Coming across Handlebars.js after learning the basics of django templating, I really wanted a way to mimic some of that functionality and store Handlebars templates in reusable, decoupled files that could be shared across projects.
Thankfully this function based on code from here helps me to do exactly that.
// render handlebars templates via ajax
function getTemplateAjax(path, callback) {
var source, template;
jqueryNoConflict.ajax({
url: path,
success: function (data) {
source = data;
template = Handlebars.compile(source);
if (callback) callback(template);
}
});
}I then abstract out a function to display the compiled template, passing in the name of the template, the css selector I am targeting and the data I want to display.
// function to compile handlebars template
function renderHandlebarsTemplate(withTemplate,inElement,withData){
getTemplateAjax(withTemplate, function(template) {
jqueryNoConflict(inElement).html(template(withData));
})
};I can then call it like this, where dataDetailsTemplate.handlebars is the name of my template, and #data-details is the css selector I am targeting, and data is what I want to display.
renderHandlebarsTemplate('dataDetailsTemplate.handlebars', '#data-details', data);Let’s go through the full data-script.js file, because there’s a lot in there that I’ve kind of picked up over the last several months.
I don’t really have an idea if it is “correct” to programmers out there, but I know that it works and doesn’t throw me errors.
In learning to use jQuery in the context of my previous CMS – which used several jQuery libraries – I found it just made sense to use a no conflict variable and it’s something I’ve just stuck with:
var jqueryNoConflict = jQuery;
When the DOM is ready I call the retriveData() function which kind of starts the whole ball rolling:
//begin main function
jqueryNoConflict(document).ready(function(){
retriveData();
});
//end main functionretriveData() looks for my flat JSON file, which set to a variable. It then uses jQuery’s getJSON method to pull the data and run it through a function called renderDataVisualsTemplate(). This is the function that will render my Handlebars template to the page with data in it.
// grab data
function retriveData() {
var dataSource = 'working-data-file.json';
jqueryNoConflict.getJSON(dataSource, renderDataVisualsTemplate);
};renderDataVisualsTemplate() gets an argument that represents my the data from my flat JSON file. This function runs my base handlebars template function using the name of my template (dataDetailsTemplate.handlebars), the css selector where I will inject my template (#data-details) and the data I will fill it with (data).
// render compiled handlebars template
function renderDataVisualsTemplate(data){
handlebarsDebugHelper();
renderHandlebarsTemplate('dataDetailsTemplate.handlebars', '#data-details', data);
};After that, I have my function to pull my Handlebars template from an external file and compile it. I’ve also included a Handlebars debugger, a “helper” function shows information about the data I am trying to work with.
The base handlebars template function looks like this, and takes three parameters: the name of the template, the css selector and the data object:
// function to compile handlebars template
function renderHandlebarsTemplate(withTemplate,inElement,withData){
getTemplateAjax(withTemplate, function(template) {
jqueryNoConflict(inElement).html(template(withData));
})
};Let’s take a look at the flat JSON file I am using to hold the data that will be rendered to the page. It’s structured as it is in the Handlebars walkthrough.
{"objects": [{"Source": "National Employment Law Project", "DataOrder": "1", "SourceLink": "http://www.nelp.org/", "Data": "12.29.2012", "Title": "The last day anyone will receive benefits from the Emergency Unemployment Compensation program unless Congress acts to renew it."}, {"Source": "Congressional Budget Office", "DataOrder": "2", "SourceLink": "", "Data": "$30,000,000,000", "Title": "Estimated cost to renew the Emergency Unemployment Compensation program through the end of 2013."}]}To render the data, the Handlebars template is structured just as it would be if it was inline on the index.html page, save for wrapping it in a script tag.
In this case, I’m asking that every instance of an object
be rendered to the page and structured in a certain way.
My HTML page isn’t any special, other than have a div that will have all kinds of data injected into it thanks to Handlebars.
<div id="data-details"></div>
Your mileage might vary, but I’ve found several practical applications of Handlebars.js just by looking at my needs.
For instance, I came from shops that used a CMS where I could add html, css and JavaScript to a CMS “asset” which was then wrapped by the site header, rail and footer. Here at SCPR, I’ve been lucky enough to have mentors who wanted to and helped to create something similar.
This project is on a custom structure that lies outside of the CMS. The header and footer are each Handlebars templates, external files that I add to each new project. If I need to change a link in the footer I change it in one place and it’s changed on all project pages using the template. Same goes for the header.
You could easily recreate something similar. Let’s say your template structure is something like:
<body>
<div id="data-header"></div>
<div id="data-container">
<div class="row-fluid">
<div class="span4">
<div id="data-details"></div>
</div>
<div class="span8">
<div id="data-visuals"></div>
</div>
</div>
</div>
<div id="data-footer"></div>
</body>You can probably spot some candidates for possible Handlebars templates now; data-header, data-details, data-visuals and data footer all make sense, where data-header and data-footer could be used on all projects.
Or say you want to quickly create a table to display some information. Using the data file from my earlier example, I can create a Handlebars template to do just that:
<table class="table">
<tbody>
<tr>
<td></td>
<td></td>
<td></td>
</tr>
</tbody>
</table>As an intermediate beginner to the world of web development, and entering my fifth year of being an “online guy” in a newsroom, I’ve found Handlebars to be a lot of fun. To increase that fun, there are all kinds of add ons and helper functions that you can use. swag.js might be the most fun thus far.
See while Handlebars tried to keep logic out of the templates – and I respect that – there some things I’d like to be able to do, and swag.js allows for some of that. For instance, if in my table example I wanted the title to be in all-caps I can do this with swag:
Or if I want to evaluate the data in my flat JSON file, I can run comparisons on my Handlebars templates.
<td>The Source is </td>
<td>No Source</td>
At the event, myself and Paul Klemstine and another took up the challenge of building on a pitch to come up with a way to submit quality of life issues – graffiti, potholes, sidewalks in need of repair – to the city of Madison from a mobile device. Pictures were a must, as was the ability to add a user’s location.
The Build Madison pitch also positioned this project as something that would tie in with the city’s existing ”Report a Problem” infrastructure.
Paul and I developed had a good time hacking together a project we called City Solver… It could aspire to be more – a foundation for an open-source See, Click, Fix or a tool for smaller municipalities – or less and left to lessons learned, of which there were many that apply to many things I find myself involved in on a daily basis.
From different angles and perspectives keep asking, “Is this a problem that needs to be solved?”
The original pitch wanted to tie an application into the city of Madison “Report a Problem” system. Needless to say that is an exponentially difficult task, and one not easily tackled by a hackathon team with 24 hours to work with.
And nevermind that the city of Madison has a comprehensive website that city residents can use to report specific issues and wasn’t looking to extend the system in the manner the pitch stated.
Finally, I knew that Open Data was on the horizon in the city, as the city council had recently passed an ordinance that would let information start following for developers to use.
In the end, our project ended up with a couple prizes from the event, and I had the chance to meet with city officials on two occassions to discuss CitySolver courtesy of Eve Galanter, a former city council member who pitched the project.
The conversation were fruitful as we were able to gain some insight into future city plans in this “Report a Problem” arena, some of which include mobile access and user feedback that we had in mind.
And as soon as the Open Data starts following, real problems might be made available to develop around.
Know who else has “solved” or attempted to “solve” this problem.
Had in been better in touch with the work being done by Code For America, I would have known that developers have open-sourced similar crowd-sourcing applications and made code available that we could have built upon.
That said, as an intermediate beginner, building something myself was very fufilling and helped increase my confidence about what I was capable of learning.
Spend more time on knowing the following:
Some advice from Code For America’s Kevin Curry helped focus ideas on this project further, when he suggested the following after I posted to a Google Group:
The key element of your plan is residents working together to close their own tickets. That’s significant. I think you should base everything off of that and make it totally obvious this is not a tool for reporting problems about the city that only the city can fix.
To Kevin’s point, after thinking more about this project as something “long-term” I was able to come up with the following:
That last one is a biggie, hence even if everything at this point is coming up roses, would anyone use this thing?
There are privacy aspects to keep in mind, and very real hurdles to overcome when considering the idea of neighbors reporting on issues with neighbors. For instance, “Thank you, I’d like to repair my broken fence, but I’m without work and my wife is sick and I haven’t made the time, so thank you for posting a photo of it online with your brutish comment.”
All that said, what follows are the bullet points from the BuildMadison presentation.
City Solver is a multi-device platform that allows everyday citizens to submit information about quality of life issues and problems they feel a municipality could or should address.
Whether using a web app, a native Android app, or their laptop, desktop computer or tablet, city residents could send images and descriptions of these issues and opportunities.
A rating system is Phase One of furthering the platform to allow neighborhood residents to band together to solve issues or to clean up the neighborhood.
