<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" media="screen" href="/~d/styles/atom10full.xsl"?><?xml-stylesheet type="text/css" media="screen" href="http://feeds.feedburner.com/~d/styles/itemcontent.css"?><feed xmlns="http://www.w3.org/2005/Atom" xmlns:thr="http://purl.org/syndication/thread/1.0" xmlns:feedburner="http://rssnamespace.org/feedburner/ext/1.0" xml:lang="en" xml:base="http://joeshaw.org/wp-atom.php">
	<title type="text">joe shaw</title>
	<subtitle type="text" />

	<updated>2010-07-27T14:22:04Z</updated>

	<link rel="alternate" type="text/html" href="http://joeshaw.org" />
	<id>http://joeshaw.org/feed/atom</id>
	

	<generator uri="http://wordpress.org/" version="3.0.1">WordPress</generator>
		<atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="self" type="application/atom+xml" href="http://feeds.feedburner.com/joeshaw" /><feedburner:info uri="joeshaw" /><atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="hub" href="http://pubsubhubbub.appspot.com/" /><entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[cherry picking a range of commits]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/QL8PInED3UY/667" />
		<id>http://joeshaw.org/?p=667</id>
		<updated>2010-07-27T14:22:04Z</updated>
		<published>2010-06-22T13:07:48Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" />		<summary type="html"><![CDATA[At work we use git, and I often want to cherry pick a series of commits from a development branch, but don&#8217;t want to merge the whole branch for whatever reason. I put out a call on Twitter for ideas, and got a handful of good ones back. Update, 27 July 2010: git 1.7.2 released [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2010/06/22/667"><![CDATA[<p>At <a href="http://litl.com">work</a> we use git, and I often want to cherry pick a series of commits from a development branch, but don&#8217;t want to merge the whole branch for whatever reason.  I put out <a href="http://twitter.com/joeshaw/status/16483721670">a call on Twitter</a> for ideas, and got a handful of good ones back.</p>
<p><b>Update, 27 July 2010:</b> git 1.7.2 released earlier this week, has support for passing a range of commits to <tt>git cherry-pick</tt>.  See the <a href="http://www.kernel.org/pub/software/scm/git/docs/RelNotes-1.7.2.txt">release notes</a>.  For my pre-1.7.2 solution, read on.</p>
<ul>
<li><a href="http://twitter.com/sandyarmstrong">Sandy</a> pointed to <a href="http://people.gnome.org/~federico/news-2009-06.html#04">a Federico blog post</a> which formats the commit range as patches, changes branches, and re-applies them.  <a href="http://twitter.com/lyager">@lyager</a> also <a href="http://twitter.com/lyager/status/16484683259">recommended</a> this method.</li>
<li>Federico himself <a href="http://twitter.com/federicomena/status/16485552052">suggested</a> <a href="http://twitter.com/federicomena/status/16485568349">using</a> <tt>rebase --onto</tt>.
<li>Garrett and <a href="http://twitter.com/havocp">Havoc</a> recommended just merging the branch and then rebasing out the commits I didn&#8217;t want.</li>
<li>Garrett also <a href="http://twitter.com/garrett/status/16484433910">suggested <tt>gitk --all</tt></a>.</li>
<li>On IRC, <a href="http://teichman.org">Peter</a> suggested combining <tt>git rev-list</tt> and <tt>cherry-pick</tt>.</li>
</ul>
<p>All fine suggestions.  I wanted something that I could easily transform into an alias since I&#8217;ve been doing this a lot lately, and I&#8217;d like it to scale to large numbers of commits.  I ended up going with Peter&#8217;s suggestion, and behold! I present to you:</p>
<blockquote><p><b><code>git apple-pick</code></b><sup><a href="#footnote">1</a></sup></p></blockquote>
<p>or, as a git alias:</p>
<blockquote><p><code>apple-pick = !sh -c 'git rev-list --reverse "$@" | xargs -n1 git cherry-pick' -</code></p></blockquote>
<p>Given a range of commits, it cherry picks them onto the current branch.  The workflow:</p>
<blockquote><p><code>$ git checkout my-branch<br />
... hackety hack, do a bunch of commits ...<br />
$ git checkout master<br />
$ git apple-pick abc123^..my-branch<br />
Finished one cherry-pick.<br />
[master abc123] tweak some junk<br />
 1 files changed, 9 insertions(+), 0 deletions(-)<br />
Finished one cherry-pick.<br />
[master def456] did some awesome stuff<br />
 6 files changed, 30 insertions(+), 22 deletions(-)<br />
Finished one cherry-pick.<br />
</code></p></blockquote>
<p>Thanks for the help guys, and I hope you gitinistas out there find it useful.  What other cool alises have you developed or found that help your daily workflow?  <a href="http://twitter.com/?status=.@joeshaw%20">Tweet them to me.</a></p>
<p><a name="footnote"></a><sup>1</sup> Federico <a href="http://twitter.com/federicomena/status/16489182407">thinks I should call it</a> <tt>git transplant</tt> instead.  He&#8217;s probably right, but mine&#8217;s cuter.</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2010/06/22/667#comments" thr:count="1" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2010/06/22/667/feed/atom" thr:count="1" />
		<thr:total>1</thr:total>
	<feedburner:origLink>http://joeshaw.org/2010/06/22/667</feedburner:origLink></entry>
		<entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[additional mbta bus feeds available]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/JjpKc9bcZP8/663" />
		<id>http://joeshaw.org/?p=663</id>
		<updated>2010-06-04T17:25:28Z</updated>
		<published>2010-06-04T17:25:28Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" />		<summary type="html"><![CDATA[The MBTA last night announced that they were adding additional routes to their real-time location feeds, with plans to have all bus routes available by the end of the summer. The new routes are 1, 4, 15, 22, 23, 28, 32, 57, 66, 71, 73, and 77. My MBTA/Google maps mash-up (blog post) has incorporated [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2010/06/04/663"><![CDATA[<p>The MBTA last night <a href="http://www.universalhub.com/2010/mbta-expand-real-time-bus-info">announced</a> that they were adding additional routes to their real-time location feeds, with plans to have <i>all</i> bus routes available by the end of the summer.  The new routes are 1, 4, 15, 22, 23, 28, 32, 57, 66, 71, 73, and 77.  My <a href="http://mbta-bus.appspot.com/">MBTA/Google maps mash-up</a> (<a href="http://joeshaw.org/2009/11/15/633">blog post</a>) has incorporated these new feeds and will automatically include future ones.</p>
<p>Right now inbound/outbound markers are not working due to a change in the MBTA&#8217;s feed.  It&#8217;s not clear if this was intentional or a bug, and I&#8217;m not the <a href="http://groups.google.com/group/massdotdevelopers/browse_thread/thread/29a781a0bfa70d6f">only</a> <a href="http://groups.google.com/group/massdotdevelopers/browse_thread/thread/6e98ac23f5323969">one</a> who is having issues with it.</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2010/06/04/663#comments" thr:count="0" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2010/06/04/663/feed/atom" thr:count="0" />
		<thr:total>0</thr:total>
	<feedburner:origLink>http://joeshaw.org/2010/06/04/663</feedburner:origLink></entry>
		<entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[avchd to mp4/h264/aac conversion]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/Qla9P8MmtiQ/655" />
		<id>http://joeshaw.org/?p=655</id>
		<updated>2010-04-29T03:30:36Z</updated>
		<published>2010-04-10T14:28:03Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" />		<summary type="html"><![CDATA[For posterity: I have a Canon HF200 HD video camera, which records to AVCHD format. AVCHD is H.264 encoded video and AC-3 encoded audio in a MPEG-2 Transport Stream (m2ts, mts) container. This format is not supported by Aperture 3, which I use to store my video. With Blizzard&#8216;s help, I figured out an ffmpeg [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2010/04/10/655"><![CDATA[<p>For posterity:</p>
<p>I have a <a href="http://www.amazon.com/gp/product/B001OI2Z2I?ie=UTF8&#038;tag=joeshaw-20&#038;linkCode=as2&#038;camp=1789&#038;creative=390957&#038;creativeASIN=B001OI2Z2I">Canon HF200 HD video camera</a>, which records to AVCHD format. AVCHD is H.264 encoded video and AC-3 encoded audio in a MPEG-2 Transport Stream (m2ts, mts) container.  This format is not supported by Aperture 3, which I use to store my video.</p>
<p>With <a href="http://www.0xdeadbeef.com/">Blizzard</a>&#8216;s help, I figured out an ffmpeg command-line to convert to H.264 encoded video and AAC encoded audio in an MPEG-4 (mp4) container.  This is supported by Aperture 3 and other Quicktime apps.</p>
<p><tt>$ ffmpeg -sameq -ab 256k -i input-file.m2ts -s hd1080 output-file.mp4 -acodec aac</tt></p>
<p>Command-line order is important, which is infuriating.  If you move the <tt>-s</tt> or <tt>-ab</tt> arguments, they may not work.  Add <tt>-deinterlace</tt> if the source videos are interlaced, which mine were originally until I turned it off.  The only downside to this is that it generates huge output files, on the order of 4-5x greater than the input file.</p>
<p><strong>Update, 28 April 2010:</strong> Alexander Wauck emailed me to say that re-encoding the video isn&#8217;t necessary, and that the existing H.264 video could be moved from the m2ts container to the mp4 container with a command-line like this:</p>
<p><tt>$ ffmpeg -i input-file.m2ts -ab 256k -vcodec copy -acodec aac output-file.mp4</tt></p>
<p>And he&#8217;s right&#8230; as long as you don&#8217;t need to deinterlace the video.  With the whatever-random-ffmpeg-trunk checkout I have, adding <tt>-deinterlace</tt> to the command-line segfaults.  I actually had tried <tt>-vcodec copy</tt> early in my experiments but abandoned it after I found that it didn&#8217;t deinterlace.  I had forgotten to try it again after I moved past my older interlaced videos.  Thanks Alex!</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2010/04/10/655#comments" thr:count="0" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2010/04/10/655/feed/atom" thr:count="0" />
		<thr:total>0</thr:total>
	<feedburner:origLink>http://joeshaw.org/2010/04/10/655</feedburner:origLink></entry>
		<entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[real-time mbta bus location + google maps mashup]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/UsVMxfjsaW8/633" />
		<id>http://joeshaw.org/?p=633</id>
		<updated>2009-11-16T03:31:48Z</updated>
		<published>2009-11-16T03:31:48Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" />		<summary type="html"><![CDATA[This weekend I read that the MBTA and Massachusetts Department of Transportation had released a trial real-time data feed for the positioning of vehicles on five of its bus routes. This is very important data to have, and while obviously everyone would like to see more routes added, it&#8217;s a start. I decided to hack [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2009/11/15/633"><![CDATA[<p>This weekend I read that the MBTA and Massachusetts Department of Transportation had released a <a href="http://www.eot.state.ma.us/developers/realtime/">trial real-time data feed</a> for the positioning of vehicles on five of its bus routes.  This is very important data to have, and while obviously everyone would like to see more routes added, it&#8217;s a start.</p>
<p>I decided to <a href="http://mbta-bus.appspot.com">hack together a mashup</a> of this data with Google Maps, to see how easy it would be.  In the end it took me a few hours on Saturday to get the site up and running, and a couple more on Sunday adding features like the drawing of routes on the map, colorizing markers for inbound vs. outbound buses, and adding reverse geocoding of the buses themselves.</p>
<p><a href="http://mbta-bus.appspot.com"><img src="http://joeshaw.org/wordpress/wp-content/uploads/2009/11/mbta-bus-real-time.PNG" alt="MBTA Real-time bus info" title="MBTA Real-time bus info" width="400" height="347" class="aligncenter size-full wp-image-635" /></a></p>
<p>To do this I used three technologies (Google App Engine, JQuery, Google Maps) and two data sources (the real-time XML feed and the MBTA Google Transit Feed Specification files).</p>
<p><strong><a href="http://appengine.google.com">Google App Engine</a></strong></p>
<p>App Engine is so perfectly suited for smaller, playtime hacks like this that it&#8217;s hard to imagine how anyone got anything done before it existed.  The tedious, up-front bootstrapping that is required in so many programming projects has been enough to completely turn me off to small, spare-time hacking projects on occasion in the past.  The brilliance behind a hosted software environment is obvious, but the amount of work to build a safe, hosted system with a fairly comprehensive set of APIs seems to be such a mountain of work that in many ways I find it surprising that anyone &#8212; even, perhaps especially, Google &#8212; built it at all.</p>
<p>I chose the Python SDK and the programming was straightforward and easy.  It takes some elements from Django, with which I am familiar from work.</p>
<p><strong><a href="http://jquery.com">JQuery</a></strong></p>
<p>A no-brainer.  Hands down the best JavaScript toolkit available.  Making the AJAX calls to get route and vehicle location information was a breeze, and the transparent handling of the XML data of the real-time feed prevents me from losing the will to live &#8212; a common feeling when dealing with XML.</p>
<p>My only complaint is with the documentation.  While the API reference is good for any given piece of the API, the examples are a little light and there is absolutely zero cross-referencing to other parts, especially ones not a part of JQuery itself.  It was not obvious, for example, how to deal with the XML document returned by the AJAX call.  It sounds like the docs are <a href="http://twitter.com/jeresig/status/5750291977">getting some work</a>, though, so this will hopefully improve.</p>
<p><strong><a href="http://code.google.com/apis/maps/">Google Maps</a></strong></p>
<p>This was my first endeavor with the Maps API, and it&#8217;s good.  It&#8217;s not the best API in the world, but it&#8217;s hardly the worst either.  Adding markers of different colors is annoying, but not so onerous as to make it tedious.  The breadth of functionality provided is impressive, but then again it has been around for a few years at this point.  Markers are easy to add, drawing the route map is absolutely trivial with a KML file, and even the reverse geocoding &#8212; which gives you a street address given a latitude/longitude pair &#8212; is straightforward.</p>
<p>The docs suck, though.  There&#8217;s no indication that a size or anchor position is required when creating an icon for a custom marker &#8212; required for colors other than red &#8212; and due to the minified JS files tracking down that error took longer than any other task in the project. Reverse geocoding mentions that a <tt>Placemark</tt> object will be returned, but that class doesn&#8217;t appear anywhere in the reference documentation.</p>
<p><strong><a href="http://www.eot.state.ma.us/developers/realtime/">Real-time feed data</a></strong></p>
<p>Lots to like.  Straightforward, easy to parse.  It&#8217;d be nice if I didn&#8217;t have to do the reverse geocoding to figure out what the street address is, but it&#8217;s not a dealbreaker.  Main downside is that it&#8217;s XML as opposed to JSON.  And of course, it&#8217;s only 5 bus routes and zero subway and commuter rail routes.</p>
<p><strong><a href="http://www.eot.state.ma.us/default.asp?pgid=content/developer&#038;sid=about#para15">MBTA Google Transit Feed Specification files</a></strong></p>
<p>A comprehensive set of data describing every transit route, every stop, and every route in the MBTA system.  An impressive set of data encoded in a format designed for Google Transit.  There is a <a href="http://code.google.com/p/googletransitdatafeed/">set of example tools</a> to view and manipulate this data, and one of those translates this data into a KML file for use with Google Maps.  I should have tweaked the tools to output only the KML for the routes I cared about, but I did this by hand instead&#8230; not a big deal for only 5 bus lines.  These KML files are fed into the Google Maps API to display the route as a blue line on the map when selected.</p>
<p><strong>POKE 47196, 201</strong></p>
<p>This is what a lot of programming is like now, for better and for worse.</p>
<p>On the one hand it is the perfect example of high-level component-oriented programming.  Data is formatted in easily parseable interchange formats and plugged into well-defined interfaces.  These interfaces plug into other interfaces.  The result is a zoomable, pannable map with real-time bus location information that updates every 15 seconds.  The lines-of-code count is around 100 including both Python and JavaScript.  With a few hours work, I built something modestly useful out of nothing.  I stand on the shoulders of giants.</p>
<p>On the other hand I didn&#8217;t really <em>build</em> anything.  This is just assembly line programming.  It was not a particularly creative endeavor, and it wasn&#8217;t challenging intellectually.  Anybody could have done it.  It&#8217;s cool, but there is little sense of accomplishment in the end product. It feels a little hollow.</p>
<p>Which is not to say that I didn&#8217;t enjoy it, or that it wasn&#8217;t worth the effort.  I learned new technology, I played with software and data that I hadn&#8217;t had the opportunity to before.  I broadened my horizons, however slightly.  And it got me to write this blog post.</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2009/11/15/633#comments" thr:count="1" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2009/11/15/633/feed/atom" thr:count="1" />
		<thr:total>1</thr:total>
	<feedburner:origLink>http://joeshaw.org/2009/11/15/633</feedburner:origLink></entry>
		<entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[the only way to drown out inane chatter is with&#8230; npr]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/5M00I--9hSw/628" />
		<id>http://joeshaw.org/?p=628</id>
		<updated>2009-09-18T15:59:45Z</updated>
		<published>2009-09-18T15:59:45Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" />		<summary type="html"><![CDATA[I discovered a way to copy some podcasts onto my 5th gen iPod Nano without using iTunes, since I was not at the Mac I use to sync it. It utilizes the voice memos feature, so it probably won&#8217;t work on other iPod models. It probably will work on Windows, though. Connect your iPod to [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2009/09/18/628"><![CDATA[<p>I discovered a way to copy some podcasts onto my 5th gen iPod Nano without using iTunes, since I was not at the Mac I use to sync it.   It utilizes the voice memos feature, so it probably won&#8217;t work on other iPod models.  It probably will work on Windows, though.</p>
<p>Connect your iPod to the Mac, and turn on hard drive access in iTunes if it&#8217;s not already on.</p>
<p>You&#8217;ll need your music or podcast to be in AAC (.m4a) format.  If you have an MP3, go into iTunes preferences -> General -> Import Settings&#8230; and set the &#8220;Import using&#8221; encoder to AAC.  Then select the tracks and go to Advanced -> Create AAC Version.</p>
<p>Copy the new .m4a file into the Recordings directory on the iPod, and give it a name like &#8220;20090101 120000.m4a&#8221;.</p>
<p>Eject your iPod and go to the voice memos feature.  In your list of voice memos you should see something like &#8220;1/1/2009 12:00 PM&#8221; &#8212; it corresponds to the filename above &#8212; and press play.  Voilà!</p>
<p>It&#8217;s not an elegant solution, but it&#8217;ll give you something new to listen to on your subway ride home when you&#8217;re desperate.</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2009/09/18/628#comments" thr:count="0" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2009/09/18/628/feed/atom" thr:count="0" />
		<thr:total>0</thr:total>
	<feedburner:origLink>http://joeshaw.org/2009/09/18/628</feedburner:origLink></entry>
		<entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[there is only one correct way to skin a cat]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/s3hgUW0almE/623" />
		<id>http://joeshaw.org/?p=623</id>
		<updated>2009-03-18T21:20:07Z</updated>
		<published>2009-03-18T16:01:46Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" />		<summary type="html"><![CDATA[I have ranted about the iPhone&#8217;s horrible iPod interface in the past, and any improvement they can make is certainly welcome. But the improvements in the iPhone OS 3.0 update seem more half-assed than a true solution. Yes, the ability to skip back 30 seconds will be nice, but it&#8217;s still a ham-fisted solution to [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2009/03/18/623"><![CDATA[<p>I have ranted about the iPhone&#8217;s <a href="http://joeshaw.org/2007/09/22/488#ipod">horrible iPod interface</a> in the past, and any improvement they can make is certainly welcome.  But the <a href="http://www.mobilecrunch.com/2009/03/17/gallery-new-podcast-features-in-iphone-os-30/">improvements in the iPhone OS 3.0 update</a> seem more half-assed than a true solution.  Yes, the ability to skip back 30 seconds will be nice, but it&#8217;s still a ham-fisted solution to the problem of exact scrolling inside a 70 minute podcast.  And the &#8220;scrubber&#8221; interface seems complicated and error-prone.  </p>
<p>The frustrating thing is that Apple already has the One True User Interface for playing audio: the click-wheel.  With its handling of acceleration you can both seek through hours of audio extremely quickly while still giving you the one-second resolution to seek to the exact point you want, and I don&#8217;t understand why it isn&#8217;t emulated on the iPhone and iPod touch.  At this point, I have to believe that there is some limitation of the touchscreen hardware which prevents it.  Sigh.</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2009/03/18/623#comments" thr:count="0" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2009/03/18/623/feed/atom" thr:count="0" />
		<thr:total>0</thr:total>
	<feedburner:origLink>http://joeshaw.org/2009/03/18/623</feedburner:origLink></entry>
		<entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[python daemon threads considered harmful]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/CvE6l6FxapA/605" />
		<id>http://joeshaw.org/?p=605</id>
		<updated>2009-02-25T17:50:30Z</updated>
		<published>2009-02-24T22:28:36Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" /><category scheme="http://joeshaw.org" term="considered harmful" /><category scheme="http://joeshaw.org" term="daemon threads" /><category scheme="http://joeshaw.org" term="python" /><category scheme="http://joeshaw.org" term="wolf" />		<summary type="html"><![CDATA[The other day at work we encountered an unusual exception in our nightly pounder test run after landing some new code to expose some internal state via a monitoring API. The problem occurred on shutdown. The new monitoring code was trying to log some information, but was encountering an exception. Our logging code was built [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2009/02/24/605"><![CDATA[<p>The other day <a href="http://itasoftware.com">at work</a> we encountered an unusual exception in our nightly pounder test run after landing some new code to expose some internal state via a monitoring API.  The problem occurred on shutdown.  The new monitoring code was trying to log some information, but was encountering an exception.  Our logging code was built on top of Python&#8217;s <code><a href="http://docs.python.org/library/logging.html">logging</a></code> module, and we thought perhaps that something was shutting down the logging system without us knowing.  We ourselves never explicitly shut it down, since we wanted it to live until the process exited.</p>
<p>The monitoring was done inside a daemon thread.  The <a href="http://docs.python.org/library/threading.html#id1">Python docs say</a> only: </p>
<blockquote><p>A thread can be flagged as a “daemon thread”. The significance of this flag is that the entire Python program exits when only daemon threads are left. &#8220;</p></blockquote>
<p>Which sounds pretty good, right?  This thread is just occasionally grabbing some data, and we don&#8217;t need to do anything special when the program shuts down.  Yeah, I remember when I used to believe in things too.</p>
<p>Despite a global interpreter lock that prevents Python from being truly concurrent anyway, there is a very real possibility that the daemon threads can still execute after the Python runtime has started its own tear-down process.  One step of this process appears to be to set the values inside <code><a href="http://docs.python.org/library/functions.html#globals">globals()</a></code> to <code>None</code>, meaning that any module resolution results in an <code>AttributeError</code> attempting to dereference <code>NoneType</code>.  Other variations on this cause <code>TypeError</code> to be thrown.</p>
<p>The code which triggered this looked something like this, although with more abstraction layers which made hunting it down a little harder:<br />
<code><br />
try:<br />
&nbsp;&nbsp;&nbsp;&nbsp;log.info("Some thread started!")<br />
&nbsp;&nbsp;&nbsp;&nbsp;try:<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;do_something_every_so_often_in_a_loop_and_sleep()<br />
&nbsp;&nbsp;&nbsp;&nbsp;except somemodule.SomeException:<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pass<br />
&nbsp;&nbsp;&nbsp;&nbsp;else:<br />
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pass<br />
finally:<br />
&nbsp;&nbsp;&nbsp;&nbsp;log.info("Some thread exiting!")<br />
</code></p>
<p>The exception we were seeing was an <code>AttributeError</code> on the last line, the <code>log.info()</code> call.  But that wasn&#8217;t even the original exception.  It was actually another <code>AttributeError</code> caused by the <code>somemodule.SomeException</code> dereference.  Because all the modules had been reset, <code>somemodule</code> was <code>None</code> too.</p>
<p>Unfortunately the docs are completely devoid of this information, at least in the threading sections which you would actually reference.  The best information I was able to find was <a href="http://mail.python.org/pipermail/python-list/2005-February/307042.html">this email to python-list</a> a few years back, and a <a href="http://mail.python.org/pipermail/python-dev/2003-September/038151.html">few</a> <a href="http://mail.python.org/pipermail/python-bugs-list/2004-July/023901.html">other</a> <a href="http://mail.python.org/pipermail/python-bugs-list/2008-January/045448.html">emails</a> which don&#8217;t really put the issue front and center.</p>
<p>In the end the solution for us was simply to make them non-daemon threads, notice when the app is being shut down and join them to the main thread.  Another possibility for us was to catch <code>AttributeError</code> in our thread wrapper class &#8212; which is what the author of the aforementioned email does &#8212; but that seems like papering over a real bug and a real error.  Because of this misbehavior, daemon threads lose almost all of their appeal, but oddly I can&#8217;t find people really publicly saying &#8220;don&#8217;t use them&#8221; except in scattered emails.  It seems like it&#8217;s underground information known only to the Python cabal.  (<a href="http://en.wikipedia.org/wiki/There_Is_No_Cabal">There is no cabal.</a>)</p>
<p>So, I am going to say it.  When I went searching there weren&#8217;t any helpful hints in a Google search of &#8220;python daemon threads considered harmful&#8221;.  So, I am staking claim to that phrase.  People of The Future: You&#8217;re welcome.</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2009/02/24/605#comments" thr:count="0" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2009/02/24/605/feed/atom" thr:count="0" />
		<thr:total>0</thr:total>
	<feedburner:origLink>http://joeshaw.org/2009/02/24/605</feedburner:origLink></entry>
		<entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[another beagle innovation stolen]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/p1TG0V1U-R4/602" />
		<id>http://joeshaw.org/?p=602</id>
		<updated>2009-02-24T16:18:33Z</updated>
		<published>2009-02-24T16:18:33Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" /><category scheme="http://joeshaw.org" term="beagle" /><category scheme="http://joeshaw.org" term="epiphany" /><category scheme="http://joeshaw.org" term="firefox" /><category scheme="http://joeshaw.org" term="safari" /><category scheme="http://joeshaw.org" term="theivery" /><category scheme="http://joeshaw.org" term="wolf" />		<summary type="html"><![CDATA[From the Safari 4 beta release: * Full History Search, where users search through titles, web addresses and the complete text of recently viewed pages to easily return to sites they’ve seen before; If you&#8217;ve been a Beagle user in the last 3 years this has been supported for Firefox and Epiphany users. But I [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2009/02/24/602"><![CDATA[<p>From the <a href="http://www.crunchgear.com/2009/02/24/safari-4-finally-a-reason-to-come-back/">Safari 4 beta release</a>:</p>
<blockquote><p>* Full History Search, where users search through titles, web addresses and the complete text of recently viewed pages to easily return to sites they’ve seen before;</p></blockquote>
<p>If you&#8217;ve been a <a href="http://beagle-project.org">Beagle</a> user in the last 3 years this has been supported for Firefox and Epiphany users.  But I wouldn&#8217;t mind seeing Firefox have this sort of indexing and search built-in either&#8230; the AwesomeBar was a great first step in that direction.</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2009/02/24/602#comments" thr:count="0" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2009/02/24/602/feed/atom" thr:count="0" />
		<thr:total>0</thr:total>
	<feedburner:origLink>http://joeshaw.org/2009/02/24/602</feedburner:origLink></entry>
		<entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[giticular cancer]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/5a6UDLVoiJg/588" />
		<id>http://joeshaw.org/?p=588</id>
		<updated>2009-01-06T03:36:14Z</updated>
		<published>2009-01-06T03:30:54Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" /><category scheme="http://joeshaw.org" term="bzr" /><category scheme="http://joeshaw.org" term="dvcs" /><category scheme="http://joeshaw.org" term="git" /><category scheme="http://joeshaw.org" term="gnome" /><category scheme="http://joeshaw.org" term="wolf" />		<summary type="html"><![CDATA[I haven&#8217;t been on d-d-l for months now, but when someone mentioned how comically entertaining the whole DVCS survey thread was, I just had to catch up. From my cursory understanding, it seems like the data would be stored as bzr repositories and then new code would be developed to export those repositories over the [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2009/01/05/588"><![CDATA[<p>I haven&#8217;t been on <a href="http://mail.gnome.org/mailman/listinfo/desktop-devel-list">d-d-l</a> for months now, but when someone mentioned how comically entertaining the whole <a href="http://mail.gnome.org/archives/desktop-devel-list/2009-January/msg00003.html">DVCS survey thread</a> was, I just had to catch up.</p>
<p>From my cursory understanding, it seems like the data would be stored as bzr repositories and then new code would be developed to export those repositories over the git protocol, so that git users could use their own tools.</p>
<p>While it seems like a neat hack, and probably a worthwhile proof-of-concept, the idea that GNOME would switch to it seems completely insane to me.  There are lots of reasons why, many addressed in the thread:</p>
<ul>
<li>Why develop this code only for GNOME, instead of developing it with the support and blessing of upstream bzr and/or git?  There the collective experience of these communities could guide and influence the development.</li>
<li>There is basically only one person developing this software, resulting in a critical piece of GNOME infrastructure with a <a href="http://en.wikipedia.org/wiki/Bus_factor">bus factor</a> of 1.  This is very bad, and when you consider that the results of the survey strongly support Git, the vast majority of developers will be using it and would be inconvenienced if the system failed.</li>
<li>GNOME has a terrible track record of abandoned software &#8212; maybe it&#8217;s not actually any worse than other 10 year old large-scale open source projects &#8212; but it is very common.  I don&#8217;t have any data to back this up, but I feel that in a lot of ways this is even more true for infrastructure that most developers never see.</li>
<li>The git-over-bzr option was never an option in the DVCS survey, but if it were it seems like it would have rated extremely lowly.  There seems to be a <a href="http://mail.gnome.org/archives/desktop-devel-list/2009-January/msg00072.html">vocal opposition</a> to it in the thread.</li>
<li>This is an abstraction, and <a href="http://www.joelonsoftware.com/articles/LeakyAbstractions.html">abstractions are always leaky</a>.  wxWindows kinda sucks because it can&#8217;t emulate all windowing systems equally.  There is no way a git compatibility layer on top of a bzr repository will ever be as good as a native solution, just like git-on-svn isn&#8217;t as good as a pure git solution.</li>
<li>What happens when (not if, when) the git protocol changes in an incompatible way?  Will we be at the mercy of someone to hack and fix the compatibility layer?  Will the original author still be around and interested enough to do it, possibly years down the road?</li>
</ul>
<p>The last point, combined with the abandonware point earlier, are what concern me the most.  In the thread, <a href="http://mail.gnome.org/archives/desktop-devel-list/2009-January/msg00048.html">David Zeuthen asked Olav Vitters</a>:</p>
<blockquote><p>Then what happens when a new version of git with a new feature, incompatible with the git-serve kludge, is released? Then we&#8217;re screwed, right? And who gets to pay? We do. We&#8217;re stuck with an old version of git. Us. The very same people who very clearly said &#8220;git&#8221;, not &#8220;bzr&#8221;.</p></blockquote>
<p>Not the most politic way of saying it, but I think the point is valid.  When I read that, I had deja vu, because I had <em>just</em> read <a href="http://mail.gnome.org/archives/desktop-devel-list/2008-December/msg00008.html">this thread</a> from early December about Bugzilla, initiated by Olav:</p>
<blockquote><p>Subject: Reduced Bugzilla functionality for 6+ months &#8212; acceptable?</p>
<p>The GNOME Bugzilla is still using 2.20. Current stable upstream is at 3.2.<br />
[...]<br />
For that the proposal is that the following is not part of the initial<br />
upgraded bgo:<br />
 * The points system<br />
 * index.cgi UI mods<br />
 * Making a new favicon<br />
 * The infomessages on show_bug.cgi<br />
 * Layout modifications for attachment table and the login box<br />
 * duplicates.cgi modifications<br />
 * Fixing the comment headers<br />
 * Patch and keyword emblems<br />
 * delete-keyword.pl, mass-reassign-bugs.pl, and year-end-stats.pl<br />
 * describeuser.cgi
</p></blockquote>
<p>In other words, upgrade the GNOME Bugzilla installation to a new version of the upstream software, and break all of GNOME&#8217;s current customizations.  Is this not exactly what will happen eventually with the git kludge?  I can foresee history repeating itself here.  Bugzilla is pretty essential to GNOME, and degradation of service is undesirable.  But a degraded, unavailable or fractured source control system is unconscionable.</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2009/01/05/588#comments" thr:count="0" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2009/01/05/588/feed/atom" thr:count="0" />
		<thr:total>0</thr:total>
	<feedburner:origLink>http://joeshaw.org/2009/01/05/588</feedburner:origLink></entry>
		<entry>
		<author>
			<name>Joe</name>
						<uri>http://joeshaw.org</uri>
					</author>
		<title type="html"><![CDATA[best.  christmas present.  ever.]]></title>
		<link rel="alternate" type="text/html" href="http://feedproxy.google.com/~r/joeshaw/~3/eBWw_e6-h-k/573" />
		<id>http://joeshaw.org/?p=573</id>
		<updated>2008-12-25T20:40:50Z</updated>
		<published>2008-12-25T20:40:50Z</published>
		<category scheme="http://joeshaw.org" term="Uncategorized" />		<summary type="html"><![CDATA[On 20 December 2008 at 1:41 am, Brette and I welcomed our son Elliot into the world. Both mom and baby are doing very well, and we&#8217;ve loved the last several days getting to know each other. This is his &#8220;black power&#8221; pose. Fight the man, little guy. The previous day Brette, my mom and [...]]]></summary>
		<content type="html" xml:base="http://joeshaw.org/2008/12/25/573"><![CDATA[<p>On 20 December 2008 at 1:41 am, Brette and I welcomed our son Elliot into the world.  Both mom and baby are doing very well, and we&#8217;ve loved the last several days getting to know each other.</p>
<p><a class="imagelink" href="http://www.flickr.com/photos/joeshaw/3136352838/" title="Baby Elliot"><img class="imagecenter" src="http://farm4.static.flickr.com/3104/3136352838_873c17a2a1.jpg" width="374" height="500" alt="[A photo by Joe Shaw]" /></a>
<div class="imagecenter">This is his &#8220;black power&#8221; pose.  Fight the man, little guy.</div>
<p>The previous day Brette, my mom and I were joking around that the baby would almost certainly come the following day &#8212; still a few days before his due date &#8212; during the snowstorm that was expected to drop up to 15 inches of snow on Boston.  He didn&#8217;t disappoint.  We found ourselves walking to the birth center about half a mile away in the 4 or 5 inches that had fallen at that point on nearly deserted roads during Friday rush hour, stopping briefly whenever Brette had a contraction.</p>
<p><img class="imagecenter" src="http://farm4.static.flickr.com/3267/3136353040_37af7cf4b0.jpg" width="500" height="375" alt="[photo]" /></p>
<p>The birth went amazingly well and I am so proud of Brette.  All of the nurses and midwives at the birth center were talking about what a wonderful job she did, and I couldn&#8217;t agree more.</p>
<p>A few things I wanted to mention:</p>
<p>If you prefer to give birth in a hospital, or want a pre-scheduled <a href="http://en.wikipedia.org/wiki/C-section">c-section</a>, go for it.  Do whatever is most comfortable and appropriate for you.  But Brette and I wanted a natural childbirth without surgical intervention unless medically necessary, and for us the <a href="http://www.cha.harvard.edu/ob_gyn/birth_center.shtml">Cambridge Birth Center</a> was the perfect place.  Situated inside an old Victorian house, it sits across the street from the Cambridge Hospital.  The rooms are homey and comforting.  The only thing that annoyed us was the constant snow plowing going on just outside our window&#8230; but once things got down to business we weren&#8217;t paying attention to any of that.  The nurses and midwives were so wonderful and accommodating.  We were allowed to walk around, bring whatever food we wanted with us, got to sleep after the birth and go home the following afternoon.  Just a great experience.</p>
<p>Ok, this is important: get a <a href="http://en.wikipedia.org/wiki/Doula">doula</a>.  I think I was as helpful and comforting as I could be for Brette, but look, I don&#8217;t have a uterus.  I never have and probably never will.  But a doula does.  I don&#8217;t know physically what it&#8217;s like to give birth.  A doula very well may have gone through it herself.  But most importantly, they have training on ways to make the laboring woman more comfortable, and her entire job is to support and comfort.  Ours was fantastic, and I can&#8217;t imagine going through the birth without her.</p>
<p>Also many thanks to my mom, who has been with us for the past week.  She has helped us tremendously with keeping the house clean and orderly, making us food all the time, giving us a relief when we&#8217;ve been overwhelmed with E. at all hours, and sharing her experience with us.</p>
<p>Merry Christmas.</p>
]]></content>
		<link rel="replies" type="text/html" href="http://joeshaw.org/2008/12/25/573#comments" thr:count="0" />
		<link rel="replies" type="application/atom+xml" href="http://joeshaw.org/2008/12/25/573/feed/atom" thr:count="0" />
		<thr:total>0</thr:total>
	<feedburner:origLink>http://joeshaw.org/2008/12/25/573</feedburner:origLink></entry>
	</feed><!-- Dynamic page generated in 0.514 seconds. --><!-- Cached page generated by WP-Super-Cache on 2010-09-05 21:44:14 -->
