<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" media="screen" href="/~d/styles/rss2full.xsl"?><?xml-stylesheet type="text/css" media="screen" href="http://feeds.feedburner.com/~d/styles/itemcontent.css"?><rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" xmlns:feedburner="http://rssnamespace.org/feedburner/ext/1.0" version="2.0">

<channel>
	<title>The Endeavour</title>
	
	<link>http://www.johndcook.com/blog</link>
	<description>The blog of John D. Cook</description>
	<lastBuildDate>Wed, 16 May 2012 20:57:40 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.8.4</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="self" type="application/rss+xml" href="http://feeds.feedburner.com/TheEndeavour" /><feedburner:info uri="theendeavour" /><atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="hub" href="http://pubsubhubbub.appspot.com/" /><feedburner:emailServiceId>TheEndeavour</feedburner:emailServiceId><feedburner:feedburnerHostname>http://feedburner.google.com</feedburner:feedburnerHostname><item>
		<title>Mars, magic squares, and music</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/-43sbvrhDGU/</link>
		<comments>http://www.johndcook.com/blog/2012/05/16/mars-magic-squares-music/#comments</comments>
		<pubDate>Wed, 16 May 2012 11:59:04 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Math]]></category>
		<category><![CDATA[Music]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11379</guid>
		<description><![CDATA[About a year ago I wrote about Jupiter&#8217;s magic square. Then yesterday I was listening to the New Sounds podcast that mentioned a magic square associated with Mars. I hadn&#8217;t heard of this, so I looked into it and found there were magic squares associated with each of solar system bodies known to antiquity (i.e. [...]]]></description>
			<content:encoded><![CDATA[<p>About a year ago I wrote about <a href="http://www.johndcook.com/blog/2011/05/03/jupiters-magic-square/">Jupiter&#8217;s magic square</a>. Then yesterday I was listening to the <a href="http://www.wnyc.org/shows/newsounds/articles/new-sounds-podcasts/2012/may/10/new-releases-january-2012-special-podcast/">New Sounds podcast</a> that mentioned a magic square associated with Mars. I hadn&#8217;t heard of this, so I looked into it and found there were magic squares associated with each of solar system bodies known to antiquity (i.e. Sun, Mercury, Venus, Moon, Mars, Jupiter, and Saturn).</p>
<p>Here is the magic square of Mars:</p>
<p style="text-align:center"><img src="http://www.johndcook.com/mars_magic_square.png" alt="" width="209" height="209" /></p>
<p>The podcast featured <a href="http://www.innova.mu/albums/zack-browning/secret-pulse">Secret Pulse</a> by Zack Browning. From the <a href="http://www.innova.mu/sites/default/files/liner-notes/817.txt">liner notes</a>:</p>
<blockquote><p>Magic squares provide structure to the music. Structure provides direction to the composer. Direction provides restrictions for the focused inspiration and interpretation of musical materials. The effect of this process? Freedom to compose.</p>
<p>The compositions on this CD use the 5&#215;5 Magic Square of Mars (Secret Pulse), the 9&#215;9 Magic Square of the Moon (Moon Thrust), and the ancient Chinese 3&#215;3 Lo Shu Square found in the Flying Star System of Feng Shui (Hakka Fusion, String Quartet, Flying Tones, and Moon Thrust) as compositional models.  The musical structure created from these magic squares is dramatically articulated by the collision of different musical worlds …</p></blockquote>
<p>I don&#8217;t know how the composer used these magic squares, but you can listen to the title track (Secret Pulse) on the <a href="http://www.wnyc.org/shows/newsounds/articles/new-sounds-podcasts/2012/may/10/new-releases-january-2012-special-podcast/">podcast</a>.</p>
<p><strong>Related posts</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2011/05/03/jupiters-magic-square/">Jupiter&#8217;s magic square</a><br />
<a href="http://www.johndcook.com/blog/2011/04/13/a-magic-kings-tour/">A magic king&#8217;s tour</a><br />
<a href="http://www.johndcook.com/blog/2011/04/06/a-knights-magic-square/">A magic knight&#8217;s tour</a><br />
<a href="http://www.johndcook.com/blog/2012/05/08/a-knights-random-walk/">A knight&#8217;s random walk</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/-43sbvrhDGU" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/16/mars-magic-squares-music/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/16/mars-magic-squares-music/</feedburner:origLink></item>
		<item>
		<title>Machine Learning in Action</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/t3BUmDQqs74/</link>
		<comments>http://www.johndcook.com/blog/2012/05/15/machine-learning-in-action/#comments</comments>
		<pubDate>Tue, 15 May 2012 13:23:14 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Python]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11369</guid>
		<description><![CDATA[A couple months ago I briefly reviewed Machine Learning for Hackers by Drew Conway and John Myles White. Today I&#8217;m looking at Machine Learning in Action by Peter Harrington and comparing the two books.
Both books are about the same size and cover many of the same topics. One difference between the two books is choice [...]]]></description>
			<content:encoded><![CDATA[<p>A couple months ago I briefly reviewed <a href="http://www.johndcook.com/blog/2012/03/07/machine-learning-for-hackers/">Machine Learning for Hackers</a> by Drew Conway and John Myles White. Today I&#8217;m looking at <a href="http://www.amazon.com/gp/product/1617290181/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=1617290181">Machine Learning in Action</a> by Peter Harrington and comparing the two books.</p>
<p>Both books are about the same size and cover many of the same topics. One difference between the two books is choice of programming language: <em>ML for Hackers</em> uses R for its examples, <em>ML in Action</em> uses Python.</p>
<p><em>ML in Action</em> doesn&#8217;t lean heavily on Python libraries. It mostly implements its algorithms from scratch, with a little help from NumPy for linear algebra, but it does not use ML libraries such as <a href="http://scikit-learn.org/stable/">scikit-learn</a>. It sometimes uses Matplotlib for plotting and uses Tkinter for building a simple GUI in one chapter. The final chapter introduces Hadoop and Amazon Web Services.</p>
<p><em>ML for Hackers</em> is a little more of a general introduction to machine learning. <em>ML in Action</em> contains a brief introduction to machine learning in general, but quickly moves on to specific algorithms. <em>ML for Hackers</em> spends a good number of pages discussing data cleaning. <em>ML in Action</em> starts with clean data in order to spend more time on algorithms.</p>
<p><em>ML in Action</em> takes 8 of the top 10 algorithms in machine learning (as selected by <a href="http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf">this paper</a>) and organizes around these algorithms. (The two algorithms out of the top 1o that didn&#8217;t make it into <em>ML in Action</em> were PageRank, because it has been covered well elsewhere, and EM, because its explanation requires too much mathematics.) The algorithms come first in <em>ML in Action</em>, illustrations second. <em>ML for Hackers</em> puts more emphasis on its examples and reads a bit more like a story. <em>ML in Action</em> reads a little more like a reference book.</p>
<p><a href="http://www.amazon.com/gp/product/1617290181/ref=as_li_ss_il?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=1617290181"><img src="http://ws.assoc-amazon.com/widgets/q?_encoding=UTF8&amp;Format=_SL160_&amp;ASIN=1617290181&amp;MarketPlace=US&amp;ID=AsinImage&amp;WS=1&amp;tag=theende-20&amp;ServiceVersion=20070822" border="0" alt="" /></a><img style="border:none !important; margin:0px !important;" src="http://www.assoc-amazon.com/e/ir?t=theende-20&amp;l=as2&amp;o=1&amp;a=1617290181" border="0" alt="" width="1" height="1" /></p>
<div id="_mcePaste" style="position: absolute; left: -10000px; top: 0px; width: 1px; height: 1px; overflow: hidden;">http://www.johndcook.com/blog/2008/06/27/wine-beer-and-statistics/#comment-170809</div>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/t3BUmDQqs74" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/15/machine-learning-in-action/feed/</wfw:commentRss>
		<slash:comments>7</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/15/machine-learning-in-action/</feedburner:origLink></item>
		<item>
		<title>Criteria for a computing setup</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/kYD7Yyeujtk/</link>
		<comments>http://www.johndcook.com/blog/2012/05/14/criteria-for-a-computing-setup/#comments</comments>
		<pubDate>Mon, 14 May 2012 11:05:36 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Computing]]></category>
		<category><![CDATA[Productivity]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11362</guid>
		<description><![CDATA[&#8220;My setup&#8221; articles have become common. These articles list the hardware and software someone uses, usually with little explanation. The subtext is often the author&#8217;s commitment to the Apple brand or to open source, to spending money on the best stuff or to avoid spending money on principle. I don&#8217;t find such articles interesting or [...]]]></description>
			<content:encoded><![CDATA[<p>&#8220;My setup&#8221; articles have become common. These articles list the hardware and software someone uses, usually with little explanation. The subtext is often the author&#8217;s commitment to the Apple brand or to open source, to spending money on the best stuff or to avoid spending money on principle. I don&#8217;t find such articles interesting or useful.</p>
<p><a href="http://blog.vivekhaldar.com/post/23021969189/my-setup">Vivek Haldar</a> has written a different kind of  &#8220;my setup&#8221; article, one that emphasizes the problems he set out to solve and the reasons for the solutions he chose. Here are a couple excerpts describing his goals for preserving his data and his health.</p>
<blockquote><p>Try to remember the oldest digital artifact that you can still  retrieve, and more importantly, decode and view. Can you? How old is it?  That should give you some idea of how hard and full of unknowns the  problem of long-term preservation is. …</p></blockquote>
<blockquote><p>If a significant fraction of your working life is spent working with  computers, and you do not yet have even the mildest RSI, you should  consider yourself extremely lucky, but not immune. Act like you do have  RSI, and change your set up right now to avoid it.</p></blockquote>
<p>I thought the best part of the article was the criteria, not the solutions. It&#8217;s not that I disapprove of his choices, but I appreciate more his explanation of the rationale behind his choices. I don&#8217;t expect anybody is going to read the article and say &#8220;That&#8217;s it! I&#8217;m going to copy that setup.&#8221; I gather that in at least one detail, his choice of version control system, Vivek wouldn&#8217;t even copy his own setup if he were to start over. But people will get ideas to consider in their own circumstances.</p>
<p><strong>Related post</strong>: <a href="http://www.johndcook.com/blog/2010/03/22/ford-chevy-arguments-in-tech/">Ford-Chevy arguments in tech</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/kYD7Yyeujtk" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/14/criteria-for-a-computing-setup/feed/</wfw:commentRss>
		<slash:comments>8</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/14/criteria-for-a-computing-setup/</feedburner:origLink></item>
		<item>
		<title>Solutions to knight’s random walk</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/QHICFsLcvQc/</link>
		<comments>http://www.johndcook.com/blog/2012/05/10/solutions-to-knights-random-walk/#comments</comments>
		<pubDate>Thu, 10 May 2012 12:10:50 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Python]]></category>
		<category><![CDATA[Math]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11350</guid>
		<description><![CDATA[My previous post asked this question:
Start a knight at a corner square of an otherwise-empty chessboard. Move  the knight at random by choosing uniformly from the legal knight-moves  at each step. What is the mean number of moves until the knight returns  to the starting square?
There is a mathematical solution that is [...]]]></description>
			<content:encoded><![CDATA[<p>My <a href="http://www.johndcook.com/blog/2012/05/08/a-knights-random-walk/">previous post</a> asked this question:</p>
<blockquote><p>Start a knight at a corner square of an otherwise-empty chessboard. Move  the knight at random by choosing uniformly from the legal knight-moves  at each step. What is the mean number of moves until the knight returns  to the starting square?</p></blockquote>
<p>There is a mathematical solution that is a little arcane, but short and exact. You could also approach the problem using simulation, which is more accessible but not exact.</p>
<p>The mathematical solution is to view the problem as a random walk on a graph. The vertices of the graph are the squares of a chess board and the edges connect legal knight moves. The general solution for the time to first return is simply 2<em>N</em>/<em>k</em> where <em>N</em> is the number of edges in the graph, and <em>k</em> is the number of edges meeting at the starting point. Amazingly, the solution hardly depends on the structure of the graph at all. It only requires that the graph is connected. In our case <em>N</em> = 168 and <em>k</em> = 2.</p>
<p>For a full explanation of the math, see this <a href="http://www.stat.berkeley.edu/~aldous/RWG/book.html">online book</a>, chapter 3, page 9. Start there and work your way backward until you understand the solution.</p>
<p>And now for simulation. The problem says to pick a legal knight&#8217;s move at random. The most direct approach would be to find the legal moves at a given point first, then choose one of those at random. The code below achieves the same end with a different approach. It first chooses a random move, and if that move is illegal (i.e. off the board) it throws that move away and tries again.  This will select a legal move with the right probability, though perhaps that&#8217;s not obvious. It&#8217;s what&#8217;s known as an accept-reject random generator.</p>
<pre class="brush: python; title: ; notranslate">
from random import randint

# Move a knight from (x, y) to a random new position
def new_position(x, y):

    while True:
        dx, dy = 1, 2

        # it takes three bits to determine a random knight move:
        # (1, 2) vs (2, 1), and the sign of each
        r = randint(0, 7)
        if r % 2:
            dx, dy = dy, dx
        if (r &gt;&gt; 1) % 2:
            dx = -dx
        if (r &gt;&gt; 2) % 2:
            dy = -dy

        newx, newy = x + dx, y + dy
        # If the new position is on the board, take it.
        # Otherwise try again.
        if (newx &gt;= 0 and newx &lt; 8 and newy &gt;= 0 and newy &lt; 8):
            return (newx, newy)

# Count the number of steps in one random tour
def random_tour():
    x, y = x0, y0 = 0, 0
    count = 0
    while True:
        x, y = new_position(x, y)
        count += 1
        if x == x0 and y == y0:
            return count

# Average the length of many random tours
sum = 0
num_reps = 100000
for i in xrange(num_reps):
    sum += random_tour()
print sum / float(num_reps)
</pre>
<p>A theorem is better than a simulation, but a simulation is a lot better than nothing. This problem illustrates how sometimes we think we need to simulate when we don&#8217;t. On the other hand, when you have a simulation <em>and</em> a theorem, you have more confidence in your solution because <a href="http://www.johndcook.com/blog/2008/01/10/complementary-validation/">each validates the other</a>.</p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/QHICFsLcvQc" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/10/solutions-to-knights-random-walk/feed/</wfw:commentRss>
		<slash:comments>10</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/10/solutions-to-knights-random-walk/</feedburner:origLink></item>
		<item>
		<title>A knight’s random walk</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/vLkHCk7_1H8/</link>
		<comments>http://www.johndcook.com/blog/2012/05/08/a-knights-random-walk/#comments</comments>
		<pubDate>Wed, 09 May 2012 00:33:51 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Math]]></category>
		<category><![CDATA[Probability and Statistics]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11342</guid>
		<description><![CDATA[Here&#8217;s a puzzle I ran across today:
Start a knight at a corner square of an otherwise-empty chessboard. Move the knight at random by choosing uniformly from the legal knight-moves at each step. What is the mean number of moves until the knight returns to the starting square?
There&#8217;s a slick mathematical solution that I&#8217;ll give later.
You [...]]]></description>
			<content:encoded><![CDATA[<p>Here&#8217;s a puzzle I ran across today:</p>
<blockquote><p>Start a knight at a corner square of an otherwise-empty chessboard. Move the knight at random by choosing uniformly from the legal knight-moves at each step. What is the mean number of moves until the knight returns to the starting square?</p></blockquote>
<p>There&#8217;s a slick mathematical solution that I&#8217;ll give later.</p>
<p>You could also find the answer via simulation: write a program to carry out a knight random walk and count how many steps it takes. Repeat this many times and average your counts.</p>
<p><img src="http://www.johndcook.com/knight.png" alt="knight" width="79" height="84" /></p>
<p><strong>Related post</strong>: <a href="http://www.johndcook.com/blog/2011/04/06/a-knights-magic-square/">A knight&#8217;s tour magic square</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/vLkHCk7_1H8" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/08/a-knights-random-walk/feed/</wfw:commentRss>
		<slash:comments>36</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/08/a-knights-random-walk/</feedburner:origLink></item>
		<item>
		<title>Web architecture</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/UZGIZH3B2c8/</link>
		<comments>http://www.johndcook.com/blog/2012/05/08/web-architecture/#comments</comments>
		<pubDate>Tue, 08 May 2012 12:00:40 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Software development]]></category>
		<category><![CDATA[Programming]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11333</guid>
		<description><![CDATA[Robert &#8220;Uncle Bob&#8221; Martin argues that an application&#8217;s purpose should determine its architecture; its delivery mechanism should not. If you look at an architectural drawing of a church, for example, you can tell it&#8217;s a church. But if you look at an architectural drawing of a web application, the fact that it&#8217;s a web application [...]]]></description>
			<content:encoded><![CDATA[<p>Robert &#8220;Uncle Bob&#8221; Martin argues that an application&#8217;s purpose should determine its architecture; its delivery mechanism should not. If you look at an architectural drawing of a church, for example, you can tell it&#8217;s a church. But if you look at an architectural drawing of a web application, the fact that it&#8217;s a web application should not be the first thing you notice.</p>
<blockquote><p>The web is a delivery mechanism. The web is a detail. The web is not particularly important to your application. … The web is a pipe. It is nothing more than a pipe. It is not the central abstraction of your application. It is not the grand, over-arching thing that makes your application the application it is. The web is just a dumb detail. And yet it dominates our code.</p></blockquote>
<p>From Robert Martin&#8217;s <a href="http://www.confreaks.com/videos/759-rubymidwest2011-keynote-architecture-the-lost-years">keynote</a> at Ruby Midwest 2011. The quote starts about 9 minutes into the presentation.</p>
<p>He goes on to explain how to let the functionality of the application determine its architecture. In the dependency diagram arrows point <em>from</em> the delivery mechanism <em>to</em> the rest of the code, but none point the other way. By following this design you get something easier to test.</p>
<p><strong>Related posts</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2009/02/19/recap-of-the-robert-martinjoel-spolsky-brouhaha/">Recap of the Robert Martin/Joel Spolksy brouhaha</a><br />
<a href="http://www.johndcook.com/blog/2008/10/27/why-there-will-always-be-programmers/">Why there will always be programmers</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/UZGIZH3B2c8" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/08/web-architecture/feed/</wfw:commentRss>
		<slash:comments>16</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/08/web-architecture/</feedburner:origLink></item>
		<item>
		<title>How do you know when someone is great?</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/6Ktw-60pcrQ/</link>
		<comments>http://www.johndcook.com/blog/2012/05/07/how-do-you-know-when-someone-is-great/#comments</comments>
		<pubDate>Mon, 07 May 2012 14:39:06 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Business]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11327</guid>
		<description><![CDATA[Roger Peng asks a good question on his blog this morning: How would you know if someone is great at data analysis? He says that while he has worked with some great data analysts, the nature of their work is that it&#8217;s hard to evaluate the work of someone you don&#8217;t know personally. And as [...]]]></description>
			<content:encoded><![CDATA[<p>Roger Peng asks a good question on <a href="http://simplystatistics.tumblr.com/post/22585430491/how-do-you-know-if-someone-is-great-at-data-analysis">his blog</a> this morning: How would you know if someone is great at data analysis? He says that while he has worked with some great data analysts, the nature of their work is that it&#8217;s hard to evaluate the work of someone you don&#8217;t know personally. And as <a href="https://twitter.com/#!/joshin4colours/status/199496712413384705">Josh Grant</a> pointed out, this isn&#8217;t unique to data analysts.</p>
<p>I immediately thought of a database administrator I know. Everyone who works with her know she&#8217;s great at her job, but I doubt anyone who doesn&#8217;t know her has ever said &#8220;They must have a great DBA!&#8221;</p>
<p>Matthew Crawford argues in <a href="http://www.amazon.com/gp/product/0143117467/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0143117467">Shop Class as Soulcraft</a> that white collar work in general is hard to objectively evaluate and that this explains why offices are so political. Employees are judged on their sensitivity and other nebulous attributes because unlike a welder, for example, they can&#8217;t be judged directly on their work. He argues that blue collar people workers greater freedom of speech at work because their work can be objectively evaluated.</p>
<p>Colleagues can identify great data analysts, DBAs, and others whose work isn&#8217;t on public display. But this isn&#8217;t easy to do through a bureaucratic process, and so technical competence is routinely under-compensated in large organizations. On the other hand, reputation spreads more efficiently outside of organizational channels. This may help explain why highly competent people are often more appreciated by their professional communities than by their employers.</p>
<p><strong>Related post</strong>: <a href="http://www.johndcook.com/blog/2009/11/22/the-computer-guy/">It doesn&#8217;t pay to be the computer guy</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/6Ktw-60pcrQ" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/07/how-do-you-know-when-someone-is-great/feed/</wfw:commentRss>
		<slash:comments>7</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/07/how-do-you-know-when-someone-is-great/</feedburner:origLink></item>
		<item>
		<title>An algebra problem from 1798</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/raKrwqKdI_8/</link>
		<comments>http://www.johndcook.com/blog/2012/05/05/ladys-diary-1798/#comments</comments>
		<pubDate>Sat, 05 May 2012 13:20:21 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[History]]></category>
		<category><![CDATA[Math]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11325</guid>
		<description><![CDATA[The Lady&#8217;s Diary was a popular magazine published in England from 1704 to 1841. It contained mathematical puzzles such as the following, published in 1798.
What two numbers are those whose product, difference of their squares, and the ratio or quotient of their cubes, are all equal to each other?
From Benjamin Wardhaugh&#8217;s new book A Wealth [...]]]></description>
			<content:encoded><![CDATA[<p><em>The Lady&#8217;s Diary</em> was a popular magazine published in England from 1704 to 1841. It contained mathematical puzzles such as the following, published in 1798.</p>
<blockquote><p>What two numbers are those whose product, difference of their squares, and the ratio or quotient of their cubes, are all equal to each other?</p></blockquote>
<p>From Benjamin Wardhaugh&#8217;s <a href="http://www.amazon.com/gp/product/0691147752/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0691147752">new book</a> A Wealth of Numbers: An Anthology of 500 Years of Popular Mathematics Writing.</p>
<p><a href="http://www.amazon.com/gp/product/0691147752/ref=as_li_ss_il?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0691147752"><img src="http://ws.assoc-amazon.com/widgets/q?_encoding=UTF8&amp;Format=_SL160_&amp;ASIN=0691147752&amp;MarketPlace=US&amp;ID=AsinImage&amp;WS=1&amp;tag=theende-20&amp;ServiceVersion=20070822" border="0" alt="" /></a><img style="border:none !important; margin:0px !important;" src="http://www.assoc-amazon.com/e/ir?t=theende-20&amp;l=as2&amp;o=1&amp;a=0691147752" border="0" alt="" width="1" height="1" /></p>
<p>See also my <a href="http://www.johndcook.com/blog/2012/05/02/reading-historical-math/">brief review</a> of <a href="http://www.amazon.com/gp/product/0691140146/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0691140146">How to Read Historical Mathematics</a> by the same author.</p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/raKrwqKdI_8" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/05/ladys-diary-1798/feed/</wfw:commentRss>
		<slash:comments>19</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/05/ladys-diary-1798/</feedburner:origLink></item>
		<item>
		<title>Beer with a confidence interval</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/ZzkmM01adG4/</link>
		<comments>http://www.johndcook.com/blog/2012/05/04/medalla-confidence-interval/#comments</comments>
		<pubDate>Fri, 04 May 2012 13:18:39 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Statistics]]></category>
		<category><![CDATA[Probability and Statistics]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11316</guid>
		<description><![CDATA[Medalla is a Puerto Rican beer. On the side of a can it says
Alcohol by volume over 4%, not more than 6%.
I&#8217;d never seen a beer give a confidence interval for its alcohol content. I&#8217;d only seen point estimates before. For example, Budweiser announces that its beer contains 5% alcohol by volume. Just 5%, no [...]]]></description>
			<content:encoded><![CDATA[<p>Medalla is a Puerto Rican beer. On the side of a can it says</p>
<blockquote><p>Alcohol by volume over 4%, not more than 6%.</p></blockquote>
<p>I&#8217;d never seen a beer give a confidence interval for its alcohol content. I&#8217;d only seen point estimates before. For example, Budweiser announces that its beer contains 5% alcohol by volume. Just 5%, no statistical details.</p>
<p>I wonder why Medalla reports an interval.</p>
<ul>
<li>Is their manufacturing process so variable that they can&#8217;t just report an average?</li>
<li>Is their manufacturing process no more variable than others, but they disclose their uncertainty?</li>
<li>Are they required to report alcohol content the way they do by local law or by the law of countries they export to?</li>
</ul>
<p><strong>Related posts</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2008/06/27/wine-beer-and-statistics/">Beer, wine, and statistics</a><br />
<a href="http://www.johndcook.com/blog/2008/09/27/wine-and-politics/">Wine and politics</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/ZzkmM01adG4" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/04/medalla-confidence-interval/feed/</wfw:commentRss>
		<slash:comments>10</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/04/medalla-confidence-interval/</feedburner:origLink></item>
		<item>
		<title>Visiting Puerto Rico</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/4iEaMcFZPlQ/</link>
		<comments>http://www.johndcook.com/blog/2012/05/04/visiting-puerto-rico/#comments</comments>
		<pubDate>Fri, 04 May 2012 12:16:30 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11311</guid>
		<description><![CDATA[I&#8217;ve been in San Juan this week, visiting the University of Puerto Rico. I&#8217;ve been here several times before, but here are a few things I noticed about Puerto Rico on this trip.
Coffee: Coffee means espresso here; I haven&#8217;t seen it brewed any other way. And it&#8217;s cheap. At UPR, a 4 oz pocillo is [...]]]></description>
			<content:encoded><![CDATA[<p>I&#8217;ve been in San Juan this week, visiting the University of Puerto Rico. I&#8217;ve been here several times before, but here are a few things I noticed about Puerto Rico on this trip.</p>
<p><strong>Coffee</strong>: Coffee means espresso here; I haven&#8217;t seen it brewed any other way. And it&#8217;s cheap. At UPR, a 4 oz <em>pocillo</em> is only $0.70, and at a bakery near my hotel a 6 oz <em>espresso con leche</em> is $1.25.</p>
<p><strong>Gasoline</strong>: You can&#8217;t pay at the pump with a credit card. You have to go inside to pay for gas. Some gas stations used to let you pay at the pump, but these have gone back to having you pay inside.</p>
<p><strong>Electronics</strong>: I&#8217;ve been told that you can&#8217;t buy electronics from Apple&#8217;s web site for delivery in Puerto Rico. You can buy Apple products in stores like Best Buy on the island, but you can&#8217;t have them shipped directly here from their web site.</p>
<p style="text-align: center;">***</p>
<p><strong>Related post</strong>: <a href="http://www.johndcook.com/blog/2012/05/04/medalla-confidence-interval/">Beer with a confidence interval</a> (Medalla)</p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/4iEaMcFZPlQ" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/04/visiting-puerto-rico/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/04/visiting-puerto-rico/</feedburner:origLink></item>
		<item>
		<title>Python as a Lisp dialect</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/E-WPgi4oDdc/</link>
		<comments>http://www.johndcook.com/blog/2012/05/03/python-as-a-lisp-dialect/#comments</comments>
		<pubDate>Thu, 03 May 2012 11:55:11 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Python]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11275</guid>
		<description><![CDATA[From Peter Norvig:
Basically, Python can be seen as a dialect of Lisp with &#8220;traditional&#8221; syntax … Python supports all of Lisp&#8217;s essential features except macros, and you don&#8217;t miss macros all that much because it does have eval, and operator overloading, and regular expression parsing, so some — but not all — of the use [...]]]></description>
			<content:encoded><![CDATA[<p>From Peter Norvig:</p>
<blockquote><p>Basically, Python can be seen as a dialect of Lisp with &#8220;traditional&#8221; syntax … Python supports all of Lisp&#8217;s essential features except macros, and you don&#8217;t miss macros all that much because it does have <code>eval</code>, and operator overloading, and regular expression parsing, so some — but not all — of the use cases for macros are covered.</p></blockquote>
<p>Source: <a href="http://norvig.com/python-lisp.html">Python for Lisp Programmers</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/E-WPgi4oDdc" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/03/python-as-a-lisp-dialect/feed/</wfw:commentRss>
		<slash:comments>9</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/03/python-as-a-lisp-dialect/</feedburner:origLink></item>
		<item>
		<title>Reading historical math</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/5I0dTq5EAJU/</link>
		<comments>http://www.johndcook.com/blog/2012/05/02/reading-historical-math/#comments</comments>
		<pubDate>Wed, 02 May 2012 18:58:34 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Math]]></category>
		<category><![CDATA[Books]]></category>
		<category><![CDATA[History]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11301</guid>
		<description><![CDATA[I recently received review copies of two books by Benjamin Wardhaugh. Here I will discuss How to Read Historical Mathematics. The other book is his anthology of historical popular mathematics which I intend to review later.
Here is the key passage, located near the end of How to Read Historical Mathematics, for identifying the author&#8217;s perspective.
But [...]]]></description>
			<content:encoded><![CDATA[<p>I recently received review copies of two books by Benjamin Wardhaugh. Here I will discuss <a href="http://www.amazon.com/gp/product/0691140146/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0691140146">How to Read Historical Mathematics</a>. The other book is his <a href="http://www.benjaminwardhaugh.co.uk/wealth.html">anthology</a> of historical popular mathematics which I intend to review later.</p>
<p>Here is the key passage, located near the end of How to Read Historical Mathematics, for identifying the author&#8217;s perspective.</p>
<blockquote><p>But not all historical mathematics <em>is</em> significant. And perhaps there is a second kind of significance, where something can be historically significant without being mathematically significant. Some historians (I&#8217;m one of them) delight in investigating mathematical writing that contains little or no important or novel mathematics: popular textbooks, self-instruction manuals, … or old almanacs and popular magazines with mathematical news or puzzles in them. These kinds of writing … are certainly significant for a historian who wants to know about popular experiences of mathematics. But they&#8217;re not significant in the sense of containing significant mathematics.</p></blockquote>
<p>Wardhaugh&#8217;s perspective is valuable, though it is not one that I share. My interest in historical math is more on the development of the mathematical ideas rather than their social context. I&#8217;m interested, for example, in discovering the concrete problems that motivated mathematics that has become more abstract and formal.</p>
<p>I was hoping for something more along the lines of a mapping from historical definitions and notations to their modern counterparts. This book contains a little of that, but it focuses more on how to read historical mathematics as a <em>historian</em> rather than as a <em>mathematician</em>. However, if you are interested in more of the social angle, the book has many good suggestions (and even exercises) for exploring the larger context of historical mathematical writing.</p>
<p><a href="http://www.amazon.com/gp/product/0691140146/ref=as_li_ss_il?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0691140146"><img src="http://ws.assoc-amazon.com/widgets/q?_encoding=UTF8&amp;Format=_SL160_&amp;ASIN=0691140146&amp;MarketPlace=US&amp;ID=AsinImage&amp;WS=1&amp;tag=theende-20&amp;ServiceVersion=20070822" border="0" alt="" /></a><img style="border:none !important; margin:0px !important;" src="http://www.assoc-amazon.com/e/ir?t=theende-20&amp;l=as2&amp;o=1&amp;a=0691140146" border="0" alt="" width="1" height="1" /></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/5I0dTq5EAJU" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/02/reading-historical-math/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/02/reading-historical-math/</feedburner:origLink></item>
		<item>
		<title>Big data is easy</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/MH9tMRBQWAI/</link>
		<comments>http://www.johndcook.com/blog/2012/05/01/big-data-is-easy/#comments</comments>
		<pubDate>Tue, 01 May 2012 14:16:12 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Statistics]]></category>
		<category><![CDATA[Probability and Statistics]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11297</guid>
		<description><![CDATA[Big data is easy; big models are hard.
If you just wanted to use simple models with tons of data, that would be easy. You could resample the data, throwing some of it away until you had a quantity of data you could comfortably manage.
But when you have tons of data, you want to take advantage [...]]]></description>
			<content:encoded><![CDATA[<p>Big data is easy; big models are hard.</p>
<p>If you just wanted to use simple models with tons of data, that would be easy. You could resample the data, throwing some of it away until you had a quantity of data you could comfortably manage.</p>
<p>But when you have tons of data, you want to take advantage of it and ask questions that simple models cannot answer. (&#8221;Big&#8221; data is often<em> indirect</em> data.) So the problem isn&#8217;t that you <em>have</em> a lot of data, it&#8217;s that you&#8217;re using models that <em>require</em> a lot of data. And that can be very hard.</p>
<p>I am not saying people should just use simple models. No, people are right to want to take advantage of their data, and often that does require complex models. (See Brad Efron&#8217;s <a href="http://www.johndcook.com/blog/2010/12/15/big-data-is-not-enough/">explanation</a> why.) But the primary challenge is intellectual, not physical.</p>
<p><strong>Related post</strong>:<a href="http://www.johndcook.com/blog/2010/12/15/big-data-is-not-enough/"></a> <a href="http://www.johndcook.com/blog/2011/09/22/big-data-and-humility/">Big data and humility</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/MH9tMRBQWAI" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/01/big-data-is-easy/feed/</wfw:commentRss>
		<slash:comments>5</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/01/big-data-is-easy/</feedburner:origLink></item>
		<item>
		<title>Traveling salesman art</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/8AgiVHGuNd0/</link>
		<comments>http://www.johndcook.com/blog/2012/05/01/traveling-salesman-art/#comments</comments>
		<pubDate>Tue, 01 May 2012 12:00:24 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Computing]]></category>
		<category><![CDATA[Math]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11285</guid>
		<description><![CDATA[Bill Cook sent me a file yesterday that renders the Endeavour photo on my blog as the solution to a 66,290-city Traveling Salesman problem.  His iPhone app Concord TSP chose 66,290 points and then solved for the shortest path connecting these points, a feat that would have strained a supercomputer a few years ago. [...]]]></description>
			<content:encoded><![CDATA[<p>Bill Cook sent me a file yesterday that renders the Endeavour photo on my blog as the solution to a 66,290-city Traveling Salesman problem.  His iPhone app <a href="http://itunes.apple.com/us/app/concorde-tsp/id498366515?mt=8">Concord TSP</a> chose 66,290 points and then solved for the shortest path connecting these points, a feat that would have strained a supercomputer a few years ago. (Bill Cook and I are not related as far as I know.)</p>
<p>Here is a thumbnail image of the full TSP tour:</p>
<p style="text-align:center"><img src="http://www.johndcook.com/endeavour_tsp.png" height="301" width="400"></p>
<p>You can find the full PDF <a href="http://www.johndcook.com/endeavour_tsp.pdf">here</a> (1.24 MB). To show some of the detail, here is a close-up from near the top-left corner of the image:</p>
<p style="text-align:center"><img src="http://www.johndcook.com/tsp_closeup.png" height="209" width="347"></p>
<p>I asked how the tour was constructed:</p>
<blockquote><p>
How do you construct a set of points whose TSP solution resembles a photograph? Is it sufficient to add more &#8220;cities&#8221; in regions where you want darker shading? And are the cities added at random with a density specified by color depth?
</p></blockquote>
<p>Bill Cook replied:</p>
<blockquote><p>
By default, the app will select the points along the line you describe: it splits the image into a grid, computes the average gray scale in each grid region, and drops a number of random cities into each grid region in proportion to the square of the average gray scale. This technique was first proposed by Bob Bosch and Adrianne Herman at Oberlin College. It is the default since it takes almost no time to compute, but I include two other options, that each take about a minute to render a large image on an iPhone 4. </p>
<p>The image of The Endeavour was created with a method Jim Bumgarnder proposed in his <a href="http://joyofprocessing.com/blog/2011/11/stipple-cam/">Stipple Cam project</a>.
</p></blockquote>
<p><strong>Related post:</strong></p>
<p><a href="http://www.johndcook.com/blog/2012/01/01/moores-law-squared/">Moore&#8217;s law squared</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/8AgiVHGuNd0" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/05/01/traveling-salesman-art/feed/</wfw:commentRss>
		<slash:comments>9</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/05/01/traveling-salesman-art/</feedburner:origLink></item>
		<item>
		<title>Conforming for tenure</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/4AR4UuorRh4/</link>
		<comments>http://www.johndcook.com/blog/2012/04/30/conforming-for-tenure/#comments</comments>
		<pubDate>Mon, 30 Apr 2012 18:00:31 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11283</guid>
		<description><![CDATA[From AnnMaria De Mars&#8217; most recent blog post:
Recently, a young person told me that I could hold to my principles about the importance of my family, honesty  and equality &#8212; and any of a hundred other things because I had “made  it”.
This troubled me. It troubles me when I hear the same thing [...]]]></description>
			<content:encoded><![CDATA[<p>From AnnMaria De Mars&#8217; most recent <a href="http://www.thejuliagroup.com/blog/?p=2373">blog post</a>:</p>
<blockquote><p>Recently, a young person told me that <em>I</em> could hold to my principles about the importance of my family, honesty  and equality &#8212; and any of a hundred other things because I had “made  it”.</p>
<p>This troubled me. It troubles me when I hear the same thing from new  Ph.Ds who are trying to get tenure.<strong> I don’t see how you can pretend to  be someone else for 5 or 10 years until you have “made it” and then be  your true self</strong>.</p></blockquote>
<p>Emphasis added.</p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/4AR4UuorRh4" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/30/conforming-for-tenure/feed/</wfw:commentRss>
		<slash:comments>12</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/30/conforming-for-tenure/</feedburner:origLink></item>
		<item>
		<title>Open source and pride</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/STAZNQkWYB8/</link>
		<comments>http://www.johndcook.com/blog/2012/04/28/open-source-and-pride/#comments</comments>
		<pubDate>Sat, 28 Apr 2012 11:38:43 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Computing]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11273</guid>
		<description><![CDATA[Liz Quilty explains how becoming an expert in open source software changed her life.
[I was a] high school dropout, I had no education, I was nobody. I&#8217;d made some poor choices, and I think at this point suddenly I was known for my knowledge rather than for my poor choices. And so it was quite [...]]]></description>
			<content:encoded><![CDATA[<p>Liz Quilty explains how becoming an expert in open source software changed her life.</p>
<blockquote><p>[I was a] high school dropout, I had no education, I was nobody. I&#8217;d made some poor choices, and I think at this point suddenly I was known for my knowledge rather than for my poor choices. And so it was quite good for me that I had some pride. I was like, this is really awesome, I&#8217;m really good at this, I really know this.</p></blockquote>
<p>Source: <a href="http://twit.tv/show/floss-weekly/208">FLOSS Weekly</a> podcast, starting around 12:00.</p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/STAZNQkWYB8" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/28/open-source-and-pride/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/28/open-source-and-pride/</feedburner:origLink></item>
		<item>
		<title>A tip on using a French press</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/HKp535ttZps/</link>
		<comments>http://www.johndcook.com/blog/2012/04/27/using-french-press/#comments</comments>
		<pubDate>Fri, 27 Apr 2012 13:02:49 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[Coffee]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11119</guid>
		<description><![CDATA[When I first bought a French press,  the instructions said to pour hot but not boiling water over the coffee. They were emphatic about what the temperature should not be, but vague about what it should be. (Boiling water extracts oils that you&#8217;d rather leave in the grounds; water a little cooler brings out [...]]]></description>
			<content:encoded><![CDATA[<p>When I first bought a <a href="http://www.amazon.com/gp/product/B005FFBV5I/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=B005FFBV5I">French press</a>,  the instructions said to pour hot but not boiling water over the coffee. They were emphatic about what the temperature should <em>not</em> be, but vague about what it <em>should</em> be. (Boiling water extracts oils that you&#8217;d rather leave in the grounds; water a little cooler brings out just the oils you want.) </p>
<p>I asked someone online &#8212; sorry, I don&#8217;t remember who &#8212; and he said that after your water boils, set the kettle off the burner and check your mail. When you come back, the water should be at the right temperature. That turned out to be good advice.</p>
<p>I don&#8217;t know whether he meant postal mail or email, but it doesn&#8217;t make much difference. Unless you get caught in replying to email and come back to room-temperature water.</p>
<p>Now if you really wanted to geek out on this, you could use <a href="http://en.wikipedia.org/wiki/Newton%27s_law_of_cooling#Newton.27s_law_of_cooling">Newton&#8217;s law of cooling</a> along with the surface area, thickness, and material composition of your kettle to compute the time to let your water cool to 200 °F (93 °C). You could assume a kettle is a half sphere …</p>
<p><strong>Related post</strong>: <a href="http://www.johndcook.com/blog/2010/03/10/cooling/">A childhood question about heat</a></p>
<p><a href="http://www.amazon.com/gp/product/B005FFBV5I/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=B005FFBV5I"><img src="http://www.johndcook.com/french_press.jpg" alt="French press photo" width="190" height="274" /></a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/HKp535ttZps" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/27/using-french-press/feed/</wfw:commentRss>
		<slash:comments>11</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/27/using-french-press/</feedburner:origLink></item>
		<item>
		<title>How variable are percentiles?</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/Zl1Q1e-lqOg/</link>
		<comments>http://www.johndcook.com/blog/2012/04/27/how-variable-are-percentiles/#comments</comments>
		<pubDate>Fri, 27 Apr 2012 11:33:15 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Statistics]]></category>
		<category><![CDATA[Probability and Statistics]]></category>
		<category><![CDATA[Python]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11228</guid>
		<description><![CDATA[Suppose you&#8217;re trying to study the distribution of something by simulation. The average of your simulation values gives you an estimate of the mean value of the thing you&#8217;re simulating.
Next you want to have an idea how much the thing you&#8217;re simulating varies. You estimate the percentiles of your distribution by the percentiles of your [...]]]></description>
			<content:encoded><![CDATA[<p>Suppose you&#8217;re trying to study the distribution of something by simulation. The average of your simulation values gives you an estimate of the mean value of the thing you&#8217;re simulating.</p>
<p>Next you want to have an idea how much the thing you&#8217;re simulating varies. You estimate the percentiles of your distribution by the percentiles of your samples. How variable are your estimates? For example, you estimate the 90th percentile of your distribution by finding the 90th percentile of your samples. How much is your estimate likely to change if you rerun you simulation with a new random number generator seed?</p>
<p>How does the variability of the percentiles depend on the percentile you&#8217;re looking at? For example, you might expect the median, i.e. the 50th percentile, to be the most stable. And as you start looking at more extreme percentiles, say the 5th or 95th, you might expect more variability. But <em>how much</em> more variability? A lot more or a little more?</p>
<p>To explore the questions above, let&#8217;s suppose we&#8217;re sampling from a standard normal distribution. (Of course if we really were sampling from something so well known as a normal distribution, we wouldn&#8217;t be doing simulation. But we need to pick something for illustration, so let&#8217;s use something simple.)</p>
<p>Suppose we simulate <em>n</em> values, sort them, and take the <em>k</em>th largest value.  This is called the <em>k</em>th order statistic. For example, we might estimate the 90th percentile of our distribution by using <em>n</em> = 1000 and <em>k</em> = 900. The distribution of the <em>k</em>th order statistic from <em>n</em> samples is known analytically (it&#8217;s a short derivation; see <a href="http://projecteuclid.org/DPubS?verb=Display&amp;version=1.0&amp;service=UI&amp;handle=euclid.aoms/1177704982&amp;page=record">here</a>) and so we can use it to make some plots.</p>
<p style="text-align:center"><img src="http://www.johndcook.com/orderstat.png" alt="" width="402" height="484" /></p>
<p>These plots use <em>n</em> = 1000. Each uses the same horizontal scale, so you can see how the distributions get wider as we move to higher percentiles. The distribution for the 95th percentile is about twice as wide as the distribution for the median.</p>
<p>Each plot is centered at the corresponding quantile for the standard normal distribution. For example, the plot for the distribution of the 75th percentile of the samples is centered at the 75th percentile of the standard normal.</p>
<p>The distribution for the sample median is unbiased, symmetric about 0. The distributions for the 75th and 90th percentiles are slightly biased to the left, i.e. they slightly underestimate the true values on average. The 95th percentile, however, is slightly biased in the opposite direction. [Update: As pointed out in the comments below, it seems the appearance of bias was a plotting artifact.]</p>
<p>If you&#8217;d like to see the details of how the plot was made, here is the Python code.</p>
<pre class="brush: python; title: ; notranslate">
import scipy as sp
import scipy.stats
import scipy.special
import matplotlib.pyplot as plt

def pdf(y, n, k):
    Phi = scipy.stats.norm.cdf(y)
    phi = scipy.stats.norm.pdf(y)
    return scipy.stats.beta.pdf(Phi, k, n+1-k)*phi

def plot_percentile(p):
    center = scipy.stats.norm.ppf(0.01*p)
    t = sp.linspace(center-0.25, center+0.25, 100)
    plt.plot(t, pdf(t, 1000, 10*p))
    plt.title( &quot;{0}th percentile&quot;.format(p) )

plt.subplot(221)
plot_percentile(50)
plt.subplot(222)
plot_percentile(75)
plt.subplot(223)
plot_percentile(90)
plt.subplot(224)
plot_percentile(95)
plt.show()
</pre>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/Zl1Q1e-lqOg" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/27/how-variable-are-percentiles/feed/</wfw:commentRss>
		<slash:comments>5</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/27/how-variable-are-percentiles/</feedburner:origLink></item>
		<item>
		<title>Simmer reading list</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/qqCfBORS1C8/</link>
		<comments>http://www.johndcook.com/blog/2012/04/26/simmer-reading-list/#comments</comments>
		<pubDate>Thu, 26 Apr 2012 12:00:52 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[Books]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11224</guid>
		<description><![CDATA[One of my friends mentioned his &#8220;simmer reading&#8221; yesterday. It was a typo &#8212; he meant to say &#8220;summer&#8221; &#8212; but a simmer reading list is interesting.
Simmer reading makes me think of a book that stays on your nightstand as other books come and go, like a pot left to simmer on the back burner [...]]]></description>
			<content:encoded><![CDATA[<p>One of my friends mentioned his &#8220;simmer reading&#8221; yesterday. It was a typo &#8212; he meant to say &#8220;summer&#8221; &#8212; but a simmer reading list is interesting.</p>
<p>Simmer reading makes me think of a book that stays on your nightstand as other books come and go, like a pot left to simmer on the back burner of a stove. It&#8217;s a book you read a little at a time, maybe out of order, not something you&#8217;re trying to finish like a project.</p>
<p>What are some of your simmer reading books?</p>
<p><strong>Related post</strong>: <a href="http://www.johndcook.com/blog/2010/02/12/great-books/">A book so good I had to put it down</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/qqCfBORS1C8" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/26/simmer-reading-list/feed/</wfw:commentRss>
		<slash:comments>32</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/26/simmer-reading-list/</feedburner:origLink></item>
		<item>
		<title>The cult of average</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/O1HqhWRAI8M/</link>
		<comments>http://www.johndcook.com/blog/2012/04/25/cult-of-average/#comments</comments>
		<pubDate>Wed, 25 Apr 2012 11:45:48 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Creativity]]></category>
		<category><![CDATA[Science]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11189</guid>
		<description><![CDATA[Shawn Achor comments on &#8220;the cult of the average&#8221; in science.
So one of the very first things we teach people in economics and statistics and business and psychology is how, in a statistically valid way, do we eliminate the weirdos. How do we eliminate the outliers so we can find the line of best fit? [...]]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.ted.com/talks/shawn_achor_the_happy_secret_to_better_work.html">Shawn Achor</a> comments on &#8220;the cult of the average&#8221; in science.</p>
<blockquote><p>So one of the very first things we teach people in economics and statistics and business and psychology is how, in a statistically valid way, do we eliminate the weirdos. How do we eliminate the outliers so we can find the line of best fit? Which is fantastic if I&#8217;m trying to find out how many Advil the average person should be taking — two. But if I&#8217;m interested in potential, if I&#8217;m interested in your potential, or for happiness or productivity or energy or creativity, what we&#8217;re doing is we&#8217;re creating the cult of the average with science. … <strong>If we study what is merely average, we will remain merely average</strong>.</p></blockquote>
<p><img class="alignnone" src="http://www.johndcook.com/cult_of_average.png" alt="" width="301" height="210" /></p>
<p><strong>Related posts</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2010/07/19/stupidity-scales/">Stupidity scales</a><br />
<a href="http://www.johndcook.com/blog/2008/10/20/nearly-everyone-is-above-average/">Above average legs</a><br />
<a href="http://www.johndcook.com/blog/2009/09/29/achievement-is-log-normal/">Achievement is not normal</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/O1HqhWRAI8M" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/25/cult-of-average/feed/</wfw:commentRss>
		<slash:comments>6</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/25/cult-of-average/</feedburner:origLink></item>
		<item>
		<title>Chaotic versus random</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/StNJ6gFybIY/</link>
		<comments>http://www.johndcook.com/blog/2012/04/24/chaos-vs-randomness/#comments</comments>
		<pubDate>Tue, 24 Apr 2012 12:00:52 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[Probability and Statistics]]></category>
		<category><![CDATA[Science]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11210</guid>
		<description><![CDATA[From John D. Barrow&#8217;s chapter in Design and Disorder:
The standard folklore about chaotic systems is that they are unpredictable. They lead to out-of-control dinosaur parks and out-of-work meteorologists. …
Classical … chaotic systems are not in any sense intrinsically random or unpredictable. They merely possess extreme sensitivity to ignorance. Any initial uncertainty in our knowledge of [...]]]></description>
			<content:encoded><![CDATA[<p>From John D. Barrow&#8217;s chapter in <a href="http://www.amazon.com/gp/product/0567088685/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0567088685">Design and Disorder</a>:</p>
<blockquote><p>The standard folklore about chaotic systems is that they are unpredictable. They lead to out-of-control dinosaur parks and out-of-work meteorologists. …</p>
<p>Classical … chaotic systems are not in any sense intrinsically random or unpredictable. They merely possess <strong>extreme sensitivity to ignorance</strong>. Any initial uncertainty in our knowledge of a chaotic system&#8217;s state is rapidly amplified in time.</p>
<p>… although they become unpredictable when you try to determine the future from a particular uncertain starting value, there may be a particular stable statistical spread of outcomes after a long time, regardless of how you started out.</p></blockquote>
<p>Emphasis added.</p>
<p><strong>Related post</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2012/04/19/random-is-as-random-does/">Random is as random does</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/StNJ6gFybIY" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/24/chaos-vs-randomness/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/24/chaos-vs-randomness/</feedburner:origLink></item>
		<item>
		<title>100x better approach to software?</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/luHL0mimlaI/</link>
		<comments>http://www.johndcook.com/blog/2012/04/23/100x-better-approach-to-software/#comments</comments>
		<pubDate>Mon, 23 Apr 2012 13:20:56 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Software development]]></category>
		<category><![CDATA[Programming]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11202</guid>
		<description><![CDATA[Alan Kay speculates in this talk that 99% or even 99.9% of the effort that goes into creating a large software system is not productive. Even if the ratio of overhead and redundancy to productive code is not as high as 99 to 1, it must be pretty high.
Note that we&#8217;re not talking here about [...]]]></description>
			<content:encoded><![CDATA[<p>Alan Kay speculates in <a href="http://tele-task.de/archive/video/flash/14029/">this talk</a> that 99% or even 99.9% of the effort that goes into creating a large software system is not productive. Even if the ratio of overhead and redundancy to productive code is not as high as 99 to 1, it must be pretty high.</p>
<p>Note that we&#8217;re not talking here about individual programmer  productivity. Discussions of 10x or 100x programmers usually assume that  these are folks who can use basically the same approach and same tools to get much more work done. Here we&#8217;re thinking about better approaches more than better workers.</p>
<p>Say you have a typical stem of 10,000,000 lines of code. How many lines of code would a system with the same desired features (not necessarily all the <em>actual</em> features) require?</p>
<ol>
<li>If the same team of developers had worked from the beginning with a better design?</li>
<li>If the same team of developers could rewrite the system from scratch using what they learned from the original system?</li>
<li>If a thoughtful group of developers could rewrite the system without time pressure?</li>
<li>If a superhuman intelligence could rewrite the system, something approaching <a href="http://en.wikipedia.org/wiki/Kolmogorov_complexity">Kolmogorov complexity</a>?</li>
</ol>
<p>Where does the wasted effort in large systems go, and how much could be eliminated? Part of the effort goes into discovering requirements. Large systems never start with a complete and immutable specification. That&#8217;s addressed in the first two questions.</p>
<p>I believe Alan Kay is interested in the third question: How much effort is wasted by brute force mediocrity? My impression from watching his talk is that he believes this is the biggest issue, not evolving requirements. He said <a href="http://queue.acm.org/detail.cfm?id=1039523">elsewhere</a></p>
<blockquote><p>Most software today is very much like an Egyptian pyramid with millions  of bricks piled on top of each other, with no structural integrity, but  just done by brute force and thousands of slaves.</p></blockquote>
<p>There&#8217;s a rule that says bureaucracies follow a 3/2 law: to double productivity, you need three times as many employees. If a single developer could produce 10,000 lines of code in some period of time, you&#8217;d need to double that 10 times to get to 10,000,000 lines of code. That would require tripling your workforce 10 times, resulting in over 57,000 programmers. That sounds reasonable, maybe even a little optimistic.</p>
<p>Is that just the way things must be, that geniuses are in short supply and ordinary intelligence doesn&#8217;t scale efficiently? How much better could we do with available talent if took a better approach to developing software?</p>
<p><strong>Related post</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2009/03/18/where-does-the-programming-effort-go/">Where does programming effort go?</a><br />
<a href="http://www.johndcook.com/blog/2012/04/17/no-silver-bullet/">No Silver Bullet</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/luHL0mimlaI" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/23/100x-better-approach-to-software/feed/</wfw:commentRss>
		<slash:comments>26</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/23/100x-better-approach-to-software/</feedburner:origLink></item>
		<item>
		<title>Random is as random does</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/_aqjWl3wJKY/</link>
		<comments>http://www.johndcook.com/blog/2012/04/19/random-is-as-random-does/#comments</comments>
		<pubDate>Thu, 19 Apr 2012 14:10:45 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Statistics]]></category>
		<category><![CDATA[Probability and Statistics]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11182</guid>
		<description><![CDATA[What is randomness? Nobody knows, or at least there&#8217;s no consensus. Everybody has some vague ideas what randomness is, but when you dig into it deeply enough you find all kinds of philosophical quandaries. If you&#8217;d like a taste of the subtleties, you could start by reading one of Gregory Chaitin&#8217;s books. Or chew on [...]]]></description>
			<content:encoded><![CDATA[<p>What is randomness? Nobody knows, or at least there&#8217;s no consensus. Everybody has some vague ideas what randomness is, but when you dig into it deeply enough you find all kinds of philosophical quandaries. If you&#8217;d like a taste of the subtleties, you could start by reading <a href="http://www.amazon.com/gp/product/1852334177/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=1852334177">one of Gregory Chaitin&#8217;s books</a>. Or chew on <a href="http://www.amazon.com/gp/product/0387955674/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0387955674">this tome</a>.</p>
<p>What is a random variable? That&#8217;s easy. It&#8217;s a measurable function on a probability space. What&#8217;s a probability space? Easy too. It&#8217;s a measure space such that the measure of the entire space is 1.</p>
<p>Probability theory avoids defining randomness by working with abstractions like random variables. This is actually a very sensible approach and not mere legerdemain. Mathematicians can prove theorems about probability and leave the interpretation of the results to others.</p>
<p>As far as applications are concerned, it often doesn&#8217;t matter whether something is random in some metaphysical sense. The right question isn&#8217;t &#8220;is this system random?&#8221; but rather &#8220;is it useful to model this system as random?&#8221; Many systems that no one believes are random can still be profitably modeled as if they were random.</p>
<p>Probability models are just another class of mathematical models. Modeling deterministic systems using random variables should be no more shocking than, for example, modeling discrete things as continuous. For example, cars come in discrete units, and they certainly are not fluids. But sometimes it&#8217;s useful to model the flow of traffic as if it were a fluid. (And sometimes it&#8217;s not.)</p>
<p>Random phenomena are studied using computer simulations. And these simulations rely on random number generators, <em>deterministic</em> programs whose output is considered random for practical purposes. This bothers some people who would prefer a &#8220;true&#8221; source of randomness. Such concerns are usually misplaced. In most cases, replacing a random number generator with some physical source of randomness would not make a detectable difference. The output of the random number generator might even be higher quality since the measurement of the physical source could introduce a bias.</p>
<p><strong>Related posts</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2010/12/06/how-to-test-a-random-number-generator-2/">How to test a random number generator</a><br />
<a href="http://www.johndcook.com/blog/2008/07/30/connecting-probability-and-number-theory/">Connecting probability and number theory</a><br />
<a href="http://www.johndcook.com/blog/2011/09/30/just-an-approximation/">Just an approximation</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/_aqjWl3wJKY" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/19/random-is-as-random-does/feed/</wfw:commentRss>
		<slash:comments>43</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/19/random-is-as-random-does/</feedburner:origLink></item>
		<item>
		<title>Just what do you mean by “number”?</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/O2OnH3fyWWQ/</link>
		<comments>http://www.johndcook.com/blog/2012/04/18/just-what-do-you-mean-by-number/#comments</comments>
		<pubDate>Wed, 18 Apr 2012 21:10:25 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[Regular expressions]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11178</guid>
		<description><![CDATA[Tom Christiansen gave an awesome answer to the question of how to match a number with a regular expression. He begins by clarifying what the reader means by &#8220;number&#8221;, then gives answers for each.

Is −0 a number?
How do you feel about √−1?
Is ⅝ or ⅔ a number?
Is 186,282.42±0.02 miles/second one number — or is it [...]]]></description>
			<content:encoded><![CDATA[<p>Tom Christiansen gave an awesome answer to the question of how to match a number with a regular expression. He begins by clarifying what the reader means by &#8220;number&#8221;, then gives answers for each.</p>
<ul>
<li>Is <code>−0</code> a number?</li>
<li>How do you feel about <code>√−1</code>?</li>
<li>Is <code>⅝</code> or <code>⅔</code> a number?</li>
<li>Is <code>186,282.42±0.02</code> miles/second one number — or is it two or three of them?</li>
<li>Is <code>6.02e23</code> a number?</li>
<li>Is <code>3.141_592_653_589</code> a number? How about <code>π</code>, or <code>ℯ</code>? And <code>−2π⁻³ ͥ</code>?</li>
<li>How many numbers in <code>0.083̄</code>?</li>
<li>How many numbers in <code>128.0.0.1</code>?</li>
<li>What number does <code>⚄</code> hold? How about <code>⚂⚃</code>?</li>
<li>Does <code>10,5 mm</code> have one number in it — or does it have two?</li>
<li>Is <code>∛8³</code> a number — or is it three of them?</li>
<li>What number does <code>ↀↀⅮⅭⅭⅬⅫ AUC</code> represent, 2762 or 2009?</li>
<li>Are <code>४५६७</code> and <code>৭৮৯৮</code> numbers?</li>
<li>What about <code>0377</code>, <code>0xDEADBEEF</code>, and <code>0b111101101</code>?</li>
<li>Is <code>Inf</code> a number?  Is <code>NaN</code>?</li>
<li>Is <code>④②</code> a number? What about <code>⓰</code>?</li>
<li>How do you feel about <code>㊅</code>?</li>
<li>What do <code>ℵ₀</code> and <code>ℵ₁</code> have to do with numbers? Or <code>ℝ</code>, <code>ℚ</code>, and <code>ℂ</code>?</li>
</ul>
<p>See his full response <a href="http://stackoverflow.com/questions/4246077/simple-problem-with-regular-expression-only-digits-and-commas/4247184#4247184">here</a>. Thanks to <a href="https://twitter.com/#!/lizardbill">Bill the Lizard</a> for pointing this out.</p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/O2OnH3fyWWQ" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/18/just-what-do-you-mean-by-number/feed/</wfw:commentRss>
		<slash:comments>4</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/18/just-what-do-you-mean-by-number/</feedburner:origLink></item>
		<item>
		<title>Eat, drink, and be merry</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/PKAgwZhE1l8/</link>
		<comments>http://www.johndcook.com/blog/2012/04/18/eat-drink-and-be-merry/#comments</comments>
		<pubDate>Wed, 18 Apr 2012 13:42:34 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Science]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11161</guid>
		<description><![CDATA[Almost every bit of health advice I&#8217;ve heard has been contradicted. Should you eat more carbs or fewer carbs? More fat or less fat? Take vitamin supplements or not? It reminds me of this clip from Sleeper in which Woody Allen wakes up after 200 years of suspended animation.

Offhand I can only think of a [...]]]></description>
			<content:encoded><![CDATA[<p>Almost every bit of health advice I&#8217;ve heard has been contradicted. Should you eat more carbs or fewer carbs? More fat or less fat? Take vitamin supplements or not? It reminds me of this clip from Sleeper in which Woody Allen wakes up after 200 years of suspended animation.</p>
<p><iframe width="420" height="315" src="http://www.youtube.com/embed/D2fYguIX17Q?rel=0" frameborder="0" allowfullscreen></iframe></p>
<p>Offhand I can only think of a couple things on which there seems to be near unanimous agreement: smoking is bad for you, and moderate exercise is good for you.</p>
<p>Here are a couple suggestions for evaluating health studies.</p>
<p>Be suspicious of linear extrapolation. It does not follow that because moderate exercise is good for you, extreme exercise is extremely good for you. Nor does it follow that because extreme alcohol consumption is harmful, moderate alcohol consumption is moderately harmful. </p>
<p>Start from a default assumption that something natural or traditional is probably OK. This should not be dogmatic, only a starting point. In statistical terms, it&#8217;s a prior distribution informed by historical experience. The more a claim is at odds with nature and tradition, the more evidence it requires. If someone says fresh fruit is bad for you, for example, they need to present more evidence than someone who says an newly synthesized chemical compound is harmful. Extraordinary claims require extraordinary evidence.</p>
<p><strong>Related post</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2010/10/21/paleolithic-nonsense/">Paleolithic nonsense</a><br />
<a href="http://www.johndcook.com/blog/2008/10/08/whats-good-for-you-in-red-wine/">What&#8217;s good for you in red wine</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/PKAgwZhE1l8" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/18/eat-drink-and-be-merry/feed/</wfw:commentRss>
		<slash:comments>10</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/18/eat-drink-and-be-merry/</feedburner:origLink></item>
		<item>
		<title>Read history and fly an airplane</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/uISC7UahpAc/</link>
		<comments>http://www.johndcook.com/blog/2012/04/17/read-history-and-fly-an-airplane/#comments</comments>
		<pubDate>Wed, 18 Apr 2012 02:49:05 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[Books]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11169</guid>
		<description><![CDATA[The &#8220;About the Author&#8221; page at the end of Programming in Emacs Lisp says
Robert J. Chassell … has an abiding interest in social and economic history and flies his own airplane.
I love the child-like element of that bio. I could just imagine a kid saying &#8220;When I grow up, I want to read about history [...]]]></description>
			<content:encoded><![CDATA[<p>The &#8220;About the Author&#8221; page at the end of <a href="http://www.gnu.org/software/emacs/emacs-lisp-intro/">Programming in Emacs Lisp</a> says</p>
<blockquote><p>Robert J. Chassell … has an abiding interest in social and economic history and flies his own airplane.</p></blockquote>
<p>I love the child-like element of that bio. I could just imagine a kid saying &#8220;When I grow up, I want to read about history and fly my own airplane!&#8221; The bio is more about what the author enjoys than about how he makes his money. Maybe more bios should be like that.</p>
<p>The bio starts out by saying that Chassell speaks about Emacs and software freedom. I thought that was just to establish his <em>bona fides</em> for writing about Emacs Lisp, but his <a href="http://en.wikipedia.org/wiki/Robert_J._Chassell">Wikipedia page</a> says he&#8217;s a full-time speaker, so perhaps this <em>is</em> how he supports himself. I would not have thought that was possible, but good for him. Apparently he earns his living by talking about something he values.</p>
<p><strong>Update</strong>: As suggested in the comments, perhaps Chassell&#8217;s livelihood does not come from his speaking. Maybe he has (or had) another career and chose not to include it in his bio. Or maybe he doesn&#8217;t need to earn a living from his speaking. In any case, it sounds like he&#8217;s doing something he loves and his bio focuses on that.</p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/uISC7UahpAc" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/17/read-history-and-fly-an-airplane/feed/</wfw:commentRss>
		<slash:comments>5</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/17/read-history-and-fly-an-airplane/</feedburner:origLink></item>
		<item>
		<title>No Silver Bullet</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/iCr3Se5pcqQ/</link>
		<comments>http://www.johndcook.com/blog/2012/04/17/no-silver-bullet/#comments</comments>
		<pubDate>Tue, 17 Apr 2012 19:25:03 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Software development]]></category>
		<category><![CDATA[Programming]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11167</guid>
		<description><![CDATA[Mike Swaim gave a presentation today entitled No Silver Bullet, an allusion to Fred Brook&#8217;s classic essay by the same title. You can download the slides here.
Mike discusses the pros and cons of the following software development techniques:

High level languages
Object oriented programming
Declarative languages
Functional programming
Data oriented design
Metaprogramming
Static typing
Duck typing
Garbage collection
Allocating on the stack
Tail calls
Resource acquisition is [...]]]></description>
			<content:encoded><![CDATA[<p>Mike Swaim gave a presentation today entitled No Silver Bullet, an allusion to Fred Brook&#8217;s classic <a href="http://www.cs.nott.ac.uk/~cah/G51ISS/Documents/NoSilverBullet.html">essay</a> by the same title. You can download the slides <a href="http://www.johndcook.com/SilverBullets.pdf">here</a>.</p>
<p>Mike discusses the pros and cons of the following software development techniques:</p>
<ul>
<li>High level languages</li>
<li>Object oriented programming</li>
<li>Declarative languages</li>
<li>Functional programming</li>
<li>Data oriented design</li>
<li>Metaprogramming</li>
<li>Static typing</li>
<li>Duck typing</li>
<li>Garbage collection</li>
<li>Allocating on the stack</li>
<li>Tail calls</li>
<li>Resource acquisition is initialization</li>
<li>Symmetric multitasking</li>
<li>Heterogeneous multitasking</li>
</ul>
<p>There&#8217;s a nice collection of links at the end of each section.</p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/iCr3Se5pcqQ" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/17/no-silver-bullet/feed/</wfw:commentRss>
		<slash:comments>10</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/17/no-silver-bullet/</feedburner:origLink></item>
		<item>
		<title>Superheroes of the Round Table</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/ciwH5G9TMZs/</link>
		<comments>http://www.johndcook.com/blog/2012/04/17/superheroes-of-the-round-table/#comments</comments>
		<pubDate>Tue, 17 Apr 2012 12:00:25 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[Books]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11129</guid>
		<description><![CDATA[The other day I was browsing the Rice library and ran across a little book called &#8220;Superheroes of the Round Table: Comics Connections to Medieval and Renaissance Literature.&#8221; It&#8217;s about how literature has influenced comic books, and how comic books shed light on literature.

I don&#8217;t know much about comic books, or about medieval and renaissance [...]]]></description>
			<content:encoded><![CDATA[<p>The other day I was browsing the Rice library and ran across a little book called &#8220;Superheroes of the Round Table: Comics Connections to Medieval and Renaissance Literature.&#8221; It&#8217;s about how literature has influenced comic books, and how comic books shed light on literature.</p>
<p style="float: right; margin-left: 10px; margin-bottom: 10px"><a href="http://www.amazon.com/gp/product/0786460687/ref=as_li_ss_il?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0786460687"><img src="http://ws.assoc-amazon.com/widgets/q?_encoding=UTF8&amp;Format=_SL160_&amp;ASIN=0786460687&amp;MarketPlace=US&amp;ID=AsinImage&amp;WS=1&amp;tag=theende-20&amp;ServiceVersion=20070822" border="0" alt="" /></a><img style="border:none !important; margin:0px !important;" src="http://www.assoc-amazon.com/e/ir?t=theende-20&amp;l=as2&amp;o=1&amp;a=0786460687" border="0" alt="" width="1" height="1" /></p>
<p>I don&#8217;t know much about comic books, or about medieval and renaissance literature, but it&#8217;s fun to see someone draw them together, especially since the former is considered low culture and the latter high culture. It reminds me of Scott McCloud&#8217;s book <a href="http://www.amazon.com/gp/product/006097625X/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=006097625X">Understanding Comics: The Invisible Art</a>, a serious book about an art form that isn&#8217;t often taken seriously.</p>
<p>I&#8217;ve only skimmed <a href="http://ws.assoc-amazon.com/widgets/q?_encoding=UTF8&amp;Format=_SL160_&amp;ASIN=0786460687&amp;MarketPlace=US&amp;ID=AsinImage&amp;WS=1&amp;tag=theende-20&amp;ServiceVersion=20070822">Superheroes of the Round Table</a>, but it looks like a fun book. It draws connections, for example, between <a href="http://www.amazon.com/gp/product/1466339780/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=1466339780">The Faerie Queene</a> and <a href="http://www.amazon.com/gp/product/0785134646/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0785134646">Iron Man</a> and between <a href="http://www.amazon.com/gp/product/1461035937/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=1461035937">The Tempest</a> and <a href="http://www.amazon.com/gp/product/0785136983/ref=as_li_ss_tl?ie=UTF8&amp;tag=theende-20&amp;linkCode=as2&amp;camp=1789&amp;creative=390957&amp;creativeASIN=0785136983">X-Men</a>.</p>
<p>Just to give a flavor of the book&#8217;s analytical style, here is a classification the book gives for Arthurian legend in comic books.</p>
<ol>
<li><em>Traditional Tale</em>. Arthur in comic book form with minimal superhero elements.</li>
<li><em>Arthurian Toybox</em>. Elements of Arthur sprinkled into other stories with no regard for literary context.</li>
<li><em>Arthur as Translator</em>. A modern superhero is dropped into Arthur&#8217;s Britain, like Mark Twain&#8217;s Connecticut Yankee.</li>
<li><em>Arthur as Collaborator</em>. Using Arthurian symbols and themes such as the sword in the stone or the round table.</li>
<li><em>Arthur Transformed</em>. Arthur placed into a new context.</li>
</ol>
<p><strong>Related post</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2011/06/15/manga-guides-to-physics-and-the-universe/">Manga guides to science</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/ciwH5G9TMZs" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/17/superheroes-of-the-round-table/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/17/superheroes-of-the-round-table/</feedburner:origLink></item>
		<item>
		<title>Random number sequence overlap</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/pcEpGBfztYg/</link>
		<comments>http://www.johndcook.com/blog/2012/04/16/random-number-sequence-overlap/#comments</comments>
		<pubDate>Mon, 16 Apr 2012 18:17:56 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Computing]]></category>
		<category><![CDATA[Probability and Statistics]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11147</guid>
		<description><![CDATA[Mike Croucher asked the following question on his blog. Suppose you draw M sequences of random numbers of length N from a random number generator. What is the probability that they will overlap?
Assumes your random number generator is a cyclical list of p unique integers. Each draw picks a random starting point in the cycle [...]]]></description>
			<content:encoded><![CDATA[<p>Mike Croucher asked the following <a href="http://www.walkingrandomly.com/?p=3555">question</a> on his blog. Suppose you draw <em>M</em> sequences of random numbers of length <em>N</em> from a random number generator. What is the probability that they will overlap?</p>
<p>Assumes your random number generator is a cyclical list of <em>p</em> unique integers. Each draw picks a random starting point in the cycle and chooses <em>N</em> consecutive values. We are particularly interested in the case <em>M</em> = 10<sup>4</sup>, <em>N</em> = 10<sup>9</sup>, and <em>p</em> = 2<sup>19937</sup> &#8211; 1 (the period of the Mersenne Twister).</p>
<p>We will simplify the problem by over-estimating the probability of overlap. The over-estimated probability will still be extremely small.</p>
<p>Assume all the draws are independent. This greatly increases the probability of repetition because now there is some chance that there are repetitions within each draw. That wasn&#8217;t possible before, provided <em>N</em> &lt; <em>p</em>.</p>
<p>The probability of no repetitions in <em>n</em> = <em>MN</em> draws is</p>
<p style="text-align:center"><img src="http://www.johndcook.com/bday1.png" alt="\prod_{i=0}^{n-1} \frac{p - i}{p} = \frac{p!}{p^{n}(p - n)!}" width="160" height="50" /></p>
<p>We can under-estimate this probability by replacing every term in the product with the smallest term. (By <em>under</em>-estimating the probability of no repetitions, we&#8217;re <em>over</em>-estimating the probability of at least one repetition.)</p>
<p style="text-align:center"><img src="http://www.johndcook.com/bday2.png" alt="\prod_{i=0}^{n-1} \frac{p - i}{p} &gt; \left(\frac{p - n + 1}{n}\right)^n = \left(1 - \frac{n-1}{p}\right)^n \approx 1 - \frac{n(n-1)}{p}" width="426" height="50" /></p>
<p>When <em>n</em> = <em>MN</em> = 10<sup>13</sup> and <em>p</em> = 2<sup>19937</sup> &#8211; 1, the last expression above is approximately 1 &#8211; 10<sup>26</sup>/2<sup>19937</sup>. That says the (over-estimated) probability of repetition is 10<sup>26</sup>/2<sup>19937</sup> or about 2×10<sup>-5976</sup>, thousands of orders of magnitude smaller than the chances of, say, drawing a royal flush.</p>
<p>Whenever you have such mind-bogglingly small probabilities, some probability outside your model has to be more important than the probability computed with the model. For example, we could question the assumption that the starting points are random and independent. Or we could question whether the random number generator was written correctly, or that the compiler correctly compiled the program, or that the operating system correctly ran the program, etc.</p>
<p><strong>Related posts</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2011/09/09/five-interesting-things-about-mersenne-primes/">Five interesting things about Mersenne primes</a><br />
<a href="http://www.johndcook.com/blog/2010/12/06/how-to-test-a-random-number-generator-2/">How to test a random number generator</a><br />
<a href="http://www.johndcook.com/blog/2011/02/07/twin-prime-conjecture-pentium-bug/">Twin prime conjecture and the Pentium division bug</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/pcEpGBfztYg" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/16/random-number-sequence-overlap/feed/</wfw:commentRss>
		<slash:comments>6</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/16/random-number-sequence-overlap/</feedburner:origLink></item>
		<item>
		<title>Dart programming language design</title>
		<link>http://feedproxy.google.com/~r/TheEndeavour/~3/Y6RYHktjj_g/</link>
		<comments>http://www.johndcook.com/blog/2012/04/14/dart-programming-language-design/#comments</comments>
		<pubDate>Sat, 14 Apr 2012 19:41:09 +0000</pubDate>
		<dc:creator>John</dc:creator>
				<category><![CDATA[Software development]]></category>
		<category><![CDATA[Programming]]></category>

		<guid isPermaLink="false">http://www.johndcook.com/blog/?p=11133</guid>
		<description><![CDATA[Gilad Bracha&#8217;s presentation on Dart is interesting even if, like me, you do not intend to learn Dart. He has some good one-liners, especially digs at JavaScript, and some interesting ideas about programming language design. (JavaScript comes up frequently in presentation because of its popularity and because Dart compiles to JavaScript.)
Application programmers, unless they&#8217;ve committed [...]]]></description>
			<content:encoded><![CDATA[<p>Gilad Bracha&#8217;s <a href="http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/Dart-A-Well-Structured-Web-Programming-Language">presentation on Dart</a> is interesting even if, like me, you do not intend to learn Dart. He has some good one-liners, especially digs at JavaScript, and some interesting ideas about programming language design. (JavaScript comes up frequently in presentation because of its popularity and because Dart compiles to JavaScript.)</p>
<blockquote><p>Application programmers, unless they&#8217;ve committed serious crimes, should not be forced to deal with [JavaScript etc.].</p></blockquote>
<blockquote><p>We&#8217;re not out to take JavaScript away. We&#8217;re not that stupid. I personally if I could I would, but Google probably wouldn&#8217;t.</p></blockquote>
<blockquote><p>There are millions of people writing JavaScript who do not understand JavaScript&#8217;s type model at all.</p></blockquote>
<p>Some of the programming language ideas in the talk:</p>
<ul>
<li>optional type systems</li>
<li>separating compile time type-checking from execution</li>
<li>delegating type inference to tools rather than the language</li>
<li>mirror-based reflection</li>
<li>which programmer mistakes should cause compiler errors and which should cause warnings</li>
</ul>
<p>Gilad Bracha argues that type systems are important, but not for the reason most people give. Type systems are a form of documentation, and they help enable programming tool support, but their role in error detection is greatly overstated. Anders Hejlsberg, designer of C#, made a similar statement at the same conference.</p>
<p><iframe style="height:227px;width:400px" src="http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/Dart-A-Well-Structured-Web-Programming-Language/player?w=400&#038;h=227" frameBorder="0" scrolling="no" ></iframe></p>
<p>See also Gilad Bracha&#8217;s comments during this <a href="http://channel9.msdn.com/Events/Lang-NEXT/Lang-NEXT-2012/Panel-Web-and-Cloud-Programming">panel discussion</a>.</p>
<p><strong>Related post</strong>:</p>
<p><a href="http://www.johndcook.com/blog/2010/06/09/dynamic-typing-and-risk-homeostasis/">Dynamic typing and anti-lock brakes</a></p>
<img src="http://feeds.feedburner.com/~r/TheEndeavour/~4/Y6RYHktjj_g" height="1" width="1"/>]]></content:encoded>
			<wfw:commentRss>http://www.johndcook.com/blog/2012/04/14/dart-programming-language-design/feed/</wfw:commentRss>
		<slash:comments>7</slash:comments>
		<feedburner:origLink>http://www.johndcook.com/blog/2012/04/14/dart-programming-language-design/</feedburner:origLink></item>
	</channel>
</rss><!-- Dynamic Page Served (once) in 0.704 seconds -->

