From an NYT article on Google’s translation services, this excerpt sums up the most critical transition in machine learning that has happened thus far:

Creating a translation machine has long been seen as one of the toughest challenges in artificial intelligence. For decades, computer scientists tried using a rules-based approach — teaching the computer the linguistic rules of two languages and giving it the necessary dictionaries.

But in the mid-1990s, researchers began favoring a so-called statistical approach. They found that if they fed the computer thousands or millions of passages and their human-generated translations, it could learn to make accurate guesses about how to translate new texts.

• Statistics: desired and feared?
• The million dollar question
• Urban mathematics
• Test driving America’s dashboard

Fascinating… far from being a psychedelic tour of the imagination, one graduate student argues that Alice in Wonderland is actually a satire of Victorian mathematics:

Yet Dodgson [Lewis Carroll] most likely had real models for the strange happenings in Wonderland, too. He was a tutor in mathematics at Christ Church, Oxford, and Alice’s search for a beautiful garden can be neatly interpreted as a mishmash of satire directed at the advances taking place in Dodgson’s field.

In the mid-19th century, mathematics was rapidly blossoming into what it is today: a finely honed language for describing the conceptual relations between things. Dodgson found the radical new math illogical and lacking in intellectual rigor. In “Alice,” he attacked some of the new ideas as nonsense — using a technique familiar from Euclid’s proofs, reductio ad absurdum, where the validity of an idea is tested by taking its premises to their logical extreme.

• On teaching math
• The mathematician’s lens
• Urban mathematics
• Enlightenment

The Sortino ratio has emerged as a popular risk measure when evaluating investments. It is a modifcation of the Sharpe ratio, a workhorse indicator of mean/variance economics.

The Sharpe ratio is constructed like this:

where is the expected return, is a benchmark hurdle, and is the standard deviation of the returns. If you buy into a Gaussian mean/variance paradigm, then the Sharpe ratio tells you how many units of excess return you receive per unit of risk you take.

The Sortino ratio is constructed similarly:

Here, is the downside deviation, or the standard deviation of returns below the benchmark. The intuition of using this statistic is that people do not penalize investments for positive volatility (i.e. unpredictable but beneficial returns); they only care about negative volatility.

And here lies the rub: it’s very easy to calculate a misleading Sortino ratio. The popular method – you’ll see it floating around the web – is to take any positive (or above-benchmark) return, change it to a zero, and calculate a standard deviation as one normally would, across all returns.

To me, that’s not right. You are artificially introducing a steady stream of zeros into your calculation, depressing the volatility calculation. A more proper way is to throw out any positive returns, and calculate the standard deviation of the negative returns (it should not be surprising that this method complies with the intuition for using the Sortino in the first place).

So the next time you’re presented with a Sortino ratio, take care to understand whether it includes zeros or not – if it does, the denominator is necessarily biased toward zero, and the ratio is overstated.

• Misreading misleading charts: VIX edition
• How to lose your money without really trying
• Charting value (maybe)
• Touché.

Apologies for the slow posts… but the NYT explains:

Wall Street trading is often described as a blood sport. But inside the great investment houses, the sport of the moment is, of all things, curling — that oddball of the Olympics that is sort of like shuffleboard on ice.

This slow-poke game, which originated in 16th-century Scotland, has captivated the Type-A world of Wall Street almost by accident. CNBC, whose market chatter is the background music on trading floors, switches to curling from Vancouver shortly after the closing bell.

I thought I was the only one going curling-crazy, but it turns out all of Wall Street has spent the last couple weeks learning a new vocabulary (just call me “Skip”) and shouting at the TV. Whether or not everyone else has been honing their skills by playing shuffleboard, I don’t know… but my plan to open an NYC curling house/alley/place (?) just got a major boost.

• About those things I said before…
• Tragedy in AAPLville
• Keeping heads out of textbooks
• Viva la banker

Via the AP:

“As we know, the global financial crisis originated neither in Russia, nor in Greece or Europe — it came from across the ocean,” Putin said.

“In the United States, we see the same problems — massive foreign debt and budget deficit,” he said.

• Measuring fiscal boosts
• Fact-checking the singularity
• SIGTARP confirms: TARP an exercise in moral hazard
• Why the administration may be missing its chance

Kevin Fox is on board the Dashboard train that I wrote about a short while ago. Having seen the sparsity of the iPad screen, and how strange iPhone-scale apps look when zoomed, I’m liking the idea more and more. Plus it would enable some form of multitasking…

(Via John Gruber)

• Tablet OS = Dashboard?
• CNN < supermarket tabloid
• Signs of the times
• MouthOff

LOST is back tonight! And what better way to prepare than an interactive timeline from the excellent NYT graphics team? A good infographic should communicate otherwise-complex ideas in a simple and intuitive manner… oh, never mind, LOST is back and that’s really what matters. Check out the timeline here!

• It’s American as sweet potatoes (but not sweet potato pie)
• The 100 users of Twitter
• Untangling charts
• Is TV getting LOST?

The Special Inspector General’s report on TARP has been released from embargo. It concludes that TARP was unsuccessful, and even its (debatable) short-term corrections are overshadowed by the extent to which it has returned the economy to its previous bubble state — “we are still driving on the same winding mountain road, but this time in a faster car.”

From the executive summary:

The substantial costs of TARP — in money, moral hazard effects on the market, and Government credibility — will have been for naught if we do nothing to correct the fundamental problems in our financial sys- tem and end up in a similar or even greater crisis in two, or five, or ten years’ time. It is hard to see how any of the fundamental problems in the system have been addressed to date.

• To the extent that huge, interconnected, “too big to fail” institutions contributed to the crisis, those institutions are now even larger, in part because of the sub- stantial subsidies provided by TARP and other bailout programs.
• To the extent that institutions were previously incentivized to take reckless risks through a “heads, I win; tails, the Government will bail me out” mentality, the market is more convinced than ever that the Government will step in as neces- sary to save systemically significant institutions. This perception was reinforced when TARP was extended until October 3, 2010, thus permitting Treasury to maintain a war chest of potential rescue funding at the same time that banks that have shown questionable ability to return to profitability (and in some cases are posting multi-billion-dollar losses) are exiting TARP programs.
• To the extent that large institutions’ risky behavior resulted from the desire to justify ever-greater bonuses — and indeed, the race appears to be on for TARP recipients to exit the program in order to avoid its pay restrictions — the current bonus season demonstrates that although there have been some improvements in the form that bonus compensation takes for some executives, there has been little fundamental change in the excessive compensation culture on Wall Street.
• To the extent that the crisis was fueled by a “bubble” in the housing market, the Federal Government’s concerted efforts to support home prices — as discussed more fully in Section 3 of this report — risk re-inflating that bubble in light of the Government’s effective takeover of the housing market through purchases and guarantees, either direct or implicit, of nearly all of the residential mortgage market.

• TIME: “It’s over!”
• Moral hazard and the NFL
• JPM, AXP ask taxpayers for money to pay back taxpayers
• The world according to Putin

Deep in southwestern Kansas, surrounded by miles and miles of absolutely nothing, is a giant stone-lined hole in the ground.

It’s not just any old hole, though — it’s the largest hand-dug well in the world. And according to the WSJ, it’s about to acquire a world-class museum:

The citizens of Greensburg are planning a $3 million Big Well museum and this month announced a contract with a high-profile design team, Ralph Appelbaum Associates Inc. of New York. The firm has designed exhibits at the American Museum of Natural History in New York, the U.S. Holocaust Memorial Museum in Washington, D.C., Bill Clinton’s presidential library in Little Rock, Ark., and the Country Music Hall of Fame in Nashville.

Greensburg has been hit by hard times recently. A 2007 tornado devastated the city’s downtown area and tourism has traditionally played a significant role in the local economy. Lately that support has been lacking:

In the 1970s and ’80s, as many as 75,000 visitors a year would stop by Greensburg to peer into the murky water. They’d drop a coin (or, oddly, a shoe) for good luck, maybe even buy a $2 ticket and descend 105 steps to the claustrophobic depths.

In recent years, however, drivers whizzing past on Highway 400 have been less prone to pull over, despite a series of promotional billboards stretched out over 50 miles to build excitement.

It’s easy to laugh at this seemingly ridiculous tourist attraction — it’s just a hole in the ground, after all. But I have to reserve judgement.

You see, I have been to the Big Well.

I’ve driven across Kansas. Twice. I’ve seen the billboards. And let me tell you, when you’re faced with nothing but bland prairie for hours, those ads look amazing: “Stop and see the giant well! Have a cold drink! Forget that you’re in Kansas, most boring state of all!” After all, your choices for stopping are the various McDonalds scattered along the highway median… or the GIANT WELL! And you probably stopped for a burger when you were only a third of the way across the state, which leaves only one option…

But here’s the kicker. Greensburg isn’t just the home of the world’s deepest human-dug hole; it’s also the home of the world’s largest pallasite meteorite! (No longer – now it’s just the world’s second largest pallasite meteorite.) One town, TWO record-setting objects which conjure immense vertical images, from the heights of space to the depths of the Earth. Moreover, under one roof (at least, until the tornado came through). And I ask you: after trekking endlessly toward Kansas’ uninterrupted horizon, how can you pass up this opportunity to transcend flatness?

For years, my brother and I have laughed privately at the odd-couple billboards we once saw (and obeyed): “See the world’s largest hand dug well… and meteorite!” Maybe now we can share that amusement with other cross-country drivers.

Oh, and just in case you still harbor some concern that the Big Well (or Kansas more generally) can’t compete in this plugged in, Disneyfied world, let them go:

In 2008, a popular vote online tabbed Greensburg’s Big Well as one of the Eight Wonders of Kansas, on par with the Underground Salt Museum in Hutchinson (and a cut above the town of West Mineral’s star attraction—”Big Brutus,” an enormous electric coal shovel).

• Dude, that’s a toad
• Math is hard!
• One way, the wrong way
• The Yom Kippur effect

This is huge. Google has enabled a Java scripting application across it’s online Apps suite. Previously available only to premier subscribers, the scripting capability is now open to anyone with a standard account as well. This competes directly with Microsoft’s cross-Office VBA, allowing users to build their own applications while using the productivity suite as a front end. It’s a very impressive and bold move to allow it over the web – and further solidifies Google’s “the web is the OS” mantra.

Read more at the official Google Enterprise blog.

• iPhones support push gmail!
• The sound of a million MobileMe subscribers screaming
• When are users right about design changes?
• Don’t Bing me

A beautiful article in the NYTimes contrasts abstract mathematics with the chilling reality of the Mexican drug cartel wars:

I was born in Mexico City, in a world that seems less and less familiar to me. I live now in the opposite corner of the continent. I am training to be a political scientist at Harvard. My passion has remained the afflictions of my homeland, but at Harvard I have found new ways to address them, to use mathematical models — matrices, vectors, equations, regressions — to understand the Mexican drug crisis.

The cartel wars are extremely violent, and the gangs are responsible for reprehensible kidnappings and deaths. They rank among the most deadly periods of organized crime in human history. The author’s goal isn’t to explain how she can analyze the wars from up in an ivory tower; it’s to describe how her mindset and toolkit inform her understanding of the world in any situation.

The article captured me because it never mentions what the author actually models. Instead, it presents her frightened thoughts and her efforts to calm herself by looking at the world through a mathematical lens. But it’s not what you think; there are no emotionally-distant mathematicians here. The author communicates her fascination with tying reality to abstract models, expecting and preempting the protest that reality is too complex and math too simple:

In this violent world, with the man in the blue Chevy whispering at me behind the window, math is my shield. Speaking up about drugs is in these parts a dangerous game. But not if you speak in the language of sigma and conditional expectations. Math protects me from the immediacy of the violence, and it protects me from them.

The beauty of my method lies in its simplicity. With mathematics I’m able to codify and simplify reality to make it manageable and, more important, malleable. I represent each possible individual as an equation in which each term symbolizes tastes, goals, profession and abilities. All people get portrayed: Policemen, politicians, citizens and drug cartels start living in this mathematical world as planes and hyperplanes and, as in real life, they interact and affect one another, sometimes colluding, sometimes colliding, sometimes neither.

I then use optimization to predict the form of interaction that will be the most probable to emerge and remain over time. Math starts speaking. It tells me, for example, under what conditions the outcome would be a drug war; when would the government prefer to cooperate with cartels; or when cruel intra-cartel purges will become the norm.

There is a part of every modeler’s mind which is constantly teasing out variables from constants. The statisticians among us may take a frequentist view, and wonder what would happen if a scene played itself out a million times; the programmers will deduce the underlying algorithms from the fuzzy result; the pure mathematicians will see manifolds everywhere:

In this abstract microcosmos, reality can be frozen or just slightly changed. I move and look at my hyperplanes from different angles. Let’s change the penalty code. No, let’s increase patrolling. Or reduce wages. Allow less contact between policemen and dealers. Assume the police force is corrupt. Assume it is not. I solve the equations and there it is. My answers come as Greek letters and probabilities.

But we all admit:

I know, I know, this is weird.

Ultimately, “free will” becomes the clarion of the independent. At least, it’s the best response to this explanation:

It may seem strange to examine this shadowy world with equations. But mathematics is transforming the social sciences. In the same way that physicists can predict the movement of atoms in space, we can use mathematics to model how individuals and groups will make decisions and interact in a society.

But free will has a (somewhat tentative) analogue in Heisenberg’s uncertainty principle, and with that philosophy and math (or theology and physics) are combined — but there’s been plenty of pop-sci written on that topic.

I found this brief article remarkable in how it was able to demonstrate the overlay mathematical thought on an extremely “human” subject without ever needing to explain either one.

(Via Drew Conway)

• On teaching math
• Signs of the apocalypse
• Don’t know much about calculus
• Dronish number nerds

Microsoft has announced the system requirements for Office 2010.

That’s news in and of itself. Once upon a time, system requirements (at least, ones that anyone paid attention to) were strictly for high-end professional software, cutting-edge games and the like: software that actually needed powerful hardware. But the real news here is that Office 2010 requires a DirectX-compatible graphics card.

Now, I don’t think Word is going to be offloading word counts to a GPU anytime soon. But Microsoft’s announcement is making waves nontheless — and I think it’s actually great. It means we’ve reached a point where our computing history is so mature that even our mass-market word processors have achieved a level of sophistication that we need to make sure of their compatibility. That’s exciting!

Certainly, Excel is an obvious candidate for hardware acceleration, which, besides accelerating simple tasks like opening large files and parallel tasks like running many equations, could finally bring true vector operations to the versatile software.

But there is bad news. I’ll let Microsoft break it to you:

If your computer has a GPU, it lets us perform graphics rendering tasks (like drawing charts in Excel, or transitions in PowerPoint) in the GPU instead of in the CPU, which parallelizes work and speeds up performance. This is particularly relevant for users of PowerPoint 2010, which will introduce some awesome new graphics and video integration features (more info at the PowerPoint team blog).

Yes, the true motivation behind the graphics upgrade is supercharging those awful 3D pie charts we know and despise.

(If you click the PowerPoint link, you’ll notice that Powerpoint 2010 looks a lot like Keynote. Just sayin’.)

• Overcharting
• Self-shadowing, briefly
• The Google Phone: this doesn’t change very much at all (yet)
• Web 3.0 is/isn’t coming

Very amusing… and true:

I especially love “The HDR Hole.” Presumably the y-axis is measured in percent of personal potential… there must be all sorts of Bayesian self-reflection stuff going on there.

(Via DataViz)

• The 100 users of Twitter
• More mainstream Bayesians
• Living in a Bayesian world
• Critiquing the Crimson

Via The Big Picture comes news of a rather substantial revision of numbers from last November:

You may recall that consensus for November’s Durable Goods had been +0.5%.  The reported data was lighter than expected at +0.2%. Looking at the revisions the Census Bureau has now incorporated into the data, we see that November actually printed at -0.7%.

The critical point is that this would represent two consecutive months of negative growth – a feat not accomplished since last January. For the record, the swing of more than 1% was officially attributed to a “processing error.”

• On revisions
• On revisions, episode VI
• On revisions, episode III
• On revisions, episode IV (A New Hope?)

(Via Piled Higher and Deeper)

• An unhelpful VIX
• Log differencing Top Chef
• Bell curves in action
• Photo finish: the Netflix prize