Bad Analysis

And Behind Door No. 1, a Fatal Flaw (New York Times)

2008-04-08T14:05:00.000-07:00

Really interesting article by John Tierney in the New York Times on cognitive dissonance.

"The Monty Hall Problem has struck again, and this time it’s not merely embarrassing mathematicians. If the calculations of a Yale economist are correct, there’s a sneaky logical fallacy in some of the most famous experiments in psychology."

Unintended Consequences (New York Times)

2008-01-27T07:08:00.000-08:00

Interesting article by Steven Levitt and Stephen Dubner about how well-meaning attempts to regulate complicated systems are likely to to have unintended consequences. Here is a short excerpt:

"Acemoglu and Angrist found that when the A.D.A. was enacted in 1992, it led to a sharp drop in the employment of disabled workers. How could this be? Employers, concerned that they wouldn’t be able to discipline or fire disabled workers who happened to be incompetent, apparently avoided hiring them in the first place.

How long have such do-good laws been backfiring? Consider the ancient Jewish laws concerning the sabbatical, or seventh year. As commanded in the Bible, all Jewish-owned lands in Israel were to lie fallow every seventh year, with the needy allowed to gather whatever food continued to grow. Even more significant, all loans were to be forgiven in the sabbatical. The appeal of such unilateral debt relief cannot be overestimated, since the penalties for defaulting on a loan at the time were severe: a creditor could go so far as to take a debtor or his children into bondage.

So for a poor Jewish sandal maker having trouble with his loan payments, the sabbatical law was truly a godsend. If you were a creditor, however, you saw things differently. Why should you lend the sandal maker money if he could just tear up the loan in Year Seven? Creditors duly gamed the system, making loans in the years right after a sabbatical, when they were confident they would be repaid, but then pulling tight the purse strings in Years Five and Six. The resulting credit drought was so damaging to poor people that it fell to the great sage Hillel to fix things."

Is a Carl Doomed To Be C Student?

2007-12-14T20:10:00.000-08:00

A study reported widely last month found that "students whose names start with the letters C or D, which denote mediocre marks in some grading systems, did not perform as well as other pupils with different initials."

As Carl Bialik points out in the Wall Street Journal, while the relationship that the researchers found was statistically significant, it doesn't mean it is important:

"University of California, Irvine, statistician Hal Stern points out something most media missed. The effect is tiny: 0.02 of a grade-point average point lower for the initials C and D (and this columnist isn't including that because of his first initial). Therein lies a lesson in the difference between statistical significance -- the confidence that there is some association between two factors -- and the strength of that association.

"In very large samples like the ones here, even small differences will be judged statistically significant," Prof. Stern says. "This means that we're confident the difference is not zero. It does not mean the difference we see is important." Prof. Nelson agrees that this effect is "so small that you shouldn't worry about it" when naming a child, though he does say the study exposes an example of how the unconscious mind can undermine conscious motivation.

But Bowling Green statistician Jim Albert warns: "You can prove any silly hypothesis ... by running a statistical test on tons of data.""

Citing Statistics, Giuliani Misses Time and Again

2007-11-30T18:28:00.000-08:00

New York Times article on Rudy Giuliani's record reporting the facts during his presidential campaign.

Four Pinocchios for Ron Paul (The Fact Checker)

2007-11-17T07:44:00.000-08:00

Good analysis of Ron Paul's claim that we could eliminate personal income taxes and still balance the budget.

Rudy Giuliani, amateur epidemiologist

2007-11-09T09:27:00.000-08:00

There have been many articles written about poor use of prostate cancer statistics by Rudy Giuliani. The main complaint is that he uses faulty logic to calculate prostate cancer survival rates. However, I think the bigger problem with Giuliani's numbers is that he cherry picks numbers to prove his points, providing no context. Once again, this appears to be a complicated issue and simplistic analysis by a politician.

Background
First, here's the background. According to a Giuliani radio ad:

"I had prostate cancer, five, six years ago. My chance of surviving prostate cancer, and thank God I was cured of it, in the United States, 82 percent. My chances of surviving prostate cancer in England, only 44 percent under socialized medicine."

As pointed out by factcheck.org and The Fact Checker at the Washington Post, the 44% figure was arrived at by simplistically dividing per capita prostate cancer mortality by per capita prostate cancer diagnoses (and subtracting that figure from 100%). Unfortunately, the people diagnosed in a given year are not the same people that die in that year, so you can't figure out what your odds of surviving prostate cancer by using this data. To determine survival rate, you need to follow the same population over a period of time (5-year survival rate is the standard metric). The 5-year survival rate in the U.K. is 74.4%.

The bigger problem
Clearly, it is troubling that a guy who wants to be part of the debate on the nation's health care doesn't have anyone on his staff that really understands the data. Equally troubling is that even after the doctors whose study he bases his claims pointed out his error, he continued to use the misleading numbers. But even if Giuliani used the proper numbers (which are 5-year survival rates of 74% in the U.K. and 98% in the U.S.), the conclusion he draws is simplistic at best.

Let me give you some statistics that would seem to refute Giuliani's conclusion about socialized medicine:

Average life expectancy in the U.K. is 79.4 years, compared to only 78.2 in the U.S.
Infant mortality rate in the U.K. is 5.01 per thousand, compared to 6.37 in the U.S.
Health care spending per capita in the U.K. is $1,675, compared to $4,271 in the U.S.

Does this mean that socialized medicine is better than private medicine? In my opinion, that would be an equally simplistic conclusion. We need to control for a lot of things in order for the data to be meaningful: population demographics, approach toward prevention, detection and treatment, etc. Unfortunately, intellectually honest analysis doesn't seem to be a Giuliani strong point.

Most Science Studies Appear to Be Tainted By Sloppy Analysis (Wall Street Journal)

2007-09-15T08:33:00.000-07:00

Interesting article by Robert Lee Holtz at the Wall Street Journal about mistakes in published research. He references the work of Dr. John Ioannidis, who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University. It also references an even more interesting paper he published in 2005.

From the article:

"Flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. 'There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims,' Dr. Ioannidis said. 'A new claim about a research finding is more likely to be false than true.'"

"The hotter the field of research the more likely its published findings should be viewed skeptically, he determined."

"Take the discovery that the risk of disease may vary between men and women, depending on their genes. Studies have prominently reported such sex differences for hypertension, schizophrenia and multiple sclerosis, as well as lung cancer and heart attacks. In research published last month in the Journal of the American Medical Association, Dr. Ioannidis and his colleagues analyzed 432 published research claims concerning gender and genes."

"'Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual,' Dr. Ioannidis said."

From the paper:
The paper lays out several factors that influence the probability of the results of a study being true:

Corollary 1: The smaller the studies conducted in a scientific field, the less likely the research findings are to be true ... other factors being equal, research findings are more likely true in scientific fields that undertake large studies, such as randomized controlled trials in cardiology (several thousand subjects randomized) than in scientific fields with small studies, such as most research of molecular predictors (sample sizes 100-fold smaller).

Corollary 2: The smaller the effect sizes in a scientific field, the less likely the research findings are to be true ... research findings are more likely true in scientific fields with large effects, such as the impact of smoking on cancer or cardiovascular disease (relative risks 3–20), than in scientific fields where postulated effects are small, such as genetic risk factors for multigenetic diseases (relative risks 1.1–1.5). Modern epidemiology is increasingly obliged to target smaller effect sizes. Consequently, the proportion of true research findings is expected to decrease.

Corollary 3: The greater the number and the lesser the selection of tested relationships in a scientific field, the less likely the research findings are to be true ... research findings are more likely true in confirmatory designs, such as large phase III randomized controlled trials, or meta-analyses thereof, than in hypothesis-generating experiments.

Corollary 4: The greater the flexibility in designs, definitions, outcomes, and analytical modes in a scientific field, the less likely the research findings are to be true ... Adherence to common standards is likely to increase the proportion of true findings. The same applies to outcomes. True findings may be more common when outcomes are unequivocal and universally agreed (e.g., death) rather than when multifarious outcomes are devised (e.g., scales for schizophrenia outcomes). Similarly, fields that use commonly agreed, stereotyped analytical methods (e.g., Kaplan-Meier plots and the log-rank test) may yield a larger proportion of true findings than fields where analytical methods are still under experimentation (e.g., artificial intelligence methods) and only “best” results are reported.

Corollary 5: The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true ... Prejudice may not necessarily have financial roots. Scientists in a given field may be prejudiced purely because of their belief in a scientific theory or commitment to their own findings. Many otherwise seemingly independent, university-based studies may be conducted for no other reason than to give physicians and researchers qualifications for promotion or tenure.

Corollary 6: The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true ... With many teams working on the same field and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritize on pursuing and disseminating its most impressive “positive” results. “Negative” results may become attractive for dissemination only if some other team has found a “positive” association on the same question. In that case, it may be attractive to refute a claim made in some prestigious journal. The term Proteus phenomenon has been coined to describe this phenomenon of rapidly alternating extreme research claims and extremely opposite refutations. Empirical evidence suggests that this sequence of extreme opposites is very common in molecular genetics.

The Media and the Median (Fallacy Files)

2007-08-26T16:01:00.000-07:00

Fallacy Files points out an amusing error in a recent New York Times science article, "The Myth, The Math, The Sex" by Gina Kolata. The article states:

"One survey, recently reported by the federal government, concluded that men had a median of seven female sex partners. Women had a median of four male sex partners. Another study, by British researchers, stated that men had 12.7 heterosexual partners in their lifetimes and women had 6.5.

But there is just one problem, mathematicians say. It is logically impossible for heterosexual men to have more partners on average than heterosexual women. Those survey results cannot be correct."

However, as Fallacy Files points out, even in a population of equal numbers of men and women over a given time period, the median number of sex partners could be different. All you need are a few very active women who raise the mean a lot but don't raise the median too much (Madonna-whore dichotomy?). From Fallacy files:

"It's discouraging that an article this innumerate would be published in the Times, and I suppose that the mathematician quoted was not the source of the confusion, but that it must have been introduced in the writing or editing. As it is, there's no evidence in the article of anything impossible in the statistics cited. A British survey is quoted, but the article doesn't indicate whether the numbers are medians, means, or what―which is a problem in itself!

Moreover, the article presents itself as busting the "myth" that heterosexual men are more promiscuous than heterosexual women. Surely, the "myth" is that the typical man has more sex partners than the typical woman. In order for this to be true, there must be some atypically promiscuous women. Whether the "myth" is true or not, I don't know―damn it, Jim, I'm a logician, not a sociologist!―but I do know that it is an empirical question and not a logical or mathematical one."

Issues are complicated. Politicians are not.

2007-06-24T13:45:00.000-07:00

Some good recent posts from factcheck.org on the 2008 presidential campaign. Here are a few excerpts:

How Giuliani made falling adoptions seem to rise using cherry-picked statistics: "In an earlier article we criticized Rudy Giuliani for saying adoptions went up 65 to 70 percent when he was mayor, when in fact adoptions at the end of his tenure were only 17 percent higher than at the start, and falling. His campaign still insists his claim is justified and offers its own interpretation of the statistical record. In this article we offer the former mayor's rationale, along with why we believe it is a classic case of how candidates and public officials sometimes use data selectively to create a false impression."

The leading three Democratic presidential candidates wax optimistic about ethanol. We provide a reality check: "Gas prices have hit record highs this year as 2008 presidential candidates outline their hopes for renewable fuels. In this story, we take a look at the reality. We focus specifically on E85, a popular ethanol-gasoline fuel blend, and the top three Democratic candidates' statements about this fuel as they fight to win votes in Iowa. We find that there are many technological bridges left to be crossed before E85, or other renewable fuels, can fulfill the role these candidates envision for them — or can start saving individual consumers cash at the pump."

Sen. John McCain has said President Bush's tax cuts have increased federal revenues. But revenues would have been even higher without them: "Republican presidential candidate Sen. John McCain has said that the major tax cuts passed in 2001 and 2003 have "increased revenues." He also said that tax cuts in general increase revenues. That’s highly misleading."

Dems Debating, the Sequel: "Former Sen. John Edwards of North Carolina talked about gas price manipulation by Big Oil where investigators have found none."

Third Time 'Round for GOP Hopefuls: "Former Massachusetts Gov. Mitt Romney committed the biggest factual fouls of the night, misleadingly asserting: That we went to war in Iraq because Saddam Hussein refused to allow weapons inspectors to come in; That there's an ocean of difference between his Massachusetts health plan and those "government takeover" plans of "every Democrat" running for president and; That Russia's income from oil exports is vastly larger than it actually is."

The Difficulty of Counting Divorces (The Numbers Guy)

2007-05-22T21:25:00.000-07:00

From The Numbers Guy (Carl Bialik) at the WSJ, an interesting look at some misleading reporting on divorce rates. The headline of an AP article says that "U.S. divorce rate lowest since 1970" and goes on to say that "despite the common notion that America remains plagued by a divorce epidemic, the national per capita divorce rate has declined steadily since its peak in 1981 and is now at its lowest level since 1970."

However, as Bialik points out, while the number of divorces per capita has gone down, so have the number of marriages per capita. So it isn't clear that the "divorce rate" as most people would define it (the chances any given marriage will end in divorce) has gone down. The misuse of this statistic by the AP article is doubly infuriating becase 1) it leads the uncritical reader to reach a potentially wrong conclusion, and 2) the question of whether divorce is more or less of a problem seems answerable.

The problem with getting a clear answer to that question is that the US Government no longer tracks detailed data on individual marriages and their outcomes. However, the crude per capita data that the AP misused can still be used to give us a hint at the answer. If you divide the divorces per capita in 2006 by marriages per capita, you get 49%, which is the source of the "50% of marriages end in divorce" stat. This number hasn't changed much since it shot up in the 1970s (in 1976 it was 51% and in 1966 it was 26%). However, the people getting divorced in a given year aren't the same ones getting married, so this number is flawed. (Although my wife pointed out to me that Renee Zellweger recently got married and divorced within the same year).

So instead, lets lag the divorce data by, say, 5 years. This seems pretty reasonable since the average marriage ending in divorce lasts ~7 years. Dividing 2006 divorces per capita by 2001 marriages per capita, we get a 46% divorce rate (closer to the number experts think is more likely correct). Again, going back to 1976, the divorce rate using this calculation was 47%.

While this analysis is very simplistic, it hardly paints a picture of steadily declining divorce rates. Here are some headlines for the AP article I think would be more appropriate:

"U.S. Divorce rate pretty much the same as it was 30 years ago"

"Americans marrying less, but not doing a much better job of choosing spouses they will stay married to"

"U.S. Marriage rate at its lowest point in the history of the country"

"U.S Divorce rate still highest in the world"

Drinking age paradox?

2007-05-03T08:17:00.000-07:00

I don't always agree with George Will, but his recent article on whether 21 is the appropriate drinking age started with me nodding my head in agreement:

"Public policy often illustrates the law of unintended consequences. Society's complexity -- multiple variables with myriad connections -- often causes the consequences of a policy to be contrary to, and larger than, the intended ones."

However, the rest of the article, whose premise is that that the legal drinking age should be lowered, is filled with half-baked arguments and numbers without context. The basic argument is that lowering the legal drinking age to 18 will help 18-21 year olds become more responsible drinkers and reduce the incidence of binge drinking and alcohol-related deaths. Will's op-ed piece as well as articles at Reason and National Review put forward some of the intellectually dishonest arguments in favor of lowering the drinking age:

Reason: "Oddly enough, high school students in much of the rest of the developed world — where lower drinking ages and laxer enforcement reign — do considerably better than U.S. students on standardized tests."

Is he seriously arguing that a lower drinking age causes higher standardized test scores? I think there may be some other variables at play here. The author either 1) is a moron or 2) thinks the people reading his article are morons.

National Review: "When it comes to alcohol, the United States is more like Indonesia, Mongolia, and Palau than the rest of the world: It is one of just four countries that requires people to be at least 21 years old to buy booze. The only countries with stiffer laws are Islamic ones."

Good argument. If Islamic countries do something, it must be wrong.

George Will: "The drinking age of 21 was one of 39 measures proposed during the 1980s by a presidential commission on drunken driving; various measures adopted did dramatically reduce the problem. But according to the National Institute on Alcohol Abuse and Alcoholism, about 5,000 people under 21 die every year from vehicular accidents, other injuries, homicides and suicides involving underage drinking."

So if the measures reduced the problem, why do you want to roll one of them back? Is 5,000 less than it used to be? If so then you should be looking for ways in addition to the current measures to improve on this number.

Haven't there been academic studies on the link between the legal drinking age and drunk-driving deaths?

Yes! Hundreds in fact. But apparently these authors didn't do the 5 minutes of research it would have taken to uncover some of them. This survey of 241 studies, from the University of Minnesota showed that "the preponderance of evidence indicates there is an inverse relationship between the MLDA and two outcome measures: alcohol consumption and traffic crashes." In other words, higher minimum legal drinking ages are correlated with less alcohol consumption and fewer traffic crashes.

The Secondary Effects of War

2007-04-14T14:41:00.000-07:00

One of the causes of poor analysis is ignoring or discounting the secondary effects of an action. Once classic example is price wars, which are usually started when one company tries to gain market share and doesn't accurately predict what its competitors will do in response.

I think one of the reasons the Iraq war has been so unsuccessful is that the secondary effects of our actions were underestimated or not seriously considered. The recent Red Cross report does a good job pointing out not only the "Shootings, bombings, abductions, murders, military operations and other forms of violence [which] are forcing thousands of people to flee their homes and seek safety elsewhere", but some of these secondary effects of war:

Health Care: "Health-care facilities are stretched to the limit as they struggle to cope with mass casualties day-in, day-out. Many sick and injured people do not go to hospital because it’s too dangerous, and the patients and medical staff in those facilities are frequently threatened or targeted."

Food: "Food shortages have been reported in several areas. According to the Iraqi Red Crescent, malnutrition has increased over the past year. The vastly inadequate water, sewage and electricity infrastructure is presenting a risk to public health."

Poverty: "Unemployment and poverty levels are rising and many families continue to rely on government food distributions to cover their immediate needs. According to government sources, an estimated one third of the population lives in poverty, while over five percent live in extreme poverty."

Physical Infrastructure: "Much of Iraq’s vital infrastructure is in a poor state of repair owing to lack of maintenance and because security constraints have impeded repair work on electrical power grids, water and sanitation systems, medical facilities and other essential facilities. Power shortages are growing worse throughout the country, including northern areas, owing largely to the failure to carry out maintenance and to increase generation capacity. Fuel shortages affecting power stations and acts of sabotage are further aggravating the crisis. As a result, water treatment plants, primary health-care centres and hospitals rely mainly on back-up generators, which often break down owing to excess usage or fall victim to the chronic fuel shortages."

Of course we aren't bombing hospitals and power plants, but our actions have indirectly caused these problems. If the proper homework was done, many of these secondary effects could have been accounted for, and we could have weighed them into our decision making process. The controversial Lancet study, which accounted for some of the human costs of the problems above, estimated 600,000 Iraqi's are dead because of our actions. I think that if we had explicitly considered these costs in our pre-war analysis, the decision we made would have been very different.

Greenpeace Bites Apple (STATS.org)

2007-04-06T16:38:00.000-07:00

Greenpeace recently published a report criticizing technology companies for their "green" practices and singling out Apple in particular. However, their criticism doesn't appear to have much data behind it, as pointed out by STATS.org. Here are a few excerpts:

"The evidence appears to be that as Apple doesn’t reclaim its unwanted or out-of-date computers, and because computers are made with chemicals that can be toxic, it’s a disgrace."

"The problem is that Greenpeace doesn’t seem to want to prove its case with actual epidemiological or toxicological data. It simple asserts that

“Right now, poison Apples full of chemicals (like toxic flame retardants, and polyvinyl chloride) are being sold worldwide. When they're tossed, they usually end up at the fingertips of children in China, India and other developing-world countries. They dismantle them for parts, and are exposed to a dangerous toxic cocktail that threatens their health and the environment.”

"It requires a little more digging to find Apple’s actual scorecard, wherein one finds that Greenpeace don’t actually measure the amount of “toxic” chemicals in Apple’s products."

"It’s easy to dismiss Greenpeace’s campaign as a cheap, self-serving publicity stunt, but it’s a stunt that has succeeded, if you look at the massive and largely uncritical media coverage. And none of the news reports critically examined the scientific basis for Greenpeace’s claims, the methodology of its ranking – or the ethics of accusing a company of being an environmental polluter without providing actual proof that there is real exposure and real harm. Apple deserves better – and so do we."

The Decoy Effect (Department of Human Behavior)

2007-04-06T16:06:00.000-07:00

Another interesting article from Shankar Vedantam, this time about biases in decision making when faced with more than two choices. Here are a few excerpts:

"Front-runners are usually focused on racing each other. They often do not realize that when people cannot decide between two leading candidates -- and it doesn't matter whether we are talking about politicians or consumer appliances -- our decision can be subtly swayed by whoever is in third place."

"Joel Huber, a Duke University marketing professor, showed how the decoy effect works with restaurants. Huber asked people whether they would prefer to eat at a five-star restaurant that was far away or at a three-star restaurant nearby. As with many choices in life, each restaurant had different advantages. If the better restaurant was also nearby, there would be no dilemma. But the question forced people to compare apples and oranges -- trade off quality against convenience -- which ensured no automatic answer."

"The human brain, however, always seeks simple answers. Enter the third candidate. Huber told some people there was also a choice of a four-star restaurant that was farther away than the five-star option. People now gravitated toward the five-star choice, since it was better and closer than the third candidate. (The three-star restaurant was closer, but not as good as the new candidate.)"

"Many people lavished hate on Ralph Nader for presumably taking votes away from the Democratic front-runner in the 2000 presidential election," said Scott Highhouse, who has studied the decoy effect at Bowling Green State University. "Research on the decoy effect suggests that Nader's presence, rather than taking votes away, probably increased the share of votes for the candidate he most resembled."

Department of Human Behavior

2007-03-25T08:59:00.000-07:00

I recently discovered this Washington Post column (thanks to Fallacy Files) called Department of Human Behavior by Shankar Vedantam. The column touches on many of the human psychological biases that can lead to errors in decision making. Here are a few examples of articles he has written about the Iraq war:

Disagree About Iraq? You're Not Just Wrong -- You're Evil

"Opponents of the war believe passionately that President Bush, his neoconservative allies and a complicit Congress deliberately misled the nation into war. Supporters of the president and the war concede that mistakes were made, especially on the question of whether Saddam Hussein had weapons of mass destruction, but say this involved no attempt to hoodwink the nation."

"What is interesting about the clash from a psychological perspective is not that supporters and critics disagree, but that large numbers of people on both sides claim to know the motives of people who disagree with them. When was the last time you heard people say that those who disagree with them on the Iraq war are well-meaning, smart, informed and thoughtful?"

"Nearly three-quarters of the people who supported the war believed that Bush was thinking about self-defense when he launched the invasion of Iraq. By contrast, fewer than 2 in 5 Americans who opposed the war were willing to grant that Bush was thinking of self-defense. Fully 70 percent of the people who supported the war said Bush was aiming to do good; only 27 percent of people who opposed the war believed that the president's motives were about doing good."

Iraq and the Danger of Psychological Entrapment

"When you invest yourself in something, it is exceedingly difficult to discard your investment. What is devilish about entrapment is not just that it can result in ever greater losses, but that those losses get you ever more entrapped, because now you have even more invested."

"[Social psychologist Scott] Plous said his alarm bells went off when he realized that President Bush was explicitly using the language of entrapment in speeches to rally support for the war. "Retreating from Iraq would dishonor the service of our brave men and women who have sacrificed in that country and have given their lives in that country, which would mean their sacrifice would be in vain," the president said recently."

""Rational decision-making should not be driven primarily by recovery of past costs," Plous said. "If you can no longer justify it in terms of what it will bring in the future and what its realistic prospects are, that is a warning sign you may have become entrapped.""

Iraq War Naysayers May Have Hindsight Bias

"Liberals' assertion that they 'knew all along' that the war in Iraq would go badly are guilty of the hindsight bias," agreed Hal Arkes, a psychologist at Ohio State University, who has studied the hindsight bias and how to overcome it. "This is not to say that they didn't always think that the war was a bad idea."

"Baruch Fischhoff, a psychologist at Carnegie Mellon University and a pioneer in the field of hindsight bias, found that Americans who made estimates about their danger after the Sept. 11, 2001, attacks recalled having made much lower estimates of risk a year later, after their fears failed to materialize."

"Indeed, research by both Fischhoff and Arkes show that people can fight the hindsight bias only when they honestly and systematically try to explain how different outcomes are possible. Such self-doubt is the exact opposite of how modern politics works: In the age of the blogosphere, certitude is king."

Blurb Watch (Fallacy Files)

2007-03-10T17:39:00.000-08:00

If you haven't seen it before, fallacyfiles.org does an occasional feature on "blurbs" from movie review taken out of context. Here are a few good ones (1, 2, 3) , and here is a sample of the most recent:

"An ad for [Norbit] has the following blurb:

"Eddie Murphy's comic skills are immense."
―Michael Wilmington, CHICAGO TRIBUNE

If you suspect that some context is missing, you're right:

Murphy's comic skills are immense, and "Dreamgirls" shows he's a fine straight dramatic actor too. So why does he want to make these huge, belching spectaculars, movies…swollen, monstrous and full of hot air…? "Norbit" makes you long for the days of "Beverly Hills Cop," when Murphy was lighter on his feet, and his movies were too."

Public school teachers underpaid?

2007-02-04T20:00:00.000-08:00

Interesting op-ed peice from the Wall Street Journal about public school teacher salaries based on this report. The basic idea is that, when looked at on an hourly basis, public school teacher salaries are good compared to other professions, which goes against the conventional logic. A few excerpts:

"According to the Bureau of Labor Statistics, public school teachers earned $34.06 per hour in 2005, 36% more than the hourly wage of the average white-collar worker and 11% more than the average professional specialty or technical worker."

"In the popular imagination, however, public school teachers are underpaid. "Salaries are too low. We all know that," noted First Lady Laura Bush, expressing the consensus view. "We need to figure out a way to pay teachers more.""

"Of course, public school teacher earnings look less impressive when viewed on an annual basis than on an hourly basis. This is because teachers tend to work fewer hours per year, with breaks during the summer, winter and spring. But comparing earnings on an annual basis would be inappropriate when teachers work significantly fewer hours than do other workers. Teachers can use that time to be with family, to engage in activities that they enjoy, or to earn additional money from other employment. That time off is worth money and cannot simply be ignored when comparing earnings. The appropriate way to compare earnings in this circumstance is to focus on hourly rates."

"Moreover, the earnings data reported here, which are taken directly from the National Compensation Survey conducted by the Bureau of Labor Statistics, do not include retirement and health benefits, which tend to be quite generous for public school teachers relative to other workers. Nor do they include the nonmonetary benefit of greater job security due to the tenure that most public school teachers enjoy."

Making bad choices (Social Science Statistics Blog)

2007-02-03T15:08:00.000-08:00

Interesting post from the Social Science Statistics Blog. It talks about a study which shows that the order in which you present information about various options can affect the decisions people make, including chosing the inferior option. A few excerpts:

"Researchers presented subjects with a set of restaurants and attributes to determined their objective "favorite." Then, two weeks later, they brought the same subjects in again and presented them with the same restaurants. This time, though, they had determined -- individually, for each subject -- the proper order of attributes that would most favor choosing the inferior alternative."

"Basically what they did is picked the attribute that most favored the inferior choice and put it first, hoping to establish that the inferior choice would get installed as the leader. The attribute that second-most favored the inferior choice was last, to take advantage of recency effects. The other attributes were presented in pairs, specifically chosen so that the ones that most favored the superior alternative were paired with neutral or less-favorable ones (thus hopefully "drowning them out.")"

"The results were that when presented with the information in this order, 61% of people chose the inferior alternative. The good news, I guess, is that it wasn't more than 61% -- some people were not fooled -- but it was robustly different than chance, and definitely more than you'd expect (since, after all, it was the inferior alternative, and one would hope you'd choose that less often). Moreover, people didn't realize they were doing this at all: they were more confident in their choice when they had picked the inferior alternative. Even when told about this effect and asked if they thought they themselves had done it, they tended not to think so (and the participants who did it most were no more likely to think they had done it than the ones who didn't)."

Going to college may be bad for your brain?

2007-01-20T10:33:00.000-08:00

From Statistical Modeling, Causal Inference, and Social Science, a post about an article from MSNBC. The article, titled "Higher education tied to memory problems later, surprising study finds", states that:

"Going to college is a no-brainer for those who can afford it, but higher education actually tends to speed up mental decline when it comes to fumbling for words later in life."

However, from the post:
"Looking at the article, we see "an older adult with 16 years of schooling or a college education scored about 0.4 to 0.8 points higher at baseline than a respondent with only 12 years of education." Based on Figures 1 and 2 of the paper, it looks like higher-educated people know more words at all ages, hence the title of the news article seems misleading."

How to read a press release (Fallacy Files)

2007-01-20T10:26:00.000-08:00

Interesting post from Fallacy Files. It shows how two completely accurate explanation of a study's finding can have a ver different impact on how it is interpreted by readers. From the post:

"One of the conventions of language use is that the order in which facts are related parallels the order in which they occurred. Another convention is that facts that are mentioned together should be relevant to one another, and a causal relationship is one form of relevance. For instance, "Mary got pregnant and got married" tells a very different story than "Mary got married and got pregnant". In the first case, you might well assume that the pregnancy causally contributed to Mary's decision to wed."

Taking Another Look At Murder Statistics (The Numbers Guy)

2007-01-20T10:12:00.000-08:00

Interesting article from The Numbers Guy about murder statistics. He points out two problems with relying on this statistic to judge the safety or police force effectiveness of a city:

1) The difference between murder and assualt is often the effectiveness of medical treatment. From the article:

"Prof. Harris was the lead author of a study in 2002 examining the decline in the murder rate between 1960 and 1999. Prof. Harris focused on the role improved medical care had on "lethality," which is the proportion of violent crimes that result in death. Even though the number of potentially deadly attacks surged in that time period, and the proportion of attacks involving guns also rose, lethality actually decreased by 70% -- to under 2% from nearly 6%. He concluded that improved medical-response times and trauma surgery were responsible for turning many would-be murders into assaults.

To demonstrate the effect of medical care, the researchers compared counties nationwide by various medical criteria, and found, for instance, that counties with at least one hospital were associated with 24% lower lethality than those without a hospital. Prof. Harris also found other medical factors that contributed to a drop in lethality of violent crimes, including the addition of physicians to a county's population and the presence of a facility for performing open-heart surgery."

2) The circumstances of a city have a big impact on violent crime. From the article:

"Think of a comparison between two hospitals' death rates (a topic I wrote about last year). If Hospital A has a higher proportion of patients suffering from life-threatening diseases, it wouldn't be fair to compare its death rate to Hospital B, which tends to treat patients with milder complaints. Prof. Friedmann's team believes that certain cities (Atlanta, Detroit) have higher underlying risk of murder than others (Denver, San Francisco).

The group relies on stats from the federal government, so its most recent ranking is for 2004. According to that ranking, San Francisco was No. 1 among 67 cities in adjusted homicide rates, despite ranking 30th in the raw rates. Atlanta, conversely, fell to No. 46 from No. 7 after the adjustments. And Detroit dropped to No. 37 on the list from No. 3 -- suggesting that the city's demographic profile helps explain what the Detroit News earlier this month called an "abysmal" crime picture for the city."

Linguistic Nonsense

2007-01-06T15:34:00.000-08:00

Fresh Air on NPR did an interesting piece about the book The Female Brain by Louann Brizendine. You can listen to the piece here or read the transcript. A few excerpts:

"Psychiatrist Louann Brizendine ... argues that most of the cognitive and social differences between the sexes are due to differences in brain structure. It's a controversial thesis. The New York Times's David Brooks and others have hailed the book as a challenge to feminist dogma, and Brizendine herself has charged that her critics are angry because her conclusions aren't politically correct. Actually, though, you can leave out the "politically" part."

"For example, Brizendine asserts that differences between men's and women's brains make women more talkative than men, and goes on to say that women on average use 20,000 words a day while men use only 7000. That factoid conforms so neatly with gender stereotypes about chatty women and taciturn men that a lot of people were indignant that anybody would spend money to discover anything so obvious. One reporter at a San Francisco TV station began his story on Brizendine by saying "Here's a news flash. Women talk more than men. Duh." Except that, duh!, it isn't true. It turns out that the figures Brizendine reported had been taken from a book by a self-help guru who had simply pulled them out of the air. And the studies that have been done generally show either that men talk slightly more than women or that the two sexes talk about the same amount."

"Or take Brizendine's claim that women on average speak twice as fast as men do. That's another cherished bit of gender lore, but no research shows anything of the sort -- the best evidence indicates that men on average speak a bit faster than women do. Nor is there any scientific basis for her claims that men think about sex every 53 seconds while women think about sex only once a day, or that women are more emotionally attentive because their more sensitive hearing enables them to hear subtle tones and nuances in speech that escape men."

"In short, saying that Brizendine's claims about sex differences in language are not exactly scientific gives "not exactly" a bad name."

Prioritizing the world's biggest problems

2007-01-06T15:25:00.000-08:00

Interesting lecture by Bjorn Lomborg, author of The Skeptical Environmentalist, about the need to prioritize the world's most important problems. The latest Copenhagen Consensus rankings are listed here.

Economist Bjorn Lomborg makes a persuasive case for prioritizing the world's biggest problems, asking "If we had $50 billion to spend over the next four years to do good in the world, where should we spend it?" His recommendations - based on the findings of the 2004 Copenhagen Consensus - controversially place global warming at the bottom of the list (and AIDS prevention at the top). Lomborg was named one of the 100 Most Influential People by Time magazine after the publication of his controversial book, The Skeptical Environmentalist which challenged widely-held beliefs that the environment is getting worse. Now the Danish economist is taking on the world's biggest problems with his Copenhagen Consensus. (Recorded February 2005 in Monterey, CA. Duration: 17:27)

The 2006 Dubious Data Awards (STATS.org)

2007-01-06T15:00:00.000-08:00

From STATS.org, the worst science stories of 2006:

Common Mistakes in Interpreting Statistics

2007-01-01T09:16:00.000-08:00

Very interesting (though long) lecture by statistician Peter Donnelly with several examples about mistakes most people make in interpreting statistics.

Statistician Peter Donnelly explores the common mistakes humans make in interpreting statistics, and the devastating impact these errors can have on the outcome of criminal trials. Donnelly is a statistics professor at Oxford University who collaborates with biologists, applying statistical models to genetics, with the hope of shedding more light on evolutionary history and the structure of the human genome (Recorded July 2005 in Oxford, UK. Duration: 22:06)