Positive results in psychology can behave like rumours: easy to release but hard to dispel. They dominate most journals, which strive to present new, exciting research. Meanwhile, attempts to replicate those studies, especially when the findings are negative, go unpublished, languishing in personal file drawers or circulating in conversations around the water cooler. “There are some experiments that everyone knows don’t replicate, but this knowledge doesn’t get into the literature,” says Wagenmakers. The publication barrier can be chilling, he adds. “I’ve seen students spending their entire PhD period trying to replicate a phenomenon, failing, and quitting academia because they had nothing to show for their time.” (…)

One reason for the excess in positive results for psychology is an emphasis on “slightly freak-show-ish” results, says Chris Chambers, an experimental psychologist at Cardiff University, UK. “High-impact journals often regard psychology as a sort of parlour-trick area,” he says. Results need to be exciting, eye-catching, even implausible. Simmons says that the blame lies partly in the review process. “When we review papers, we’re often making authors prove that their findings are novel or interesting,” he says. “We’re not often making them prove that their findings are true.”

I have briefly discussed the lack of replication in economics here, but in short, the issue is that once a finding is published, there are practically no incentives for people to replicate those findings.

There are two reasons for this. The first is that journals tend to want to publish only novel results, so even if you manage to confirm someone else’s findings, there will be few takers for your study unless you do something significantly different… in which case you’re no longer doing replication.

The second is the tendency to publish only studies in which the authors find support for their hypothesis. This is known as “publication bias.”

For example, suppose I hypothesize that the consumption of individuals increases as their income increases, and suppose I find support for that hypothesis using data on US consumers. This result eventually gets published in a scientific journal. Suppose now that you decide to replicate my finding using Canadian data and you fail to replicate my findings. Few journals would actually be interested in such a finding. That’s because failing to reject the null hypothesis in a statistical test is not surprising (after all, you’ve staked 90, 95, or 99 percent of the probability mass on the null hypothesis that consumption is not associated with income), but also because, as Yong’s article highlights, that would not exactly be an “exciting, eye-catching” result.

I am currently dealing with such a “negative finding” in one of my papers, in which I find that land titles do not have the positive impact on productivity posited by the theoretical literature in Madagascar, a context where donors have invested hundreds of millions of dollars in various land titling policies. Perhaps unsurprisingly, the paper has proven to be a very tough sell.

(HT: David McKenzie.)

[The gay] population is on the whole law-abiding and productively employed, and having a below-normal fertility rate does not impose the same costs on the education and welfare systems as the heterosexual population does. It is thus not surprising that in response to the President’s announcement of his support for homosexual marriage, Republican leaders cautioned their followers not to be distracted by this issue from the problems of the US economy. This was tacit acknowledgment that homosexual marriage, and homosexual rights in general, have no economic significance.

It seems that the only remaining basis for opposition to homosexual marriage, or to legal equality between homosexuals and heterosexuals in general, is religious. Many devout Christians, Jews, and Muslims are strongly opposed to homosexual marriage, and to homosexuality more generally. Why they are is unclear. If as appears homosexuality is innate, and therefore natural (and indeed there is homosexuality among animals), and if homosexuals are not an antisocial segment of the population, why should they be thought to be offending against God’s will? Stated differently, why has sex come to play such a large role in the Abrahamic religions? I do not know the answer. But whatever the answer, the United States is not a theocracy and should hesitate to enact laws that serve religious rather than pragmatic secular aims, such as material welfare and national security.

The emphasis is mine, but here is more from Richard Posner, and here is his co-blogger Gary Becker’s response.

Interestingly, in his post, Posner brings up the 1975 Supreme Court decision in Loving v. Virginia. In that case, the Court held that prohibiting interracial marriage was unconstitutional.

I believe that is an important precedent which will likely be instrumental in eventually allowing gay marriage. From the Wikipedia article for Loving v. Virginia:

Although the Majority Opinion of the New York Court of Appeals in Hernandez v. Robles (2006) (which was overridden by the New York State Legislature via enactment of the Marriage Equality Act in 2011) rejected any reliance upon the Loving case as controlling upon the issue of same-sex marriage, holding that:

[T]he historical background of Loving is different from the history underlying this case. [...] But the traditional definition of marriage is not merely a by-product of historical injustice. Its history is of a different kind. The idea that same-sex marriage is even possible is a relatively new one. Until a few decades ago, it was an accepted truth for almost everyone who ever lived, in any society in which marriage existed, that there could be marriages only between participants of different sex. A court should not lightly conclude that everyone who held this belief was irrational, ignorant or bigoted. We do not so conclude.

In the August 4, 2010 federal district court decision in Perry v. Schwarzenegger, which overturned California’s Proposition 8 (which restricted marriage to opposite-sex couples), Judge Vaughn Walker cited Loving v. Virginia to conclude that “the [constitutional] right to marry protects an individual’s choice of marital partner regardless of gender.” On more narrow grounds, the Ninth Circuit Court of Appeals affirmed.

In no way does this change what happened last week in my own state of North Carolina, but it does provide hope that things can change for the better.

Look at lists of “100 Things to Do Before You Die,” and you’ll find them dominated by exotic sensations of one kind or another (“Skydive”; “Shower in a waterfall”; “Eat jellied eels from a stall in London”).

Really? This is the best we can do? This is what it’s all about? These are the things that make our lives worth living? When I think about what really makes me happy, what I really crave, I come up with a very different list: concentrated, purposeful work, especially creative work; being with people I love; feeling like I’m part of something larger. Meaning, connectedness, doing strenuously what you do well: not sights, not thrills, and not even pleasures, as welcome as they are. Not passivity, not letting the world come in and tickle you, but creativity, curiosity, altruism, engagement, craft. Raising children, or teaching students, or hanging out with friends. Playing music, not listening to it. Making things, or making them happen. Thinking hard and feeling deeply.

None of which involve spending money, except in an ancillary way. None of which, in other words, are consumer experiences.

(HT: Andrew Sullivan.)

Though this obviously omits many other food staples, the underlying assumption is that various kinds of food are substitutes (imperfect ones, but substitutes nonetheless) for one another.

In cases where a more refined notion of food prices is used for discussion, the food price measure used is often the Food and Agriculture Organization (FAO) of the United Nations’ food price index, which encompasses five categories of food: cereals, dairy, meat, oils and fats, and sugar.

The FAO’s food price index, however, does not include fish and seafood. But since fish and seafood are a key source of protein for almost half of the world’s population, this is an important omission that can lead to making the wrong policy recommendations.

Some of my coauthors thus developed a fish price index similar to the other food (i.e., cereals, dairy, meat, oils and fats, and sugar) price indices already used by the FAO. We recently wrote a paper discussing this new fish price index, which the FAO will incorporate in its food price index sometime this year.

Here is the abstract of our resulting PLoS ONE article on the FAO’s fish price index, titled “Fish Is Food: The FAO’s Fish Price Index,” which was published this week:

World food prices hit an all-time high in February 2011 and are still almost two and a half times those of 2000. Although three billion people worldwide use seafood as a key source of animal protein, the Food and Agriculture Organization (FAO) of the United Nations — which compiles prices for other major food categories — has not tracked seafood prices. We fill this gap by developing an index of global seafood prices that can help to understand food crises and may assist in averting them. The fish price index (FPI) relies on trade statistics because seafood is heavily traded internationally, exposing non-traded seafood to price competition from imports and exports. Easily updated trade data can thus proxy for domestic seafood prices that are difficult to observe in many regions and costly to update with global coverage. Calculations of the extent of price competition in different countries support the plausibility of reliance on trade data. Overall, the FPI shows less volatility and fewer price spikes than other food price indices including oils, cereals, and dairy. The FPI generally reflects seafood scarcity, but it can also be separated into indices by production technology, fish species, or region. Splitting FPI into capture fisheries and aquaculture suggests increased scarcity of capture fishery resources in recent years, but also growth in aquaculture that is keeping pace with demand. Regionally, seafood price volatility varies, and some prices are negatively correlated. These patterns hint that regional supply shocks are consequential for seafood prices in spite of the high degree of seafood tradability.

Any article, journalistic or scientific, that sparks a debate typically winds up looking more like a good manuscript 700 years ago than a magazine piece only 10 years ago. The truth is that every decent article now aspires to become the wiki of its own headline.

Sure, there is still the authority that comes of being a scientist publishing a peer-reviewed paper, or a journalist who’s reported a story in depth, but both such publications are going to be crowd-reviewed, crowd-corrected and, in many cases, crowd-improved. (And sometimes, crowd-overturned.) Granted, it does require curating this discussion, since yahoos and obscenity mavens tend to congregate in comment sections.

That’s from a New York Times op-ed in last weekend’s Sunday Review by Jack Hitt, who is also a frequent contributor to This American Life (here is my favorite This American Life story by Jack Hitt).

Hitt’s point should be be taken more seriously by academics. In all fairness, however, in some corners of academia, the idea is being taken seriously: the AEJs — the four new journals of the American Economic Association — have comments section for every published article (I don’t know why the AEA has not also done so for its flagship journal, the American Economic Review.)

Unfortunately, readers of the AEJs seem to be slow to embrace that change, as few articles appear to have garnered any comments. Moreover, a quick look at the latest issue of each AEJ indicates no comments at all. Perhaps the problem is that one needs to be a member of the AEA to comment.

If those comments thread ever take off, and if other journals start offering similar comment sections, this would be a cheap, quick way of building canonical knowledge within any discipline, as I discussed in my previous post on this topic.

Contract farming is widely perceived as a means of increasing welfare in developing countries. Because of smallholder self-selection in contract farming, however, it is not clear whether contract farming actually increases grower welfare. In an effort to improve upon existing estimates of the welfare impacts of contract farming, this paper uses the results of a contingent-valuation experiment to control for unobserved heterogeneity among smallholders. Using data across several regions, firms, and crops in Madagascar, results indicate that a 1-percent increase in the likelihood of participating in contract farming is associated with a 0.5-percent increase in household income, among other positive impacts.

If I had to summarize the paper’s contribution informally, I’d say the estimates it presents of the welfare impacts of contract farming have better internal and external validity than those found in previous studies.

Click here for an ungated, older version (link opens a .pdf document), but note that the results in the ungated version had not undergone peer review, so they are not as solid.

The recent wave of randomized trials in development economics has provoked criticisms regarding external validity and the neglect of political economy. We investigate these concerns in a randomized trial designed to assess the prospects for scaling-up a contract teacher intervention in Kenya, previously shown to raise test scores for primary students in Western Kenya and various locations in India. The intervention was implemented in parallel in all eight Kenyan provinces by a nongovernmental organization (NGO) and the Kenyan government. Institutional differences had large effects on contract teacher performance. We find a significant, positive effect of 0.19 standard deviations on math and English scores in schools randomly assigned to NGO implementation, and zero effect in schools receiving contract teachers from the Ministry of Education. We discuss political economy factors underlying this disparity, and suggest the need for future work on scaling up proven interventions to work within public sector institutions.

Bold et al.’s finding points to an important problem with the findings of many randomized controlled trials (RCTs): No matter how careful one is in ensuring that subjects are randomly assigned to the treatment and control groups, almost all RCTs rely on only one implementing partner.

Implementation Bias

Take the RCT my co-authors and I are currently running in Mali, for example. Insurance markets are nonexistent in most developing countries, and agriculture is a risky business due to uncertain weather, among other sources of uncertainty. So we wanted to know whether having access to insurance significantly affected the production decisions (and, ultimately, the welfare) of cotton producers in southern Mali.

In order to answer our research question, we randomly assigned cotton producer cooperatives to the treatment (i.e., offered the insurance) and control (i.e., not offered the insurance) groups. And because we cannot force people to buy insurance, we randomly offered discounts (i.e., 0, 25, 50 percent off) on the price of the insurance to coops within the treatment group so as to encourage take up.

So far, so good? Not quite. In order to implement this project, we partnered up with an NGO — the implementing partner in this context, whose job it is to sell the insurance. Ideally, however, we would have partnered with at least one other implementing partner and then randomized over which implementing partner gets to implement the insurance.

Why? Because with only one implementing partner, it is impossible to tell whether the success or failure of any intervention is due to the perception people in the treatment group have of the implementing partner. Put more succinctly: Why should farmers trust a health NGO when it offers them supposedly improved seeds? The answer lies in having more than one implementing partner and in also randomizing over which partner gets to implement in which village.

I realize that implementation bias is an external validity problem, and that the solution to external validity problems is replication. But as long as researchers in the social sciences have little to no incentives for replication, that solution remains a bit of a cop-out. Maybe we will converge toward a situation where replication will be a necessary condition for publication.

In a post on the World Bank’s Development Impact blog, Gabriel Demombynes discusses how he expects the Bold et al. finding to be a kind of Rorschach test for people’s perceptions of RCTs.

Portrait of Artistotle (Source: Wikimedia Commons.)

I was in Washington last month to discuss my work on food prices, in which I look at whether food prices cause social unrest, at an event whose goal was to discuss the link between climate change and conflict.

As many readers of this blog know, disentangling causal relationships from mere correlations is the goal of modern science, social or otherwise, and though it is easy to test whether two variables x and y are correlated, it is much more difficult to determine whether x causes y.

So while it is easy to test whether increases in the level of food prices are correlated with episodes of social unrest, it is much more difficult to determine whether food prices cause social unrest.

In my work, I try to do so by conditioning food prices on natural disasters. To make a long story short, if you believe that natural disasters only affect social unrest through food prices, this ensures that if there is a relationship between food prices and social unrest, that relationship is cleaned out of whatever variation which is not purely due to the relationship flowing from food prices to social unrest. In other words, this ensures that the estimated relationship between the two variables is causal. This technique is known as instrumental variables estimation.

Identifying Causal Relationships vs. Ruling Out All Other Causes

As with almost any other discussion of a social-scientific issue nowadays, the issue of causality came up during one of the discussions we had at that event in Washington. It was at that point that someone implied that it did not make sense to talk of causality by bringing up the following analogy:

Imagine a house on stilts. Mold has been accumulating on the stilts for a number of years. One day, a hurricane comes along, and the house collapses. Can we really say that the hurricane caused the house to collapse when the mold has been eating away at the stilts for a long time?

I am paraphrasing, but the idea was that it was impossible to talk of causality given that most things had both proximate causes and distal causes — the hurricane and the mold, respectively, in the example above.

When the house-on-stilts analogy was brought up, many shrank and responded that establishing whether x causes y is not the same as saying “x is the only cause of y.”

Likewise, establishing that food prices cause social unrest is not the same thing as saying that food prices are the only cause of social unrest. Of course there are other factors.

When food prices rise sharply, people might be rioting in Lagos, but they are unlikely to riot in Milwaukee, which tells us that there must be something else going on.

But the fact that something else might going on does not mean that we cannot ask whether x causes y. When we talk about identifying causal relationships, we are not talking about ruling out all other possible causes. Sometimes a cigar is just a cigar, and when we ask whether x causes y, what we are really asking is “Does x cause y?,” and not ”Is x the only cause of y?” The former is answerable; the latter is akin to asking about the unmoved mover of Aristotle’s Metaphysics.

In this article, we argue that when patron-client relations are grounded in economic relationships, such as between landlord and worker, we should expect clientelism to influence not just how public policy, the state, and the political system work, but also how the economy works. We develop a simple model of the economic consequences of electoral clientelism when voting behavior can be observed. Landlords/patrons provide economic rents to workers, and in exchange workers vote for parties favored by landlords. As votes are used by the landlords to accumulate political rents, vote control increases the demand for labor and for land. The model implies that the introduction of the Australian ballot, which destroys this form of clientelism, should lead to a fall in the price of land in those areas where patron-client relationships are strongest. We test the predictions of the model by examining in detail the evolution of land prices in Chile around May 31, 1958, for which we collected original data. A characteristic of rural Chile at this time were patron-client relations based on the inquilinaje system, by which a worker, the inquilino, entered into a long-term, often hereditary, employment relationship with a landlord and lived on his landlord’s estate. We show that the introduction of the Australian ballot in 1958 led to a fall of about 26% in land prices in the areas where these patron-client relationships were predominant.

This article is a followup of sorts to Baland and Robinson’s 2010 American Economic Review article, in which they show that the introduction of the secret ballot in 1958 in Chile led to greater support for labor-friendly political parties.

The Floor of the Chicago Board of Trade

A farmer lives with two time horizons in mind. One is the months-long growing seasons his crops abide by. The other is the immediate reality of having to feed his family each day, regardless of the price of grain at harvest in three months, whether a drought will wither plants in the field, or whether perfect rains will yield a bumper crop.

Across rural Africa, such uncertainty hounds smallholder farmers—which is nearly everyone. In Ethiopia, 80 percent of the population of more than 80 million are small-scale farmers and produce 95 percent of the country’s agricultural output.

If more and better information within agricultural markets can make uncertainty recede like darkness in front of a candle, the Ethiopian Commodity Exchange is a bank of high-powered floodlights. A commodity exchange that broadcast crop prices to rural farmers not only helps them get higher prices for their produce, but also improves the food distribution system to resist shortages in times of drought.

That’s from a recent article in GOOD magazine by Tate Watkins.

The article discusses the potential for commodity exchanges to improve food security in Africa. In a nutshell, at times of impending food scarcity, commodity exchanges can help by raising food prices, which can help avert food crises and famines. This helps food consumers by improving their food security.

But commodity exchanges can help food producers by smoothing prices over time. That is, commodity exchanges can help reduce the uncertainty over the prices farmers will face come harvest time, which in turn leads farmers to making more efficient production decisions.

What about Commodity Speculation?

This is in stark contrast with the oft-touted “fact” according to which commodity speculation caused the food crisis of 2008. If you are interested in commodity speculation, see this Energy Economics article by Scott Irwin and Dwight Sanders, in which the authors argue that there is little to no causal evidence that commodity speculation led to the 2008 spike in food prices.

Tate interviewed me for the GOOD magazine article quoted above, and one of the things I said ended up making it to the article. I will always be grateful to Tate for bringing to my attention the Kansas City Star‘s style guide, which supposedly helped Ernest Hemingway develop his distinctive style. Tate has his own blog here, and you can follow him on Twitter here.