Levitt’s publishers characterize him as a “rogue economist,” yet he received his Ph.D. from MIT, holds the title of Alvin H. Baum Professor at the University of Chicago, and has served as editor of the completely mainstream Journal of Political Economy. Further “rogue” credentials revealed by Levitt’s online C.V. include an undergraduate degree from Harvard, a research fellowship with the American Bar Foundation, membership in the Harvard Society of Fellows, a fellowship at the National Bureau of Economic Research, and a stint as a consultant for “Corporate Decisions, Inc.”

That’s all well and good, but, on the other hand, I too have degrees from Harvard and MIT and I also taught at the University of Chicago. But what really clinches it is that this month I gave a talk for an organization called the Corporate Executive Board. No kidding.

In my defense, I’ve never actually called myself a “rogue.” But still . . .

The much remarked “readability” of the book must have played a part in this smooth passage from the page to the screen, since “readability” has to do not only with freedom from obscurity but, paradoxically, with freedom from the actual sensation of reading [emphasis added]—of the tug and traction of words as they move thoughts into place in the mind. Requiring, in fact, the least reading, the most “readable” book allows its characters to slip easily through nets of words and into other forms. Popular art has been well defined by just this effortless movement from medium to medium, which is carried out, as Leslie Fiedler observed in relation to Uncle Tom’s Cabin, “without loss of intensity or alteration of meaning.” Isabel Archer rises from the page only in the hanging garments of Henry James’s prose, but Scarlett O’Hara is a free woman.

Well put. I wish Pierpoint would come out with another book. But I think this sort of book is out of fashion nowadays. There are zillions of uncollected book reviews and literary essays that I’d love to see in book form (the hypothetical collected reviews of Anthony West, Alfred Kazin, and many others) but it seems like it won’t ever happen.

I have just come across your paper in the 2009 American Scientist. Another problem that I frequently come across is when people do power spectral analyses of signals. If one has 1200 points (fairly modest in this day and age) then there are 600 power spectral estimates. People will then determine the 95% confidence limits and pick out any spectral estimate that sticks up above this, claiming that it is significant. But there will be on average 30 estimates that stick up too high or too low. So in general there will be 15 spectral estimates which are higher than the 95% confidence limit which could happen just by chance. I suppose that this means that you have to set a much higher confidence limit, which would depend on the number of data in your signal.

I would also like your opinion about a paper in the Proceedings of the National Academy of Science, “The causality analysis of climate change and large-scale
human crisis” by David D. Zhang, Harry F. Lee, Cong Wang, Baosheng Li, Qing Pei, Jane Zhang, and Yulun An.

These authors take whole series of annual data from 1500 to 1800, giving 301 data in all and do linear correlations between pairs of data sets them. But some of the data sets only have data at longer intervals, such as 25 years. So the authors linearly interpolate the data to give an annual signal and then assume that they still have 301 data. Is this legitimate?

My reply:

1. For your spectral estimation problem, I think it would best to fit some sort of hierarchical model for the 600 parameters.

2. I didn’t actually read the paper, but from your description I’d think it might be a good idea for them to bootstrap their data to get standard errors.

Here’s what it looked like:

“The case against the Federal minimum wage,” huh? That surprised me, as I had the vague impression that Clover was on the far left of the American political spectrum. But I guess he could have some sort of wonky thing going on, or maybe there’s some unexpected twist? It seemed a bit off of Clover’s usual cultural-criticism beat, so I clicked through to take a look . . . and it was just a boring set of paragraphs on the minimum wage.

Hmmmm. I went back to the homepage, looked around more carefully, and realized that the blog is fake, the online equivalent of those fake book spines that are used to simulate rows of books on a bookshelf.

I don’t know what happened. My guess is that Clover got tired of blogging and let the domain name lapse, and then some loser entrepreneur noticed it was still getting some hits (from DeWitt’s blog?) so they put up a fake blog.

I can only assume it was all done automatically? Somebody has a webcrawler that looks for dead sites with links, then buys them up for something close to $0 and fills ‘em with crap? Yuck.

Also see DataMarket, InfoChimps, and a few older links in The Future of Data Analysis.

If you access the data through the API, you can build live visualizations like this:

Of course, you could just go to the source. Roy Mendelssohn writes (with minor edits):

Since you are both interested in data access, please look at our service ERDDAP:

http://coastwatch.pfel.noaa.gov/erddap/index.html

http://upwell.pfeg.noaa.gov/erddap/index.html

Please do not be fooled by the web pages. Everything is a service (including search and graphics) and the URL completely defines the request, and response formats are easily changed just by changing the “file extension”. The web pages are just html and javascript that use the services. For example, put this URL in your browser:

http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAsstamday.png?sst[(2010-01-16T12:00:00Z):1:(2010-01-16T12:00:00Z)][(0.0):1:(0.0)][(30):1:(50.0)][(220):1:(240.0)]

Now if you use R:


library(ncdf4)
library(lattice)
download.file(url="http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAsstamday.nc?sst[(2010-01-16T12:00:00Z):1:(2010-01-16T12:00:00Z)][(0.0):1:(0.0)][(30):1:(50.0)][(220):1:(240.0)]", destfile="AGssta.nc")
AGsstaFile<-nc_open('AGssta.nc')
sst<-ncvar_get(AGsstaFile,'sst',start=c(1,1,1,1),count=c(-1,-1,-1,-1))
lonval<-ncvar_get(AGsstaFile,'longitude',1,-1)
latval<-ncvar_get(AGsstaFile,'latitude',1,-1)
image(lonval,latval,sst,col=rainbow(30))

Or if you use Matlab:

link='http://coastwatch.pfeg.noaa.gov/erddap/griddap/erdBAsstamday.mat?sst[(2010-01-16T12:00:00Z):1:(2010-01-16T12:00:00Z)][(0.0):1:(0.0)][(30):1:(50.0)][(220):1:(240.0)]';
F=urlwrite(link,'cwatch.mat');
load('-MAT',F);
ssta=reshape(erdBAsstamday.sst,201,201);
pcolor(double(ssta));shading flat;colorbar;

The two services above allow access to literally petabytes of data, some observed some from model output. I realize you guys don’t usually work in these fields, but this is part of a significant NOAA effort to make as much of its data available as possible. One more thing, if you use “last” as the time, you will always get the latest data, This allows people to set up web pages that track the latest (algal bloom) conditions, such as done by one of my colleagues.

BTW – for people who want a GUI to help with the extract from within the app, there is a product called the Environmental Data Connector that runs in ArcGIS, Matlab, R and Excel.

Roy’s links inspired me to write another blog post, which is forthcoming.

This post is by Aleks Jakulin, follow him at @aleksj.

Nowadays there are rules. You can break the rules, but then you’re Breaking. The. Rules. Which is a distinctive choice all its own.

Consider academic writing. Serious works of economics or statistics tend to be written in a serious style in some version of plain academic English. The few exceptions (for example, by Tukey, Tufte, Mandelbrot, and Jaynes) are clearly exceptions, written in styles that are much celebrated but not so commonly followed.

A serious work of statistics, or economics, or political science could be written in a highly unconventional form (consider, for example, Wallace Shawn’s plays), but academic writers in these fields tend to stick with the standard forms. The consensus seems to be that straight prose is the clearest way to convey interesting and important ideas. Serious popular writers such as Oliver Sacks and Malcolm Gladwell follow a slightly different formula, going with the magazine-writing tradition of placing ideas inside human stories. But they still, by and large, are trying to write clear prose.

When it comes to data graphics, though, we’re back in the freewheeling 1700s. Maybe that’s a good thing, I don’t know. But what I do know is there’s no standard way of displaying quantitative information, nor is there any acceptance of the unique virtues of the graphical equivalent of clear prose.

Serious works of social science nowadays use all sorts of data display, from showing no data at all, to tables, to un-designed Excel-style bar charts, to Cleveland-style dot and line plots, to creative new data displays, to ornamental information visualizations. The analogy in writing style would be if some journal articles were written in the pattern of Ezra Pound, others like Ernest Hemingway, and others in the style of James Joyce or William Faulkner.

I won’t try to make the case that everybody should do graphs the way I do. I accept that some people communicate with tables, others prefer infovis, and others prefer no quantitative information at all. I just think it’s interesting that prose style is so standardized—I’ve had submissions to journals criticized on the grounds that my writing is too lively!—but when it comes to display of data and models, it’s the Wild West.

For example . . .

Kaiser points to this graph from the book Poor Economics by Abhijit Banerjee and Esther Duflo:

In case you’re curious what’s actually going on here, Kaiser helpfully replots the data in a readable form:

I’d be interested in what my infovis friends would say about this. The best argument I can think of in favor of the Banerjee and Duflo graph, besides its novelty and (perhaps) attractiveness, is that its very difficulty forces the reader to work, to put in so much effort to figure out what’s going on that he or she is then committed to learning more. In contrast, one might argue that Kaiser’s direct plot is so clear that the reader can feel free to stop right there. I don’t really believe this argument—I’d rather have the clear graph and convey more information—but that’s the best I can do.

That said, if a book has dozens of informative Kaiser-style graphs, I can see the benefit of having a few goofy ones just to mix things up a bit.

Using rare surnames we track the socio-economic status of descendants of a sample of English rich and poor in 1800, until 2011. We measure social status through wealth, education, occupation, and age at death. Our method allows unbiased estimates of mobility rates. Paradoxically, we find two things. Mobility rates are lower than conventionally estimated. There is considerable persistence of status, even after 200 years. But there is convergence with each generation. The 1800 underclass has already attained mediocrity. And the 1800 upper class will eventually dissolve into the mass of society, though perhaps not for another 300 years, or longer.

Read more at Steven’s blog. The idea of rare names to perform this analysis is interesting – and has been recently applied to the study of nepotism in Italy.

I haven’t looked into the details of the methodology, but rare events have their own distributional characteristics, and could benefit from Bayesian modeling in sparse data conditions. Moreover, there seems to be an underlying assumption that rare names are somehow uniformly represented in the population. They might not be. A hypothetical situation: in feudal days, rare names were good at predicting who’s rich and who’s not – wealth was passed through family by name. But then industrialization perturbed the old feudal order stratified by name into one that’s stratified by skill and no longer identifiable by name.

Let’s scrutinize this new methodology! With power comes responsibility.

This post is by Aleks Jakulin.

I [Karl] personally would avoid sports entirely, as I view the subject to be insufficiently serious. . . . Certainly lots of statisticians are interested in sports. . . . And I’m not completely uninterested in sports: I like to watch football, particularly Nebraska, Green Bay, and Baltimore, and to see Notre Dame or any team from Florida or Texas lose.

But statistics about sports? Yawn.

As a person who loves sports, statistics, and sports statistics, I have a few thoughts:

1. Not everyone likes sports, and even fewer are interested in any particular sport. It’s ok to use sports examples, but don’t delude yourself into thinking that everyone in the class cares about it.

2. Don’t forget foreign students. A lot of them don’t even know the rules of kickball, fer chrissake!

3. Of the students who care about a sport, there will be a minority who really care. We had some serious basketball fans in our class last year.

4. I think the best solution is to cover examples in all sorts of topics, including but not limited to sports. I’ve been trying to work in more examples from areas such as cooking, sewing, and shopping.

5. In my experience, students looove education examples, stories about grades, studying, and so forth. But maybe that’s just at the sorts of colleges where I’ve taught: Columbia, Harvard, Berkeley, Chicago. Perhaps students at less elite institutions are less interested in grades.

6. Getting back to Karl’s point about sports being unimportant: Yeah, I pretty much agree with him on that one. Psychologists and economists who study sports will make the claim that the research has larger value, for example in studying decision making or in isolating some cognitive process (as in the justly-celebrated “hot-hand” study), but ultimately I think sports are valuable for their own sake. Sports are a form of art, it’s not a topic such as medicine or education that has much interest beyond itself. That’s ok, though, as long as we’re honest about it, and as long as we also include examples that interest other students in the class.

7. Whenever you teach an applied example well, you induce some subject-matter learning. When I teach sex ratios of births, I give the probability as 0.485, not 0.5, and students learn a little bit of biology. When I teach a sports example, students learn a bit about sports and psychology (for example, the hot hand). The one thing I never never like to do is use complicated gambling examples. I have no interest in teaching students the rules of craps or the probability of getting three of a kind in a poker hand. There are lots of probability examples out there that have the same level of complexity but apply to real-world situations.

Despite empirical psychologists’ nominal endorsement of a low rate of false-positive findings (≤ .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We [Simmons, Nelson, and Simonsohn] present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

Whatever you think about these recommendations, I strongly recommend you read the article. I love its central example:

To help illustrate the problem, we [Simmons et al.] conducted two experiments designed to demonstrate something false: that certain songs can change listeners’ age. Everything reported here actually happened.

They go on to present some impressive-looking statistical results, then they go behind the curtain to show the fairly innocuous manipulations they performed to attain statistical significance.

A key part of the story is that, although such manipulations could be performed by a cheater, they could also seem like reasonable steps to a sincere researcher who thinks there’s an effect and wants to analyze the data a bit to understand it further.

We’ve all known for a long time that a p-value of 0.05 doesn’t really mean 0.05. Maybe it really means 0.1 or 0.2. But, as this paper demonstrates, that p=.05 can often mean nothing at all. This can be a big problem for studies in psychology and other fields where various data stories are vaguely consistent with theory. We’ve all known about these problems but it’s only recently that we’ve been aware of how serious they are and how little we should trust a bunch of statistically significant results.

Sanjay Srivastava has some comments here. My main comment on Simmons et al. is that I’m not so happy with the framing in terms of “false positives”; to me, the problem is not so much with null effects but with uncertainty and variation.

In brief:

1. I was unfair to call him a Tucker Carlson.

2. Murray talks a lot about upper-class liberals. That’s fine but I think his discussion would be improved by also considering upper-class conservatives, given that I see the big culture war occurring within the upper class.

3. Using the case of Joe Paterno as an example, I discuss why Murray’s “preach what you practice” advice could be difficult to carry out in practice.

Murray on the top 5%

David Frum quoted Murray as writing that the top 5% “tends to be liberal—right? There’s no getting around it. Every way of answering this question produces a yes.” In response, Frum and I both pointed out that, no, Americans in the top 5% of income are less likely to be liberal, compared to the average American, and are more likely to vote Republican.

Those numbers are correct, but it was unfair to present them as a contradiction of Murray, who when talking in his book about the top 5% is not talking about income. Murray defines “the broad elite” as “most successful 5 percent of the people working in the professions and managerial positions,” including top military officers, government officials, business executives, professionals, and the media, a set of occupations that include, in Murray’s words, “23 percent of all employed persons ages 25 or older.” He’s talking about the top 5% (in “success,” as broadly defined, which is related to but not quite the same as income) in these professions.

After his offhand remark about the upper class being liberal (more on that below), Murray takes pains to emphasize that this popular impression is exaggerated, writing, “the essence of the culture of the new upper class is remarkably consistent across the political spectrum.” The concept of upper-class people being liberal is not central to Murray’s argument; if anything, his point is the opposite, to de-emphasize the liberal tilt of “famous academics, journalists, Hollywooders, etc.” and rather make the point that, that whatever the political attitudes are of the new upper class, their attitudes and actions isolate them from mainstream America.

Getting back to Murray’s upper 5%: as he defines them, I’d guess they are more conservative than the average American on economic issues and more liberal than the average American on social issues. But I can’t really be sure.

Rather than defining the American upper class as including some job categories but not others, I’d prefer to include all the high-income groups and say that the American upper class is highly divided—that is, polarized. Murray does address much of this in his comparison of different sorts of SuperZips (high-income zip codes), so maybe it’s just a matter of emphasis: from my analysis of survey data (as in the graphs posted earlier), I see the big culture war occurring within the upper class, whereas Murray focuses on differences in attitudes and lifestyles comparing rich to poor.

As I noted earlier, upper-income liberals, while a minority of upper-income Americans, are still an influential group and worth studying. But alongside them is an even larger group of upper-income conservatives.

I think Murray and I are basically in agreement about the facts here. If you take narrow enough slices and focus on the media, academia, and civilian government, you can find groups of elites with liberal attitudes on economic and social issues. But I’m also interested in all those elites with conservative attitudes. Statistically, they outnumber the liberal elites. The conservative elites tend to live in different places than the liberal elites and they tend to have influence in different ways (consider, for example, decisions about where to build new highways, convention centers, etc., or pick your own examples), and those differences interest me.

In summary, it was unfair of me to lump Murray in with Tucker Carlson as a statistics-mangler. I think that any focus on upper-class liberals would gain more context by contrasting them with the more numerous upper-class conservatives, but Murray’s real point has little to do with political attitudes, and if you remove his comments about the purported liberalism of elites, nothing is really taken away from his main arguments.

“The New American Divide”

Murray describes his book (see also this Wall Street Journal article) as “about an evolution in American society” in the past half-century, “leading to the formation of classes that are different in kind and in their degree of separation from anything that the nation has ever known,” with a new upper class that now lives a life that is qualitatively different from the experiences of most Americans.

I see this argument having the following logical implications, in the context of Murray’s conservative political attitudes (i.e. that he favors low taxes and low public spending):

As I read it, Murray’s argument plus his political opinions imply the following story: Rich liberals lead personally admirable and economically productive lives, but they are tied to a false ideology of socialism and social permissiveness. This left-wing ideology may have its appeal, but in the long term, or even the medium term, it does no favors for most poor and middle-income Americans, as it leads to economic stagnation (the natural result of money spent through the government’s political process rather than through the decisions of individuals and private businesses) and social disaster (all the problems that arise with families when individuals attempt to live their lives without restraint).

Murray writes about culturally and politically influential elites because they have the ability to influence American attitudes, both thorough their economic power and through their representation in the news and entertainment media. Murray writes about politically liberal rich elites because he disagrees with their politics. From Murray’s point of view, there’s no point in writing about rich conservatives (for example, that dude who’s funding Rick Santorum) because they are already doing what he wants, advocating for lower taxes, lower government spending, and more restrictions on the behavior of lower-income Americans.

The above is not a quote; it’s just my attempt to draw out the implications of Murray’s thesis that the upper class should “preach what it practices” and recommend to ordinary Americans the attitude of long-term responsibility.

Just to be clear, let me emphasize that Murray’s book does not distinguish between a “good” elite that’s conservative and a “bad” elite that’s liberal. He considers the new upper class as problematic as a class. My point above is that, given his political views, it makes sense for Murray to be more concerned about the attitudes of the liberal elite, a concern Murray can have without implying any moral criticism on his part.

Again, Murray never writes anything like the bit I have above about economic stagnation; this is just my interpretation of the implications of his concerns in the context of his economic beliefs.

And let me also make clear that Murray does not consider the politics of the new upper class in making his case that it’s problematic. Even if the American upper class were 100% conservative, Murray could still be concerned about their disconnect with the masses. But I think the contrast between liberal and conservative views is relevant given Murray’s own attitudes.

One way to see this is to consider Murray’s political quiz, “How thick is your bubble,” where he challenges his upper-class readers to assess their points in common to the ordinary Americans. One of Murray’s questions is, “Have you ever participated in a parade not involving global warming, a war protest, or gay rights?” The bit about gay rights is cute, but it also serves to separate out the liberals in the audience. After all, lots of non-elites go to gay rights parades. What if Murray had asked, “Have you ever participated in a parade not involving the pro-life or Tea Party movements?” This might not be the best example; my point is that there are lots of ways to separate the elites from the non-elites. Elites are more likely to know a business executive, more likely to buy a new SUV, more likely to fly business class, more likely to attend professional sporting events (those tickets are expensive!), less likely to rent rather than their homes, less likely to ride public transportation, and so on. Murray’s quiz is interesting but he chooses to separate elites from non-elites in a particular way that makes me think he’s sensitive to the attitudes of politically liberal elites in particular.

Difficulties of the recommendation to “preach what you practice”

Murray does not consider the case of Joe Paterno, but in many ways the Penn State football coach fits his story well. Paterno was said to live an exemplary personal and professional life, combining traditional morality with football success—but, by his actions, he showed little concern about the morality of his players and coaches. At a professional level, Paterno rose higher and higher, and in his personal life he was a responsible adult. But he had an increasing disconnect with the real world, to the extent that horrible crimes were occurring nearby (in the physical and social senses) but he was completely insulated from the consequences for many years. Paterno’s story is symbolic of upper-income America: you can live an ordinary life in an ordinary house and still feel like a regular guy but still live in a bubble.

Paterno was a political conservative so he doesn’t quite match with Murray’s story, but he’s otherwise a good fit, a man who lived by a code of personal morality that he did not expect of others.

Joe Paterno is an extreme example, but I think his story is relevant, to explain the difficulty of the “preach what you practice” guideline. My claim is that “preaching,” to make a difference, requires actions as well as words. While Paterno did not espouse a nonjudgmental stance on rape, assault, etc., in his actions he expressed a hands-off policy. I see no reason to think that Paterno believed these crimes committed by his coach and players were OK, he just didn’t seem to think it was his role to do anything about it. I don’t place myself above Paterno in any moral sense—I certainly don’t monitor the after-hours activities of my own students and employees—I just see it as an example of the social distance that Murray writes about, that an authority figure such as Paterno can feel it’s acceptable to be so isolated in this way.

Murray’s argument is a step forward in sophistication compared to some other discussions of the culture war. Old-style conservatives such as Michael Barone have characterized upper-income liberals as being frivolous “trustfunders” who do “not to have to work very hard” and “have done nothing to earn their money,” slackers who “revel in looking down on” the common people.

in contrast, Murray tones down the Snidely Whiplash rhetoric and describes upper-class liberals as people who are living admirable lives but who are giving irresponsible advice because of their deluded social theories. His recommendation is, “When it comes to marriage and the work ethic, the new upper class must start preaching what it practices.”

The Paterno example illustrates the difficulty of this recommendation. What he had to do was not simply preach against rape and violence, but to act to stop it. Paterno was acting like the new upper class and simply looking away, allowing crimes to happen under his umbrella of protection. Unfortunately, this sort of behavior would seem to be characteristic of the old upper class as well, so I’m not sure how new this all is.

My point is that preaching values in a real way is not so easy; it requires hard work and direct involvement, not just talk. I don’t think Murray would disagree with me here. He writes that conscientious people should “voice their disapproval of those who defy these norms,” but it takes more than voicing disapproval. The kind of disapproval that makes a difference takes work and is risky. Joe Paterno could have reported the crimes of his coach and his students to the police, but at a possible cost to his reputation. Or, to choose a more homely example, just try telling an acquaintance that he or she is not conscientiously raising his or her kids. That won’t be a costless conversation to you! Again, it might be a good idea, but it’s hard to think about Murray’s suggestions without considering their challenges.

Upper-class liberals and upper-class conservatives

Setting aside the difficulties of implementing his recommendations, I see two limitations of Murray’s thesis. The first is a matter of selection. Let’s divide Americans into upper and lower income categories. (Murray just talks about whites, but I think the arguments apply to the general population; my guess is that after the reception of his Bell Curve book, Murray just thought it would be safest to leave race out of his discussions entirely.) Murray is comparing rich liberals to poor everybodys, but he just as well could be looking at rich conservatives. By focusing on the cultural contradictions of liberalism, Murray piques the attention of the liberal elite while lulling the conservative elite into a false sense of security. But I think he’s telling only part of the story, as I emphasized in graphs such as this:

My second problem with Murray’s argument is that it has a bit of a self-contradictory nature. As David Frum has noted, Murray criticizes upper-income Americans for (a) shunning lower-income cigarette smokers, but also for (b) not shaming lower-income people for poor life choices. But smoking is a poor life choice, no?

Elsewhere Murray states that upper-income Americans are more likely to go to church, and it seems that he would like these upper-class people to encourage churchgoing among the mass of Americans. But at another place he says that the elites themselves should try going to church, just like the common people do. So which is it: is churchgoing an admirable habit, along the lines of marriage and hard work, that the elites should encourage others to do, or is churchgoing a bit of homespun Americana, like watching football on TV and eating at Applebee’s, that the top 5% should reconnect with?

The point of these examples is not that Murray is wrong, either in his prescriptions or in his recommendations—much here depends on one’s economic views about taxation and government spending—but rather that his argument keeps going in two opposite directions at once. From one side he argues that the upper class has good habits that they should transmit to ordinary Americans; on the other side he says that the upper class should become more like the rest of the country. But I can’t see how you can have it both ways. This connects to my earlier point that much could be gained by considering the diversity of attitudes among the upper class.

Summary

This whole discussion got started because Murray was writing something about social class and David Frum and I fired back with statistics about income. But Murray is not writing about income; in fact, he explicitly states,

The new-upper-class culture is not the product of great wealth. It is enabled by affluence—people with common tastes and preferences need enough money to be able to congregate—but it is not driven by affluence. It is driven by the distinctive tastes and preferences that emerge when large numbers of cognitively talented people are enabled to live together in their own communities. You can whack the top income centile back to where it was in the 1980s, and it will have no effect whatsoever on the new-upper-class culture that had already emerged by that time.

I don’t know how true that is, but to be fair to Murray, he’s talking about cultural attitudes, not income. Based on my own interests, I’d take this the next step and consider the divisions between liberals and conservatives within America’s elites. My suggestions along those lines don’t contradict what Murray’s saying but rather represent additional things to think about.

What I’d really like to see is a graph with all 366 days of the year. It would be easy enough to make. That way we could put the Valentine’s and Halloween data in the context of other possible patterns. While they’re at it, they could also graph births by day of the week and show Thanksgiving, Easter, and other holidays that don’t have fixed dates. It’s so frustrating when people only show part of the story.

The data are publicly available, so maybe someone could make those graphs? If the Valentine’s/Halloween data are worth publishing, I think more comprehensive graphs should be publishable as well. I’d post them here, that’s for sure.

What the sophisticates thought in September 2008

Political opinions of U.S. military

The origin of essentialist reasoning

Corrected equation

This post is by Phil.

In the comments to an earlier post, I mentioned a problem I am struggling with right now. Several people mentioned having (and solving!) similar problems in the past, so this seems like a great way for me and a bunch of other blog readers to learn something. I will describe the problem, one or more of you will tell me how to solve it, and you will win…wait for it….my thanks, and the approval and admiration of your fellow blog readers, and a big thank-you in any publication that includes results from fitting the model.  You can’t ask fairer than that!

Here’s the problem.  The goal is to estimate six parameters that characterize the leakiness (or air-tightness) of a house with an attached garage.  We are specifically interested in the parameters that describe the connection between the house and the garage; this is of interest because of the effect on the air quality in the house  if there are toxic chemicals (gasoline, car exhaust, etc.) in the garage, but I won’t go into the motivation of the experiments, I’ll just describe them. (See below the fold for the rest)

A researcher puts a “blower door” — basically just a big fan — in the front door of the house. The fan is ramped up in speed, or is sped up in stages, so as to gradually pressurize the house relative to the outdoors.  The flow rate through the fan is measured, so you know how much air is going into the house.  That amount has to leak out of the house, either to the outdoors or into the garage.  The researcher measures the pressure difference between the house and the outdoors, and between the garage and the outdoors; of course, from these s/he can determine the pressure difference between the house and the garage.

All of the air that flows in (Q_{ho}, and don’t try to tell me it should be Q_{oh} because I know) has to flow out.  The first equation in the graphic is the conservation equation if you consider drawing a boundary around just the house: the air that flows into the house is the amount that flows directly to the outdoors plus the amount that flows into the garage.  P_ho and P_hg are the pressure differences between the house and the outdoors and between the house and the garage, which are measured. n_ho and n_hg are “flow exponents,” and C_ho and C_hg are “flow parameters”; these are to be estimated.  The flow exponents are expected to be between about 0.5 and 0.7. If a pressure is negative, then P^n is to be interpreted as sign(P)*abs(P)^n.

The second equation shows what happens if you draw look at the flows through the entire house-garage boundary. Again, everything that flows in has to flow out, either from the house to the outdoors or the garage to the outdoors. The flow from the garage to outdoors introduces two additional parameters.

So, finally, the problem.  I have a bunch of measurements of pressures (P_ij) and flows through the blower door (Q_ho).  If the model and data were perfect, there would be a unique set of C_ij and n_ij values such that Qho could be predicted from P_ho, P_hg, and P_go with no error.  But of course the data are not perfect. One of the main problems is that the pressure measurements can be systematically wrong: for example, if the outdoor pressure measurement is made on the lee side of the building, the house-outdoor pressure difference will tend to be overestimated.  Doing something simple like minimizing the RMS difference between predictions and measurements turns out to lead to systematic bias (e.g. n_ho is overestimated), in a sort of nonlinear analog to regression dilution. The obvious solution is to fit a statistical model that incorporates the error.  I think I could code this up myself, but I decided that this is a great opportunity to learn JAGS, which is similar to BUGS (which I have used before).  But I ran into trouble, perhaps from not fully understanding the JAGS or BUGS coding rules.

Here’s what I want:

1. The actual pressure difference between the house and outdoors P_{ho} is normally distributed about the measured pressure Pmeas_{ho}, with uncorrelated errors with standard deviation of 2 Pascals. Actually the mean error will not be zero as this implies, but for now let’s start this way. (In practice the blower door operator sets a desired pressure, and the blower door adjusts its flow automatically until the measured pressure matches the desired pressure).
2. The measured pressure difference between the garage and outdoors Pmeas_{go} is normally distributed about the actual pressure P_{go}, uncorrelated errors with s.d. of 2 Pa.
3. The error in the house-garage pressure difference is the difference between the P_{ho} error and the P_{go} error.
4. The measured flow Qmeas_{ho} is normally distributed about the actual Q_{ho} with error 20 cubic feet per minute.
5. The actual value of Q_{ho} that is predicted from the right side of either of the equations in the graphic has normal error with standard deviation 20 cubic feet per minute.

I run into some problems when trying to implement this in JAGS. I am tempted to post my JAGS model here — it’s only 20 lines — but I’m afraid of contaminating the mind of you, the reader, with bad ideas. (The code does not run).  I’ll post it in the comments tomorrow if people want it.

But here’s what I’d love: could one of you JAGS or BUGS experts help me out here?  There are a few issues that I don’t know how to handle, such as the fact that the error in Pmeas_hg is equal to the difference between the error in Pmeas_ho and Pmeas_go, and the fact that the model seems to need two equations that have Qmeas_{ho} on the left side but I think that that is not allowed.

Inputs are sets of Pmeas_{ho}[i], Pmeas_{go}[i], Qmeas_{ho}[i]; desired outputs are estimates (with uncertainty) of all three of the C_{ij} values and all three of the n_{ij} values, and of course, while we’re at it, estimates of the actual values P_{ho}[i], P_{go}[i], P_{hg}[i].

C’mon, twenty lines of JAGS or BUGS code, how hard can it be?

Larry Wasserman, “Low Assumptions, High Dimensions”:

This article was refreshing to me because it was so different from anything I’ve seen before. Larry works in a statistics department and I work in a statistics department but there’s so little overlap in what we do. Larry and I both work in high dimesions (maybe his dimensions are higher than mine, but a few thousand dimensions seems like a lot to me!), but there the similarity ends. His article is all about using few to no assumptions, while I use assumptions all the time. Here’s an example. Larry writes:

P. Laurie Davies (and his co-workers) have written several interesting papers where probability models, at least in the sense that we usually use them, are eliminated. Data are treated as deterministic. One then looks for adequate models rather than true models. His basic idea is that a distribution P is an ad- equate approximation for x1,…,xn, if typical data sets of size n, generated under P look like x1,…,xn. In other words, he asks whether we can approximate the deterministic data with a stochastic model.

This sounds cool. And it’s so different from my world! I do a lot of work with survey data, where the sample is intended to mimic the population, and a key step comes in the design, which is all about probability sampling. I agree that Wassserman’s (or Davies’s) approach could be applied to surveys—the key step would be to replace random sampling with quota sampling, and maybe this would be a good idea—but in the world of surveys we would typically think of quota sampling or other nonprobabilistic approaches as an unfortunate compromise with reality rather than as a desirable goal. In short, typical statisticians such as myself see probability modeling as a valuable tool that is central to applied statistics, while Wasserman appears to see probability as an example of an assumption to be avoided.

Just to be clear: I’m not at all saying Wasserman is wrong in any way here; rather, I’m just marveling on how different his perspective is from mine. I can’t immediately see how his assumption-free approach could possibly be used to estimate public opinion or votes cross-classified by demogtaphics, income, and state. But, then again, maybe my models wouldn’t work so well on the applications on which Wasserman works. Bridges from both directions would probably be good.

With different methods and different problems come different philosophies. My use of generative modeling motivates, and allows, me to check fit to data using predictive simulation. Wasserman’s quite different approach motivates him to understand his methods using other tools.

In an article from 1989, “Meta-analysis in medical research: Strong encouragement for higher quality in individual research efforts,” Keith O’Rourke and Allan Detsky apply this principle to the process of publication of scientific research:

From the statistical point of view, there really is no escape from performing a de facto meta-analysis. One can either judge the effectiveness of a therapy based solely on the most recent study and ignore all previous studies, a method which is equivalent to giving the most recent study weight 1.Oand all previous studies weight 0, or try to choose the weights on some scientific basis . . . If important differences in study findings exist they must be identified and explained.

That most researchers realize the need for stating their results in the context of previous trials is evidenced by the literature review section in almost all scientific articles. Meta-analysis is a further development and refinement of this approach offering a more rigorous and coherent treatment of past research work. It is tempting to propose that no experimental results should be published without inclusion of an appropriate meta-analysis. In effect, one might suggest that a literature review section ought to be based on an explicitly described methodology in place of the usual ad hoc approach.

So far, nothing exceptional. But then O’Rourke and Detsky continue:

What is it about meta-analysis that will actually help bring about improvement in individual research efforts? . . . the comprehensive, rigorous, and public peer review that a meta-analysis entails will encourage high quality participation by members of the research community in the resolution of the inadequacies. . . .

With a better understanding of meta-analysis in the context of the full scientific research process, meta-analysis is seen as a key element for improving individual research efforts and their reporting in the literature. This in turn will further enhance the role of meta-analysis in helping clinicians and policy makers answer clinical questions.

The idea (if I’m reading O’Rourke and Detsky correctly) is that, not only is meta-analysis appropriate for summarizing existing dat, also the threat or promise of meta-analysis provides an incentives for researchers to follow better practices in their new projects. If you know (or think there’s a high probability) that your work will be processed through a rigorous meta-analysis, this motivates you to be careful, to supply replication materials (otherwise your study will get a low weight), etc. That’s where the game theory comes in.

Rowan did not merely plagiarize to fill in some gaps or cover some technical material that he was too lazy to rewrite; rather, he put together an entire novel out of others’ material. Rowan writes (as part of a longer passage that itself appears to be dishonest; see the November 15, 2011 5:36 AM comment later on in the thread):

I [Rowan] sat there with the books [by others] on my kitchen table and typed the passages up word for word. I had a plot in mind, initially, and looked for passages that would work within that context. People told me the initial plot was dull (spies being killed all over Europe – no one knows why), so I changed it to be more like the premise of McCarry’s “Second Sight” which was a whole lot more interesting. I had certain things I wanted to see happen in the initial plot: a double cross, a drive through the South of France, a raid on a snowy satellite base. Eventually I found passages that adhered to these kinds of scenes that only meant changing the plot a little bit here and there. It felt very much like putting an elaborate puzzle together. Every new passage added has its own peculiar set of edges that had to find a way in.

The problem is not that he cut and pasted but that he didn’t acknowledge the sources. Although if he’d done that, he might’ve been up against some copyright infringement problems.

A commenter writes:

The whole thing about this that is so sad is that, yes, writing is hard work, and sending your words into the world to be read and judged is hard. But writing also brings me great joy and satisfaction. And that joy is what Quentin has cheated himself out of because he was scared.

I don’t know about that. Putting together an entire novel out of existing scraps and pieces—that’s pretty impressive to me. Quilting may be less technically impressive than weaving but it’s a skill all its own. Similarly, rappers have stolen lots of 70s riffs but they’ve added something of their own.

Literary vs. academic theft

The commenters also discuss other literary plagiarists such as Jacob Epstein, Patricia Waddell, Richard Condon, and Jerzy Kosinski.

Based on all these examples, literary plagiarism seems a bit different than academic plagiarism. (And both are different from journalists who take from blogs without giving credit.)

Goodwin, Fischer, Wegman, Tribe, Ayres, Dershowitz, etc etc etc are doing just fine in their careers. They don’t need to plagiarize; they seem to do it out of a sense of obligation, or because they’re too lazy to figure things out themselves. (Or maybe for one of these reasons.) It’s less effort to copy than to fully read external material, incorporate it into one’s worldview, and rewrite it in a way that is coherent with one’s larger argument.

In contrast, for literary plagiarists there is skill involved in patching together complementary material from others’ published work, seeking passages that are obscure enough or bland enough to escape notice. I don’t think that even Ed Wegman’s strongest defenders would argue that he applied any wit or creativity in his ripoffs of Wikipedia and other published sources. But I think you can admire the skill (if not the dishonesty) of a literary copyist who can put together a whole novel out of others’ material.

Getting energy from the reader

I also liked this comment, later on in the thread:

Books contain energy, and when you purposefully use words found in other books, you pull that energy into your own work.

I [the commenter] think the energy comes from the author, and all her experiences and deliberations, but also it comes from the people working on the book: editors, artists, marketing, etc. I’d go so far as to say that people reading and responding to books can contribute to their energy as well.

Interesting point. I believe the reader can supply a lot, and some books stimulate this by having lots of hooks, as it were, to connect to the readers’ thoughts and experiences. Think of all those memoirs that work by reminding readers of their own childhoods. Or consider Nassim Taleb’s books. Many of my correspondents were surprised that I responded so positively to Fooled by Randomness and The Black Swan, but I really enjoyed the experience of reading them with pen in hand, it was just the right book to bring out lots of thoughts that I had within myself.

A link between intellect and temperament has long been the subject of speculation. . . . Studies of the artistically inclined report linkage with familial depression, while among eminent and creative scientists, a lower incidence of affective disorders is found. In the case of developmental disorders, a heightened prevalence of autism spectrum disorders (ASDs) has been found in the families of mathematicians, physicists, and engineers. . . .

We surveyed the incoming class of 2014 at Princeton University about their intended academic major, familial incidence of neuropsychiatric disorders, and demographic variables. . . . Consistent with prior findings, we noticed a relation between intended academic majors and ASDs. Looking for relations between other neuropsychiatric disorders and academic interest we also noted a heightened prevalence of bipolar disorder, major depressive disorder and substance abuse in the families of those pursuing the humanities. A composite score based on these four heritable disorders was strongly correlated with a student’s intended academic major. Thus, familial risk toward a spectrum of psychopathologies can predict propensity toward technical versus humanist interests.

When I spoke with Sam last year we discussed various ways to analyze the data as well as various interpretations of the results, but I don’t actually remember any of our conversation except for the bit where he described to me how they conducted their study.

David Frum quotes Murray as writing, in an echo of now-forgotten TV personality Tucker Carlson, that the top 5% of incomes “tends to be liberal—right? There’s no getting around it. Every way of answering this question produces a yes.”

[I’ve interjected a “perhaps” into the title of this blog post to indicate that I don’t have the exact Murray quote here so I’m relying on David Frum’s interpretation.]

Frum does me the favor of citing Red State Blue State as evidence, and I’d like to back this up with some graphs.

Frum writes:

Say “top 5%” to Murray, and his imagination conjures up everything he dislikes: coastal liberals listening to NPR in their Lexus hybrid SUVs. He sees that image so intensely that no mere number can force him to remember that the top 5% also includes the evangelical Christian assistant coach of a state university football team. . . .

To put it in graphical terms:

In blue America (where Charles Murray, David Brooks, David Frum, and I live), rich people are a bit more economically conservative then poor people but this is balanced by rich people being more socially liberal. As a result, rich and poor have similar voting patterns in the blue states.

In contrast, in red America (where Charles Murray and David Brooks locate the forgotten majority), rich people are both economically and socially conservative (at least they were in 2000, which is when the data for this graph came from).

America’s top 5% of income includes rich households in red, purple, and blue America—and, as the graph above shows, this represents a large variation in political views.

Another way to put it is that, as we say in our book, the culture war is not a battle between rich liberals and poor (or middle-class) conservatives or even a battle between rich conservatives and lower-income liberals. Rather, the culture war is between rich liberals and rich conservatives.

It’s not the Prius vs. the pickup truck, it’s the Prius vs. the Hummer.

There are more rich conservatives than rich liberals (just as there are more rich Republican voters than rich Democratic voters) but the minority of upper-income Americans who are rich do play an important role in our society. It’s worth spending some time thinking about rich liberals, but it’s also worth remembering that most rich Americans are not liberal.

I think Charles Murray is interested in religion too, so let me throw in this set of graphs that subsets the population according to religious attendance:

Just to hack at this a little more: data from 2000, 2004, and 2008 showing the income distribution of voters self-classified by ideology (liberal, moderate, or conservative) and party identification (Democrat, Independent, or Republican):

There are more rich conservatives than rich liberals.

And here are some maps of how different ethnic groups voted in 2008 (click for the large version):

The second row of maps gives the answer to your questions about white America, or at least those white Americans who vote. (The evidence is that, compared to voters, nonvoters have lower income and are more likely to favor income redistribution. So I don’t think that moving from “white voters” to “white America” would change our story.)

A paper by Wood, Douglas, and Sutton looks at “Beliefs in Contradictory Conspiracy Theories.”  Unfortunately the  subjects were 140 undergraduate psychology students, so one wonders how general the results are.  I found this sort of arresting:

In Study 1 (n=137), the more participants believed that Princess Diana faked her own death, the more they believed she was murdered.  In Study 2 (n=102), the more participants believed that Osama Bin Laden was already dead when U.S. Special Forces raided his compound in Pakistan, the more they believed he is still alive.

As the article says, “conspiracy advocates’ distrust of official narratives may be so strong that many alternative theories are simultaneously endorsed in spite of any contradictions between them.”  But I think the authors overstate things when they say “One would think that there ought to be a negative correlation between beliefs in contradictory accounts of events — the more one believes in a particular theory, the less likely rival theories will seem.”  Well, one might think that, but actually a positive correlation makes sense to me.  I can see how, if you really think that a lot of what the government says is a lie, you would think “well, I don’t know exactly which part of the Bin Laden account is a lie but they are probably lying about something; maybe he was already dead, or maybe he’s still alive now, but I don’t know which.”  The authors realize this is what is going on, they just make too much of how surprising it should be.

David Hendry, “Empirical Economic Model Discovery and Theory Evaluation”:

Hendry presents a wide-ranging overview of scientific learning, with an interesting comparison of physical with social sciences. (For some reason, he discusses many physical sciences but restricts his social-science examples to economics and psychology.)

The only part of Hendry’s long and interesting article that I will discuss, however, is the part where he decides to take a gratuitous swing at Bayes. I don’t know why he did this, but maybe it’s part of some fraternity initiation thing, like TP-ing the dean’s house on Halloween.

Here’s the story. Hendry writes:

‘Prior distributions’ widely used in Bayesian analyses, whether subjective or ‘objective’, cannot be formed in such a setting either, absent a falsely assumed crystal ball. Rather, imposing a prior distribution that is consistent with an assumed model when breaks are not included is a recipe for a bad analysis in macroeconomics. Fortunately, priors are neither necessary nor sufficient in the context of discovery.

I could just laugh this off—but as someone who has published two books and hundreds of articles on applied Bayesian statistics, I think I’ll take Hendry seriously.

Let me start with the tone. I generally don’t like when people take words or phrases that you disagree with them and put them in quotes. If you’re going to put “prior distributions” and “objective” in quotes, then please show the same disrespect to your other terms: “falsely” . . . “crystal ball” . . . “breaks” . . . “recipe” . . . “macroeconomics” . . . “discovery.”

But let me get to the substance. First, Hendry’s right. No statistical method is necessary. With sufficient effort, I think you can solve all statistical problems with Bayesian methods, or with robust methods, or with bootstrapping, or with any number of alternative approaches. Fuzzy sets would probably work too. Different approaches have different advantages, but I’m sure that if Hendry adopts a self-denying ordinance and decides to never use priors, he can solve all sorts of data analysis problems. He’ll just have to work really hard sometimes. But, to be fair, there are some problems that I have to work really hard on too. In short: econometrics methods tend to require more effort in complicated settings, but they often have appealing robustness properties. It’s fair enough that Hendry and I place different values on robustness vs. modeling flexibility.

My most serious criticism with Hendry’s above paragraph is the old, old story: he’s singling out Bayesian methods and priors as being particularly bad. Meanwhile all those likelihood functions and assumptions of additivity, symmetry, etc. all just sneak in. Hendry’s standing at the back window with a shotgun, scanning for priors coming over the hill, while a million assumptions just walk right into his house through the front door.

Here’s Hendry’s summary:

The pre-existing framework of ideas is bound to structure any analysis for better or worse, but being neither necessary nor sufficient, often blocking, and unhelpful in a changing world, prior distributions should play a minimal role in data analyses that seek to discover useful knowledge.

I’m going to have to disagree. I could give a million examples of useful knowledge that can be discovered with the aid of prior distributions. For example, where are the houses in the U.S. that have high radon levels? What are the effects of redistricting? How much perchloroethylene does the body metabolize? What is public opinion on gay rights by state? Or, for a classic from Mosteller and Wallace in 1960, classify the authorship of the Federalist Papers using 1960s technology.

I’m not saying that Hendry and his colleagues need to be using Bayesian methods in his applied research. I’m not even saying that Bayesian methods are needed to solve the problems listed in the above paragraph. In practice these problems were indeed solved using Bayesian inference, but I think other approaches could get there too. What I am saying is, why is Hendry so sure that “prior distributions should play a minimal role” etc.? I’m really bothered when people go beyond the simple and direct, “I have no personal experience with Bayesian inference solving a useful problem” to prescriptive (and wrong) statements such as “prior distributions should play a minimal role.” And it’s just silly to say that priors are “unhelpful in a changing world.” I’d think an econometrician would know about time series models!

Hendry also pulls the no-true-Scotsman trick:

Fortunately, priors are neither necessary nor sufficient in the context of discovery. For example, children learn whatever native tongue is prevalent around them, be it Chinese, Arabic or English, for none of which could they have a ‘prior’. Rather, trial-and-error learning seems a child’s main approach to language acquisition: see Clark and Clark (1977). Certainly, a general language system seems to be hard wired in the human brain (see Pinker 1994; 2002) but that hardly constitutes a prior. Thus, in one of the most complicated tasks imaginable, which computers still struggle to emulate, priors are not needed.

This is a no-true-Scotsman argument because, when confronted with an example in which our brains figure things out using a pre-existing structure (not for Chinese, Arabic, or English, but for human language in general), Hendry simply says that this system that is “hard wired in the human brain . . . hardly constitutes a prior.” Huh? It’s definitely a prior. That’s the whole point: our brains are tuned to decode human language.

Why does this bug me so much about a few throwaway paragraphs in an otherwise-pretty-good-article? Hendry’s anti-Bayesian sentiments are no more clueless than those earlier expressed by, say, John DiNardo. The difference is that DiNardo was just venting his opinions and was pretty open about this, whereas Hendry’s presenting his prejudices with an air of expertise. If Hendry wants to work on “replacing unrestricted non-linear functions by an encompassing theory-derived form, such as an ogive,” then fine. His theoretical models of model selection seem interesting and could perhaps be useful. I just wish he’d cut out the part where he implicitly disparages the work of Mosteller and Wallace, Lax and Phillips, and a few zillion other researchers who’ve used Bayesian methods to solve problems.

It’s not too late for Hendry to reform (I hope). All he needs to do is to retreat to present the positive virtues of his preferred inferential approach along with his explanations as to why Bayesian methods have not seemed useful for him. He’s an econometrician, he doesn’t work in toxicology and that’s fine. I think both his positive and his negative statements would be stronger if he would be more aware of the limits of his own experience. Just as, in mathematics, a theorem is clearer if you understand the range of its applicability and the areas where there are counterexamples.

In applications, statistical models are often restricted to what produces reasonable estimates based on the data at hand. In many cases, however, the principles that allow a model to be restricted can be derived theoretically, in the absence of any data and with minimal applied context. We illustrate this point with three well-known theoretical examples from spatial statistics and time series. First, we show that an autoregressive model for local averages violates a principle of invariance under scaling. Second, we show how the Bayesian estimate of a strictly-increasing time series, using a uniform prior distribution, depends on the scale of estimation. Third, we interpret local smoothing of spatial lattice data as Bayesian estimation and show why uniform local smoothing does not make sense. In various forms, the results presented here have been derived in previous work; our contribution is to draw out some principles that can be derived theoretically, even though in the past they may have been presented in detail in the context of specific examples.

I just love this paper. But it’s only been cited 17 times (and four of those were by me), so I must have done something wrong. In retrospect I think it would’ve made more sense to write it as three separate papers; then each might have had its own impact. In any case, I hope the article provides some enjoyment and insight to those of you who click through.

I wrote that I’m not a big fan of subjective priors. Various commenters had difficulty with this point, and I think the issue was most clearly stated by Bill Jeffreerys, who wrote:

It seems to me that your prior has to reflect your subjective information before you look at the data. How can it not?

But this does not mean that the (subjective) prior that you choose is irrefutable; Surely a prior that reflects prior information just does not have to be inconsistent with that information. But that still leaves a range of priors that are consistent with it, the sort of priors that one would use in a sensitivity analysis, for example.

I think I see what Bill is getting at. A prior represents your subjective belief, or some approximation to your subjective belief, even if it’s not perfect. That sounds reasonable but I don’t think it works. Or, at least, it often doesn’t work.

Let’s start with a simple example. You hop on a scale that gives unbiased measurements with errors that have a standard deviation of 0.1 kg. To do Bayesian analysis, you assign a N(0,10000^2) prior on your true weight. That doesn’t represent your subjective belief! It’s not even an approximation. No problem—it works fine for most purposes—but it’s not subjective.

More generally, think of all the linear and logistic regressions we use. Instead of thinking of these as subjective beliefs, I prefer to think of the joint probability distribution as a model, reflecting a set of assumptions. In some settings these assumptions represent subjective beliefs, in other settings they don’t.

This article from 2002 might help. If I could go back and alter it, I’d add something on weakly informative priors, but I still agree with the general approach discussed there.

P.S. Just to give an example of what I mean by prior information: The analyses in Red State Blue State all use noninformative prior distributions. But a lot of prior information comes in, in the selection of what questions to study, what models to consider, and what variables to include in the model. For example, as state-level predictors we include region of the country, Republican vote in the previous presidential election, and average state income. Prior information goes into the choice and construction of all these predictors. But the prior distribution is a particular probability distribution that in this case is flat and does not reflect prior knowledge.

One way to think about informative prior distributions is as a form of smoothing: when setting the parameters of a probability distribution based on prior knowledge, we are imposing some time smoothness on the parameters. I think that’s probably a good idea and that the Red State Blue State analyses (among others) would be better for it. I didn’t set up this prior structure because I wasn’t easily equipped to do so and it seemed like too much effort, but perhaps at some future time this sort of structuring will be as commonplace as hierarchical modeling is today.

Steps include “Make whimsical sparkles by drawing an ellipse using the Ellipse Tool,” “Rotate the sparkles . . . Give some sparkles less Opacity by using the Transparency Palette,” and “Add a haze around each sparkle by drawing a white ellipse using the Ellipse Tool.”

The punchline:

Now, the next time you need to include a boring graph in one of your designs you’ll be able to add some extra emphasis and get people to really pay attention to those numbers!

P.S. to all the commenters: Yeah, yeah, do your contrarian best and tell me why chartjunk is actually a good thing, how I’m just a snob, etc etc.

I passed this on to urbanist Richard Florida and received the following response:

This is a tough one with lots of causality issues. Generally speaking universities make places stronger. But this is mainly the case for smaller, college towws. Boulder, Ann Arbor and so on, which also have very high human capital levels and high levels of creative, knowledge and professional workers.

For big cities the issue is mixed. Take Pittsburgh with CMU and Pitt or Baltimore with Hopkins, or St Louis. The list goes on and on.

Kevin Stolarick and I framed this very crudely as a transmitter reciever issue. The university in a city like this can generate a lot of signal, in terms of innovation or even human capital and the city may not receive it or push it away. A long ago paper by Mike Fogarty showed how innovations in Pittsburgh and Cleveland, by universities in these communities, tended to be picked up in Silicon Valley or even Tokyo.

I responded: Another factor in the interaction is: how good does the university have to be? Glaeser cited UW and Seattle, but that’s kind of a funny example, because I don’t think UW was such a great university 30 years ago. On the other hand, given the existence of Boeing and Microsoft, UW is good enough to do the job of providing a center for the creative class. Perhaps Ohio State (another good but not great university) has played a similar role in Columbus.

Florida replied:

Better is better. I think both are over threshold, but having taught at OSU at the very beginning of my career, it brings both plusses and minuses. It was an open admission school. The faculty was very, very mixed. And a huge football factory. Gates and Allen amongothers have pumped big wads of cash into UW, and it is good in computers and biosciences.

Both strike me as regional talent hubs, which probably trumps university quality.

Portland is another outlier with lots of talent/ human capital attraction and pretty crappy universities.

Florida also sent along this article and this blog.

Also, Hal Varian wrote:

There is a literature that attempts to assess the impact of university research on the local economy. One person I know well who works in this area is Marie Thursby. Click on her vitae to see the kind of work that has been done. This is pretty careful research, though of course it is hard to pin down causality…

P.S. Originally I wrote that UW is not such a great university, which may or may not be true but is sort of beside the point since the real issue is whether UW’s past greatness contributed to Seattle’s current prosperity. So I clarified that I’m really talking about UW thirty or so years ago.

Stephen Senn, “You May Believe You Are a Bayesian But You Are Probably Wrong”:

I agree with Senn’s comments on the impossibility of the de Finetti subjective Bayesian approach. As I wrote in 2008, if you could really construct a subjective prior you believe in, why not just look at the data and write down your subjective posterior. The immense practical difficulties with any serious system of inference render it absurd to think that it would be possible to just write down a probability distribution to represent uncertainty. I wish, however, that Senn would recognize my Bayesian approach (which is also that of John Carlin, Hal Stern, Don Rubin, and, I believe, others). De Finetti is no longer around, but we are!

I have to admit that my own Bayesian views and practices have changed. In particular, I resonate with Senn’s point that conventional flat priors miss a lot and that Bayesian inference can work better when real prior information is used. Here I’m not talking about a subjective prior that is meant to express a personal belief but rather a distribution that represents a summary of prior scientific knowledge. Such an expression can only be approximate (as, indeed, assumptions such as logistic regressions, additive treatment effects, and all the rest, are only approximations too), and I agree with Senn that it would be rash to let philosophical foundations be a justification for using Bayesian methods. Rather, my work on the philosophy of statistics is intended to demonstrate how Bayesian inference can fit into a falsificationist philosophy that I am comfortable with on general grounds.

In practice, I think we use confidence intervals and hypothesis tests as a way to avoid acknowledging uncertainty. We set up some rules and then act as if we know what is real and what is not. Even in my own applied work, I’ve often enough presented 95% intervals and gone on from there. But maybe that’s just not right.

I was thinking about this after receiving the following email from a psychology student:

I [the student] am trying to conceptualize the lessons in your paper with Stern with comparing treatment effects across studies. When trying to understand if a certain intervention works, we must look at what the literature says. However this can be complicated if the literature has divergent results. There are four situations I am thinking of. FOr each of these situations, assume the studies are randomized control designs with the same treatment and outcome measures, and each situation refers to a different treatment. It is easiest for me to put it into a table. In each of these situations only 1 of 2 published studies is found to be statistically significant.

	Effect	se	Sig	Sig in diff	Result
Situation 1
Study A	.5	.05	Y	X	Treatment is effective
Study B	.4	.2
Situation 2
Study C	.5	.1	Y	Y	Unclear, needs more replications
Study D	.1	.1
Situation 3
Study E	.41	.2	Y	X	Unclear, needs more replications
Study F	.14	.2
Situation 4
Study G	.7	.3	Y	X	Null/needs more replications
Study H	.19	.1

Here, Situation 1 refers to 2 studies that have similar effects in magnitude, though the larger of the 2 studies (smaller se) is the only sig one. SInce the difference between the two effects is itself, not statistically significant, we should conclude treatment in situation 1 is effective (this seems to be in line with your paper).
In situation 2 there are 2 equally sized experiments that differ in treatment effect and significance. Since the difference between the estimates is statistically significant, one concludes the paradigm needs more replications.
In situation 3 the 2 studies have 2 effects, one is statistically significant while the other is not. However in this situation study F is neither statistically nor substantively significant. Unlike situation 1 it would seem unwise to conclude Treatment in situation 3 is effective and we need more replications.
Situation 4 is just some result I cam across in a research synthesis, where a smaller study (larger se) had a statistically sig effect, but a larger one did not. It would seem in this situation the true effect is null and the stat sig effect is a type 1 error. However the difference between studies is not stat sig, would this matter?

I replied that my quick reaction is that it would be better if there were data from more studies. With only two studies, your inference will necessarily depend on your prior information about effectiveness and variation of the treatments.

The student then wrote:

That is my reaction as well. Unfortunately sometimes the only data we have is from a small number of studies, and not enough to necessarily run a meta-analysis on. In addition, the hypothetical situations I sent you are sometimes all we know about the effectiveness and variation in treatments, because it is all the evidence we have. What I am trying to better understand is if your paper is addressing situation 1 ONLY, or if it is making inferences or statements about the evidence in the other situations I presented.

To which I replied that I don’t know that our paper gives any real recommendations. In a decision problem, I think ultimately it’s necessary to bite the bullet and decide what prior information you have on effectiveness rather than relying on statistical significance.

This is a problem under classical or Bayesian methods. Either way, it’s standard practice to summarize uncertainty in a way that encourages deterministic thinking.

My contribution is called Induction and Deduction in Bayesian Data Analysis. I’ll also post my reactions to the other articles. I wrote these notes a few weeks ago and could post them all at once, but I think it will be easier if I post my reactions to each article separately.

To start with my best material, here’s my reaction to David Cox and Deborah Mayo, “A Statistical Scientist Meets a Philosopher of Science.” I recommend you read all the way through my long note below; there’s good stuff throughout:

1. Cox: “[Philosophy] forces us to say what it is that we really want to know when we analyze a situation statistically.”

This reminds me of a standard question that Don Rubin (who, unlike me, has little use for philosophy in his research) asks in virtually any situation: “What would you do if you had all the data?” For me, that “what would you do” question is one of the universal solvents of statistics.

2. Mayo defines scientific objectivity as concerning “the goal of using data to distinguish correct from incorrect claims about the world” and contrasts this with so-called objective Bayesian statistics. All I can say here is that the terms “subjective” and “objective” seem way overloaded at this point. To me, science is objective in that it aims for reproducible findings that exist independent of the observer, and it’s subjective in that the process of science involves many individual choices. And I think the statistics I do (mostly, but not always, using Bayesian methods) is both objective and subjective in that way.

3. Cox discusses Fisher’s rule that it’s ok to use prior information in design of data collection but not in data analysis. Like a lot of hundred-year-old ideas, this rule makes sense in some contexts but not in others. Consider the notorious study in which a random sample of a few thousand people was analyzed, and it was found that the most beautiful parents were 8 percentage points more likely to have girls, compared to less attractive parents. The result was statistically significant (p<.05) and published in a reputable journal. But in this case we have good prior information suggesting that the difference in sex ratios in the population, comparing beautiful to less-beautiful parents, is less than 1 percentage point. A classical design analysis reveals that, with this level of true difference, any statistically-significant oberved difference in the sample is likely to be noise. (Even conditional on statistical significance, the observed difference has an over 40% chance of being in the wrong direction and will overestimate the population difference by an order of magnitude.) At this point, you might well say that the original analysis should never have been done at all---but, given that it has been done, it is essential to use prior information to interpret the data and generalize from sample to population.

Where did Fisher’s principle go wrong here? The answer is simple—and I think Cox would agree with me here. We’re in a setting where the prior information is much stronger than the data. If one’s only goal is to summarize the data, then taking the difference of 8% (along with a confidence interval and even a p-value) is fine. But if you want to generalize to the population—which was indeed the goal of the researcher in this example—then it makes no sense to stop there.

Cox illustrates the difficulty in a later quote: “[Bayesians'] conceptual theories are trying to do two entirely different things. One is trying to extract information from the data, while the other, personalistic theory, is trying to indicate what you should believe, with regard to information from the data and other, prior, information treated equally seriously. These are two very different things.”

Yes, but Cox is missing something important! He defines two goals:
(a) Extracting information from the data.
(b) A “personalistic theory” of “what you should believe.”
I’m talking about something in between, which is inference for the population. I think Laplace would understand what I’m talking about here. The sample is (typically) of no interest in itself, it’s just a means to learning about the population. But my inferences about the population aren’t “personalistic”—at least, no more than the dudes at CERN are personalistic when they’re trying to learn about particle theory from cyclotron experiments, and no more than the Census and the Bureau of Labor Statistics are personalistic when they’re trying to learn about the U.S. economy from sample data.

4. Cox: “There are situations where it is very clear that whatever a scientist or statistician might do privately in looking at data, when they present their information to the public or government department or whatever, they should absolutely not use prior information, because the prior opinions on some of these prickly issues of public policy can often be highly contentious with different people with strong and very conflicting views.”

Maybe. But I don’t think Cox even believes this statement himself if it were taken literally. For example, right now I’m working on the politically controversial problem of reconstructing historical climate from tree rings. We have a lot of prior information on the processes under which tree rings grow and how they are measured. I don’t think anyone would want to just take raw numbers from core samples as a climate estimate! All the tools from Statistical Methods for Research Workers won’t take you from tree rings to temperature estimates. You need some scientific knowledge and prior information on where these measurements came from.

So let me interpret what I think Cox was saying. I take him to be dividing any scientific inference into two parts, inside and outside. Priors are allowed in the inside work of scientific modeling, which uses lots of external information, from the basic assumptions that the data correspond to your scientific goals, through the mathematical form of the transfer function, down to details such as an assumption of normally-distributed measurement errors, which might be supported based on prior experimental evidence. But Cox would prefer to avoid priors in the outside problem. In my example, I assume he’d allow prior information on the tree-ring measurement process—I don’t see how you can get anywhere otherwise—but he’d rather not combine with external estimates of the temperature series. That’s a tenable position. It doesn’t avoid all the controversy—manipulations of the data model can map in predictable ways to changes in the final inferences—but it could make sense.

I’ve followed this approach in much of my own applied work, using noninformative priors and carefully avoiding the use of prior information in the final stages a statistical analysis. But that can’t always be the right choice. Sometimes (as in the sex ratio example above), the data are just too weak—and a classical textbook data analysis can be misleading. Imagine a Venn diagram, where one circle is “Topics that are so controversial that we want to avoid using prior information in the statistical analysis” and the other circle is “Problems where the data are weak compared to prior information.” If you’re in the intersection of these circles, you have to make some tough choices!

More generally, there is a Bayesian solution to the problem of sensitivity to prior assumptions. That solution is sensitivity analysis: perform several analyses using different reasonable priors. Make more explicit the mapping from prior and data to conclusions. Be open about sensitivity, don’t try to sweep the problem under the rug, etc etc. And, if you’re going that route, I’d also like to see some analysis of sensitivity to assumptions that are not conventionally classified as “prior.” You know, those assumptions that get thrown in because they’re what everybody does. For example, Cox regression is great, but additivity is a prior assumption too! (One might argue that assumptions such as additivity, logistic links, etc., are exempt from Fisher’s strictures by virtue of being default assumptions rather than being based on prior information—but I certainly don’t think Mayo would take that position, given her strong feelings on Bayesian default priors.)

My point here is that all statistical methods require choices—assumptions, if you will. Not all your choices can be determined or even validated from the data at hand. If you don’t want your choices to be based on prior information, what other options do you have? You can rely on convention—using methods that appear in major textbooks and have stood the test of time—or maybe on theory. Both these meta-foundational approaches have their virtues but neither is perfect: Conventional methods are not necessarily good (as can be seen by noting that for many problems there are multiple conventional methods that give different results), and theory often doesn’t help (for example classical confidence intervals and hypothesis tests are insufficient in the simple sex-ratio problem noted above).

The answer does seem to be that the academic world, like other worlds, is run by the politicians, and sensitively scrupulous people tend to leave politics to other people, while people with genuine work to do certainly have no time as well as no taste for committee-rigging and the associated techniques. And then of course there are the forces of native stupidity reinforced by that blind hostility to criticism, reform, new ideas and superior ability which is human as well as academic nature.

Not that I’ve ever read anything by Mrs. Leavis (or, as the Brits used to write, Mrs Leavis). The above quote is one of the epigraphs to a book by Richard Kostelanetz. Whom I’ve never heard of, except in a footnote in John Rodden’s classic Orwell study, The Politics of Literary Reputation.

I’ll have more to say about Orwell in another post, but for now let me return to the above Leavis quote, to which I have three reactions:

1. On a personal level, I’m on Leavis’s side. I’d much rather work (or blog, which I feel is related to my work and is also a public service) than spend time on academic politics: forming coalitions, doing the pre-meeting meetings, trading favors, kissing up and kicking down, and all the rest.

To put it another way, I don’t like political games because (a) I’m not good at manipulation and deception, and (b) Much of politics is zero-sum, and I prefer to collaborate in positive-sum activities such as writing Stan.

2. But on a more practical level, somebody needs to do the dirty work. Every once in awhile. I’ve encountered some administrators who are good at “committee-rigging,” etc., and others who show less political ability. I’ve seem people use political processes in a pointless destructive way—power for the sake of power—but others can use their political skills to foster smooth cooperation.

To put it another way, I require the political efforts of others to create the safe space I need to do my work. And it’s a special bonus when these political efforts are not “reinforced by that blind hostility to criticism, reform, new ideas and superior ability.”

3. As a political scientist, I recognize that politics is necessary. There’s no such thing as a non-political process. Politics is how we fight against entropy. Whatever non-politicized zones we have in life are often the result of continued political effort. As the saying goes, the price of liberty is eternal vigilance.

Ultimately I’ll have to go with #3.

Image via Wikipedia

Robert Zubrin writes in “How Much Is an Astronaut’s Life Worth?” (Reason, Feb 2012):

…policy analyst John D. Graham and his colleagues at the Harvard Center for Risk Analysis found in 1997 that the median cost for lifesaving expenditures and regulations by the U.S. government in the health care, residential, transportation, and occupational areas ranges from about $1 million to $3 million spent per life saved in today’s dollars. The only marked exception to this pattern occurs in the area of environmental health protection (such as the Superfund program) which costs about $200 million per life saved.

Graham and his colleagues call the latter kind of inefficiency “statistical murder,” since thousands of additional lives could be saved each year if the money were used more cost-effectively. To avoid such deadly waste, the Department of Transportation has a policy of rejecting any proposed safety expenditure that costs more than $3 million per life saved. That ceiling therefore may be taken as a high-end estimate for the value of an American’s life as defined by the U.S. government.

This reminds me of my old article on Value of Life – where the hidden cost of the Iraq war for the US comes to 720,000 lives lost (based on the huge cost).

In their article they employ a charming bit of economics jargon, using the word “earn” to mean “how much money you make.” They “propose an automatic extra tax on the income of the top 1 percent of earners.” I assume their tax would apply to unearned income as well, but they (or their editor at the Times) are just so used to describing income as “earnings” that they just threw that in. Funny.

Also, there’s a part of the article that doesn’t make sense to me.

Ayres and Edlin first describe the level of inequality:

In 1980 the average 1-percenter made 12.5 times the median income, but in 2006 (the latest year for which data is available) the average income of our richest 1 percent was a whopping 36 times greater than that of the median household.

Then they lay out their solution:

Enough is enough. . . . we propose an automatic extra tax on the income of the top 1 percent of earners — a tax that would limit the after-tax incomes of this club to 36 times the median household income.

This seems fair enough to me, but one thing that puzzles me is: my impression is that Ayres and Edlin feel that the rich have too much as it is already? So why freeze inequality at the current rate? (Yes, inequality could decline, but if it’s on an inexorable upward trend, my quick guess would be that maxing this ratio at 36 would be nearly equivalent to setting the ratio to 36.) Given the U.S. budget crisis, why 36? Why not 30, or 20, or 15?

P.S. When we last heard from Ayres he was supplying advice for young people who were rich or expecting to be rich. So I think it’s fair to say he’s no class warrior, that he’d like to keep income inequality at the current level but no lower.

And please note that I’m neither endorsing the Ayres/Edlin plan nor criticizing it. (Given my lack of expertise in macroeconomics, I’m certainly not the one you’d go running to, asking for an informed opinion on a proposed tax plan.) I’m just asking a question.

P.S. And here’s how to do it!

Conflict of interest disclaimer: I was paid by Google last year to give a short course.

I’m reading your Multilevel modeling book and am trying to apply it to my work. I’m concerned with how to estimate a random intercept model if there are hundreds/thousands of levels. In the Gibbs sampling, am I sampling a parameter for each level? Or, just the hyper-parameters? In other words, say I had 500 zipcode intercepts modeled as ~ N(m,s). Would my posterior be two dimensional, sampling for “m” and “s,” or would it have 502 dimensions?

My reply: Indeed you will have hundreds or thousands of parameters—or, in classical terms, hundreds or thousands of predictive quantities. But that’s ok. Even if none of those predictions is precise, you’re learning about the model.

See page 526 of the book for more discussion of the number of parameters in a multilevel model.

Reilly and Zeringue then present their analysis. Their simple little predator-prey model with a weakly informative prior way outperforms the standard big-ass autoregression models. Check this out:

Or, to put it into numbers, when they fit their model to the first 80 years and predict to the next 34, their root mean square out-of-sample error is 1480 (see scale of data above). In contrast, the standard model fit to these data (the SETAR model of Tong, 1990) has more than twice as many parameters but gets a worse-performing root mean square error of 1600, even when that model is fit to the entire dataset. (If you fit the SETAR or any similar autoregressive model to the first 80 years and use it to predict the next 34, the predictions are a disaster—the predicted values quickly go toward the mean and can’t even attempt to track the curve.)

As Reilly and Zeringue note, the above graph shows potential room for improvement in the model, but even as is, it shows the huge benefits that can be obtained by attempting to model the underlying process rather than simply fitting the data using a conventional family of models.

(It’s funny for me to emphasize this point, given how often I use conventional models such as linear and logistic regression.)

P.S. The title and text above have been modified to reflect comments below with reference to models fit to the lynx data in the ecology literature. There appears to be not enough communication between ecologists and statisticians. The statistical point above still holds—a simple model with some reasonable structure can outperform a generic data-fitting model such as an autoregression—but you should probably check out some of the references given in the comments if you’re interested in the lynx example or ecology models more generally.

I have to admit that I have a bit of a monocultural temperament myself. I have strong feelings about the right and wrong way to do things, and I don’t have much patience for what seems to me to be the wrong way. As a result, I’ve often disparaged or ignored important statistical developments because some small aspect of the new idea didn’t fit with my thinking. (On the plus side, I think I’ve disparaged or ignored lots more bad ideas thad deserve oblivion.)

I’ve always been suspicious of the hedgehog/fox distinction because my impression is that just about everybody likes to think of him or herself as a fox. Being a hedgehog is like being “ideological”; most of us like to think of ourselves as pragmatic foxes. And in any case I think most statisticians are foxes.

One of the many positive outcomes of my mugging at Berkeley was a commitment to pluralism (for example, see here).

Beyond this, I move away from my natural monocultural instincts by teaching classes that include material I wouldn’t otherwise cover, by listening carefully to people I respect who do things in a different way than I do, and by thinking hard about why certain methods or attitudes which seem silly to me, still remain popular.

Finally, my approach as a political scientist and public opinion researcher is to understand the views of others. I think I have a pretty good grip on why it can make sense for people to vote for Gingrich or Romney or Obama or Santorum or whatever, and I’m interested in understanding political ideologies as they manifest themselves in different areas (even in statistics, where political views range from Dennis Lindley to Jacob Wolfowitz).

“Moving beyond monoculture” doesn’t mean that I abandon my skepticism but it means that I should at least try to understand other approaches to looking at the world.

P.S. I thought the above discussion would be more useful than yet another argument about the extent to which modern education is such a scam etc.

Alongside all this are the plagiarism scandals, which are uninteresting from a scientific context but are relevant in that, in many cases, neither the institutions housing the plagiarists nor the editors and publishers of the plagiarized material seem to care. Perhaps these universities and publishers are more worried about bad publicity (and maybe lawsuits, given that many of the plagiarism cases involve law professors) than they are about scholarly misconduct.

Before going on, perhaps it’s worth briefly reviewing who is hurt by the publication of flawed research. It’s not a victimless crime. Here are some of the malign consequences:

- Wasted time and resources spent by researchers trying to replicate non-findings and chasing down dead ends.

- Fake science news bumping real science news off the front page.

- When the errors and scandals come to light, a decline in the prestige of higher-quality scientific work.

- Slower progress of science, delaying deeper understanding of psychology, medicine, and other topics that we deem important enough to deserve large public research efforts.

This is a hard problem!

There’s a general sense that the system is broken with no obvious remedies. I’m most interested in presumably sincere and honest scientific efforts that are misunderstood and misrepresented into more than they really are (the breakthrough-of-the-week mentality criticized by Ioannides and exemplfied by Bem). As noted above, the cases of outright fraud have little scientific interest but I brought them up to indicate that, even in extreme cases, the groups whose reputations seem at risk from the unethical behavior often seem more inclined to bury the evidence than to stop the madness.

If universities, publishers, and editors are inclined to look away when confronted with out-and-out fraud and plagiarism, we can hardly be surprised if they’re not aggressive against merely dubious research claims.

In the last section of this post, I briefly discuss several examples of dubious research that I’ve encountered, just to give a sense of the difficulties that can arise in evaluating such reports.

What to do (statistics)?

My generic solution to the statistics problems involved in estimating small effects is to replace multiple comparisons by multilevel modeling, that is, to estimate configurations rather than single effects or coefficients. This tactic won’t solve every problem but it’s my overarching conceptual framework. There’s lots room for research on how to do better in particular problem settings.

What to do (scientific publishing)?

I have clearer ideas of resolutions (at least in the short term) of the Bem paradox; in short, what to do with dubious but potentially interesting findings.

So far there seem to be two suggestions out there: Either publish such claims in top journals (as for example Bem’s in JPSP, or the contagion-of-obesity paper in NEJM), or the journals should reject them (perhaps from some combination of more careful review of methodology, higher standards than classical 5% significance, and Bayesian skepticism).

The problem with the publish-in-top-journals strategy is that it ensures publicity for some mistakes and it creates incentives for researchers to stretch their statistics to get a prestigious publication.

The problem with the reject-’em-all-and-let-the-Arxiv-sort-’em-out strategy is that it’s perhaps too rigorous. So many papers have potential methodological flaws. Recall that the Bem paper was published, which means in some sense that its reviewers thought the paper’s flaws were no worse than what usually gets published in JPSP. Long-term, sure, we’d like to improve methodological rigor, but in the meantime a key problem with Bem’s paper was not just its methodological flaws, it was also the implausibility of the claimed results.

So here’s my proposed solution. Instead of publishing speculative results in top journals such as JPSP, Science, Nature, etc., publish them in lower-ranked venues. For example, Bem could publish his experiments in some specialized journal of psychological measurement. If the work appears to be solid (as judged by the usual corps of referees), then publish it, get it out there. I’m not saying to send the paper to a trash journal; if it’s good stuff it can go in a good journal, the sort where peer review really means something. (I assume there’s also a journal of parapsychology but that’s probably just for true believers; it’s fair enough that Bem etc would like to publish somewhere that outsiders would respect.)

Under this system, JPSP could feel free to reject the Bem paper on the grounds that it’s too speculative to get the journal’s implicit endorsement. This is not suppression or censorship or anything like it, it’s just a recommendation that the paper be sent to a more specialized journal where there will be a chance for criticism and replication. At some point, if the findings are tested and replicated and seem to hold up, then it could be time for a publication in JPSP, Science, or Nature.

From the other side, this should be acceptable to the Bems and Fowlers who like to work on the edge. You still get your ideas out there in a respectable publication (and you still might even get a bit of publicity), and then you, the skeptics, and the rest of the scientific community can go at it in public.

There have also been proposals for more interactive publications of individual articles, with bloglike opportunities for discussion and replies. That’s fine too, but I think the only way to make real progress here is to accept that no individual article will tell the whole story, especially if the article is a report of new research. If the Bem finding is real, this can be demonstrated in a series of papers in some specialized journal.

Appendix: Individual cases can be tough!

I’ve encountered a lot of these borderline research findings over the past several years, and my own reaction is typically formed by some mix of my personal scientific knowledge, the statistical work involved, and my general impressions. Here are a few examples:

“Beautiful women have more daughters”: I was pretty sure this one was empty just based on my background knowledge (the claim was an difference of 8 percentage points, which is much more than I could possibly expect based on the literature). Careful review of the articles led me to find problems with the statistics.

Dennis the dentist, Laura the lawyer, and the proclivity of Dave Kingman and Vince Koleman to strike out a lot: I was ready to believe the Dennis/Laura effect on occupations and only slightly skeptical of the K effect on strikeouts, but then the work was later strongly criticized on methodological grounds. Still, my back-of-the-envelope calculation let me to believe that they hypothesized effects could be there.

Warming increases the risk of civil war in Africa: This one certainly could be true but something about it rang some bells in my head and I’m skeptical. The statistical evidence here is vague enough that I could well take the opposite tack, believing the claim and being skeptical about skepticism of it. To be honest, if I knew these researchers personally I might very well be more inclined to trust the result. (And that’s not so silly: if I knew them personally I could ask them a bunch of questions and get a sense of where their belief in this finding is coming from.)

“45% hitting, 25% fielding, and 25% pitching”: I was skeptical here because it was presented as a press release with no link to the paper but with enough details to make me suspect that the statistical analysis was pretty bad.

“Minority rules: scientists discover tipping point for the spread of ideas”: I don’t know if this should be called “junk science” or just a silly generalization from a mathematical model. Here I was suspicious because the claim was logically inconsistent and the study as a whole fit the pattern of physicists dabbling in social science. (As I wrote at the time, I’ll mock what’s mockable. If you don’t want to be mocked, don’t make mockable claims.)

“Discovered: the genetic secret of a happy life”: There’s potentially something here but the differences are much smaller than implied by the headlines, the news articles, or even the abstract of the published article.

Whatever medical breakthrough happens to have been reported in the New York Times this week: I believe all of these. Even though I know that these findings don’t always persist, when I see it in the newspaper and I know nothing about the topic, I’m inclined to just believe.

That’s one reason the issue of flawed research is important! I’m as well prepared as anyone to evaluate research claims, but as a consumer I can be pretty credulous when the research is not close to my expertise.

If there is any coherent message from the above examples, it is that my own rules for how to evaluate research claims are not clear, even to me.

Good stuff, I recommend you all read it.

Earlier this year, a major psychology journal published a paper suggesting that there was some evidence for “pre-cognition,” a form of ESP. Stuart Ritchie, a doctoral student at the University of Edinburgh, is part of a team that tried, but failed, to replicate those results. Here, he tells the Chronicle of Higher Education’s Tom Bartlett about the difficulties he’s had getting the results published.

Several journals told the team they wouldn’t publish a study that did no more than disprove a previous study. . . . An editor at another journal said he’d “only accept our paper if we ran a fourth experiment where we got a believer [in ESP] to run all the participants, to control for . . . experimenter effects.”

My reaction is, this isn’t as easy a question as it might seem. At first, one’s reaction might share Ritchie’s frustration that a shoddy paper by Bem got published while Ritchie’s careful replication got dinged. But, as I wrote when the issue came up on the sister blog:

Setting aside the whole “psychic powers” thing, it makes sense to me not to run the new experiment. After all, it’s hardly news that ESP doesn’t work. If “ESP doesn’t work” were publishable, you could fill up a journal many times over with such findings. And what would be the point of that? Better to start a new journal with some catchy title such as Replications of Well-Known Findings. In the physics division, you could have articles demonstrating that objects fall down, not up. In the chemistry division, you could publish demonstrations that H2 + O2 yields H2O plus energy. The biology section could have a paper demonstrating that cats and dogs can’t produce offspring. And so on.

So I don’t know the answer here. On one hand, we can hardly require or even expect that journals fill their pages with dog-bites-man nonreplications. (And, even in a computerized era where there are no page limits, there are still constraints on the time of editors and reviewers.) On the other hand, this leads to an asymmetry where crap gets on the front page and the refutation doesn’t even get published on page B16.

Our original comment (Lindquist and Sobel, 2011) made explicit the types of assumptions neuroimaging researchers are making when directed graphical models (DGMs), which include certain types of structural equation models (SEMs), are used to estimate causal effects. When these assumptions, which many researchers are not aware of, are not met, parameters of these models should not be interpreted as effects. . . . [Judea] Pearl does not disagree with anything we stated. However, he takes exception to our use of potential outcomes notation, which is the standard notation used in the statistical literature on causal inference, and his comment is devoted to promoting his alternative conventions. [Clark] Glymour’s comment is based on three claims that he inappropriately attributes to us. Glymour is also more optimistic than us about the potential of using directed graphical models (DGMs) to discover causal relations in neuroimaging research . . .

Lindquist and Sobel’s arguments make sense to me, except on one point. They consider a causal setting z -> x -> y, where z is the treatment variable, x is the intermediate outcome, and y is the ultimate outcome, and much of their discussion centers on estimating the causal effect of x on y. I have two difficulties with their perspective:

1. If x is an observed variable that is not directly manipulated, I don’t know if it makes sense to talk about the effect of x on y, unconditional on the intervention that was used to change x. In their example, I’d talk about “the effect of x on y, if x is changed through z.” Different z’s can induce different effects of x on y.

2. Lindquist and Sobel talk about the effect of z on x. If z=0 or 1, they write x(z), so that the causal effect of z on x is x(1) – x(0) (or, more generally, x(1) compared to x(0), but we lose nothing by considering simple differences here). So far, so good.

But I get stuck at the next step, where they define the effect of x on y. If x can equal 0 or 1, they write y(z,x), so that the causal effect of x on y, conditional on z, is y(z,1) – y(z,0). At least, I think that’s what they’re saying.

The trouble is, I don’t see how the two parts of this model fit together. For any given item in the experiment, I think they’re following the rule that x(z) has a particular (although maybe unknown) value. But then I don’t see what it means to look at y(z,1) – y(z,0). For any particular value of z, it seems to me that only one of these two terms is possible. (For example, if x(z)=1, then y(z,1) is defined but y(z,0) seems meaningless.)

I’m not saying that this framework is wrong, just that I don’t understand it.

That said, Lindquist and Sobel’s criticisms of Pearl and Glymour seem sound to me.

P.S. I wrote this last month and put it in the queue. Since then I’ve noticed that Pearl has responded to Lindquist and Sobel; see here. I don’t find Pearl’s response to be so convincing—I agree with Lindquist and Sobel’s statement that the graphical or structural equation modeling expression looks simple and appealing but the underlying assumptions in those expressions are not so clear. But you can judge for yourself; as I wrote in my discussion of the book by Morgan and Winship, it’s good to have muultiple expressions for a model, as different users are looking for different things.

To be specific, Pearl contrasts three expressions of a single model, the causal chain Z—>X—>Y. Here’s Pearl:

Pearl characterizes the third expression is a more meaningful and clear display.

In contrast, Lindquist and Sobel argue that the above graphical expression appears clear only because it sweeps the model’s assumptions under the rug. Lindquist and Sobel write:

None of this seems clear and simple to me! Speaking of clear and simple, I’m reminded of a scene, several decades ago, when a bunch of us on the county math team won some competition, and the prize was that we each got to choose one of several math books. One of the books was called Elementary Linear Algebra, and I remember making a disdainful remark to my friend that I didn’t want something elementary. My friend replied, “Linear algebra is not elementary.” Good point.

Which brings back another memory: our coach for the Mathematical Olympiad program was an unbelievably grumpy old man. At one point he interrupted one of his lectures to rant about how all the calculus books now are wasting their space with applications. At some point, he said, they’re gonna come up with a book called Applied Calculus with Applications. That all seemed natural to me at the time but in retrospect I’m amazed by how brainwashed we all were. There was one kid there who I recall was interested in engineering problems rather than number theory etc., but that was an unusual preference. (I just looked him up and, amazingly, he grew up to be an engineering researcher!) The other thing I remember about the grumpy coach dude, besides his personality (which, in retrospect, was perhaps necessary to keep a bunch of 15-year-old boys in line; even nerds can make trouble), was that he thought it was cheating to use calculus or analytic geometry. His favorite sorts of problems used elaborate arguments from classical geometry and he always felt we should be able to solve these without resorting to technical means.

As I’ve remarked more than once in this space, I feel lucky in retrospect to have been pretty unprepared for the Olympiad program, with the result that I didn’t do very well there, gradually lost interest in this sort of competitive event, and decided I didn’t want to be a pure mathematician. I think it must’ve been really hard on the kids who were top performers but didn’t happen to be Noam. It was easier for those of us in the bottom half of the group.

I am a recent graduate looking for a bit of advice. While I took intro classes on math and statistics in my undergraduate degree as a political science major, I find myself university-less and seeking to develop my statistics toolkit.

I work for an NGO in the international development field. I think that a solid statistics foundation would offer me not only more career opportunities, but more importantly, a deeper and more nuanced understanding of the processes and problems that interest me. I’m talking about field experiments and practical quantitative and qualitative data analysis.

I have plenty of free time, ambition, and enthusiasm to improve this part of my toolbox, but I lack an attachment to an institution and much in the way of financial resources. How would you go about making a concentrated effort at acquiring an understanding of the field and its actual application in something like R or Stata, which I admit to never having used?

Perhaps I am simply asking about web resources or best texts, but any broad advice would be much appreciated too.

My gut recommendation is to start with a problem you care about and figure out what you need to get a reasonable solution, then go to the next problem, and so forth. For books, you could start with The Statistical Sleuth and my book with Jennifer. If you want to learn R, just try to make some pretty and useful graphs, that will motivate you to be able to do more.

Any other suggestions?

A colleague sends in some suugestions based on a recent experience with a package update:

1. Always use the R dev version to write a package. Not the current stable release. The R people use the R dev version to check your package anyway. If you don’t use the R dev version, there is chance that your package won’t pass the check. In my own experience, every time R has a major change, it tends to have new standards and find new errors in your package with these new standards. So better use the dev version to find out the potential errors in advance.

2. After submission, write an email to claim it. I used to submit the package to the CRAN without writing an email. This was standard operating procedure, but it has changed. Writing an email to claim about the submission is now a requirement. There is a good reason. The R team is afraid that the package was not submitted by a legal developer. So there is a security issue involved here. Write an email to remind them that you submit a package, not a virus.

3. The R people are busy. The number of R packages submitted to CRAN is growing exponentially. So the R people’s working loads are heavy. We should understand their situation and try to work with them to solve the package issues, when problems come up.

The first two lessons are the most important. If you have done the first two, I believe you won’t need the third one.

I’ve never actually written an R package myself—my last experience with this sort of thing was several years ago, using dyn.load and dyn.load2 in S—but I’ve used many R packages and I’ve contributed to several widely-used R packages. So I really appreciate the effort put in by the central R people, and I’m posting this note as a way to make their lives easier and also help the people who are writing and updating R packages.

I [Pearl] turned Bayesian in 1971, as soon as I began reading Savage’s monograph The Foundations of Statistical Inference [Savage, 1962]. The arguments were unassailable: (i) It is plain silly to ignore what we know, (ii) It is natural and useful to cast what we know in the language of probabilities, and (iii) If our subjective probabilities are erroneous, their impact will get washed out in due time, as the number of observations increases.

Thirty years later, I [Pearl] am still a devout Bayesian in the sense of (i), but I now doubt the wisdom of (ii) and I know that, in general, (iii) is false.

He elaborates:

The bulk of human knowledge is organized around causal, not probabilistic relationships, and the grammar of probability calculus is insufficient for capturing those relationships. Specifically, the building blocks of our scientific and everyday knowledge are elementary facts such as “mud does not cause rain” and “symptoms do not cause disease” and those facts, strangely enough, cannot be expressed in the vocabulary of probability calculus. It is for this reason that I consider myself only a half-Bayesian.

Interesting. The Neyman-Rubin framework of potential outcomes does allow for casual reasoning within a probabilistic structure, but indeed it does not allow for statements such as “mud does not cause rain.” In the potential outcomes notation, one could define a random variable y=1 for rain or 0 for no rain, and define y^1 to be the outcome under treatment and y^2 to be the outcome under control. But it would not make sense for “mud” to be a treatment: in the potential-outcomes framework, a treatment is something that you do, not something such as “mud” that you observe.

I’m not saying here that Pearl’s framework is a good or bad idea; my point here is that I’m agreeing that he indeed seems to be asking questions that cannot be addressed by probability models.

Some of my earlier discussions with Pearl are here.