I’ve been blogging a lot lately about plagiarism (sorry, Bob!), and one thing that’s been bugging me is, why does it bother me so much. Part of the story is simple: much of my reputation comes from the words I write, so I bristle at any attempt to devalue words. I feel the same way about plagiarism that a rich person would feel about counterfeiting: Don’t debase my currency!

But it’s more than that. After discussing this a bit with Thomas Basbøll, I realized that I’m bothered by the way that plagiarism interferes with the transmission of information:

Much has been written on the ethics of plagiarism. One aspect that has received less notice is plagiarism’s role in corrupting our ability to learn from data: We propose that plagiarism is a statistical crime. It involves the hiding of important information regarding the source and context of the copied work in its original form. Such information can dramatically alter the statistical inferences made about the work.

In statistics, throwing away data is a no-no. From a classical perspective, inferences are determined by the sampling process: point estimates, confidence intervals and hypothesis tests all require knowledge of (or assumptions about) the probability distribution of the observed data. In a Bayesian analysis, it is necessary to include in the model all variables that are relevant to the data-collection process. In either case, we are generally led to faulty inferences if we are given data from urn A and told they came from urn B.

A statistical perspective on plagiarism might seem relevant only to cases in which raw data are unceremoniously and secretively transferred from one urn to another. But statistical consequences also result from plagiarism of a very different kind of material: stories. To underestimate the importance of contextual information, even when it does not concern numbers, is dangerous.

Here’s our full article (which has just appeared in the American Scientist). It features two of the recurring characters from this blog. Here’s our conclusion:

Scholars in fields ranging from psychology to history to computer science have recognized that stories are part of how people understand the world. As statisticians, we can consider reasoning from stories as a form of approximate inference. From this perspective, statistical principles should provide some approximate guidance about the potential biases and precision of such inferences. One key principle is not to throw away information and, if discarding data is for some reason necessary, to describe as clearly as possible the mechanism by which the relevant information was excluded. Plagiarism violates both these rules and, as such, is a violation of statistical ethics, beyond any other considerations of moral behavior.

P.S. I’m more interested in scientific plagiarism than the legal or literary variety, but this 2004 news article by Daniel Hemel and Lauren Schuker (which I found by googling *laurence tribe plagiarism*) is full of good quotes. Here’s my favorite part:

Tribe’s mea culpa comes just three weeks after another prominent Harvard faculty member—Climenko Professor of Law Charles J. Ogletree—publicly apologized for copying six paragraphs almost word-for-word from a Yale scholar in a recent book, All Deliberate Speed.

Last fall, Frankfurter Professor of Law Alan M. Dershowitz also battled plagiarism charges. And in 2002, Harvard Overseer Doris Kearns Goodwin admitted that she had accidently copied passages from another scholar in her bestseller The Fitzgeralds and the Kennedys.

University President Lawrence H. Summers told The Crimson in an interview last week—before the allegations against Tribe surfaced—that he did not see “a big trend” of plagiarism problems at the Law School as a result of the charges against Ogletree and Dershowitz, but indicated that a third case would change his mind.

“If you had a third one, then I would have said, okay, you get to say this is a special thing, a focused problem at the Law School,” Summers said of the recent academic dishonesty cases.

He declined comment last night.

The post To Throw Away Data: Plagiarism as a Statistical Crime appeared first on Statistical Modeling, Causal Inference, and Social Science.

The relevance of statisticians to researchers in different fields of social science.

Regression discontinuity.

Free expression vs. not wanting to make anyone personally uncomfortable.

Political coalitions are diverse (and there’s no use pretending otherwise).

According to David Brooks, staying out of jail is a conservative value.

I’ve heard of the IRB, but this is ridiculous.

This will make Richard Florida very happy.

I don’t know whether to call it communism or crony capitalism . . ..

Concepts and folk theories.

This should keep youall busy for awhile.

The post Recently in the sister blog appeared first on Statistical Modeling, Causal Inference, and Social Science.

Last month, the faculty in charge of the course wrote the following public letter:

The United States is in the midst of a debate over the value of a traditional college education. Why enroll in a place like Columbia College when you can obtain an undergraduate degree for $10,000 or learn everything from Massive Open Online Courses? In more parochial terms, what is the value added by approaches such as Columbia’s Core Curriculum? Recently students in our Core Course, Frontiers of Science (FoS), provided a partial answer. The FoS faculty designed a survey to gauge the scientific skills and knowledge of the Class of 2016 both before and after taking FoS. In an assembly held during orientation week last August, 966 first-year Columbia College students answered questions covering basic skills such as statistics, probability, and the reading and analysis of graphs, as well as content to be taught during the fall session of FoS. For the 519 College students taking FoS in the fall semester, the same survey was administered again as part of the final exam. The mean score for the initial orientation survey was less than 28%. The mean score for the same questions at the end of the semester was 76%! (The margin of error in both cases was ±1.0%.) To control for gains independent of FoS, 167 first-year College students, who did not take FoS in the fall, answered the same survey questions again at the start of the spring semester. The mean score of those students was 31%, not substantially greater than they and their classmates had scored on the survey during orientation. This type of research always has its limitations. However, the results with and without FoS are so different that the conclusion is inescapable. The scientific habits and knowledge that FoS imparts are new to, and effectively learned by, first-year students after one semester of intense study . . .

This sounds pretty good, especially the part about “basic skills such as statistics, probability, and the reading and analysis of graphs.” My first thought is that, if these skills are so important, maybe all the students at Columbia should be taking a course in probability and statistics! But then I remembered that our intro stat course isn’t so great (I know, I’ve taught it several times), so maybe it’s just as well if some biologists, physicists, etc., create a new statistics module from scratch. Seriously, I have lots of ideas of how we could teach intro prob/stat better, but when I actually try to do it, I get all tangled in the details. So I can’t very well object to outsiders taking a shot at it. As users of statistics, they might have a better idea than I do of how to teach the subject.

Similarly, I wouldn’t be surprised if the engineers could teach a better intro physics class than the physicists could. And it could well make sense to have biologists teach first-year chemistry, and psychologists teach first-year biology.

What, then, would the statistician teach? First year math, of course. It would make the mathematicians cringe, but it might be closer to the students’ own level. Statisticians are users of math, and we could teach the subject from a user’s perspective. Just as, arguably, the collection of scientists who run Frontiers of Science might be teaching probability and statistics in a way more useful to freshman than whatever my colleagues and I in the stat department could come up with.

OK, now that I’ve established my complete acceptance of the idea that this team can, and perhaps should, be teaching probability and statistics to Columbia’s freshmen, let me say that there are a bunch of things about the above-quoted letter that bother me.

First, the economic basis for their argument: “Why enroll in a place like Columbia College when you can obtain an undergraduate degree for $10,000 or learn everything from Massive Open Online Courses?”

It’s not new that you can obtain an undergraduate degree for $10,000! Until relatively recently, in-state tuition at state schools was cheap, pretty much everywhere. At Columbia you get the top professors teaching what they know best. But that argument doesn’t work for Frontiers of Science, which features non-statisticians teaching statistics, and a particle physicist who strips off his clothes to get students to recognize that nuclear weapons proliferation is scary.

As for online education, it’s great, but if you can “learn everything” from a mook, you can learn everything by buying a stack of textbooks. Or maybe not. I just don’t see that the results from this study have anything to do with mooks, one way or another.

Second, the pre-test and post-test. This actually does seem like a good idea, and I remain embarrassed that my statistics colleagues and I don’t do this in our own classes. That said, “The margin of error in both cases was ±1.0%.” Are they kidding? And “the conclusion is inescapable”?? We’d need to know more about the post-test conditions of those 167 freshmen who were not in the course. One student writes that the students in Frontiers of Science were given “premade cheat sheets during the final” with formulas. Maybe the 167 controls were given the cheat sheets too, but such sheets might not mean much without some prior coaching.

What was on the quiz? Nicholas Christie-Blick, one of the authors of the letter quoted above, writes, “The questions and answers will be released. . . . Nor does it matter much what the questions are. What matters is that they are representative of the range of skills in scientific thinking that we call Scientific Habits.”

I’d say something snarky here, but then again I remember that my own intro stat course hasn’t always gone so well, nor is it clear how much my students ever learned.

So, in quick summary, I’m suspicious of the claims of efficacy of the Frontiers of Science course. But I’m sympathetic with their efforts to measure anything at all. Measurement in education is tough.

The post Evaluating Columbia University’s Frontiers of Science course appeared first on Statistical Modeling, Causal Inference, and Social Science.

A good study does not need to be methodologically original, but it should be methodologically sound. When we do surveys we worry about nonresponse. To take a few hundred people off MTurk and not even look for possible nonresponse bias, this is not serious.

But, again, it’s not so much that this paper was flawed, as that it had nothing much positive to offer.

Just to be clear: I’m really really really really not trying to censor such work, and I’m really really really really not saying this work should not be published. What I’m saying is that the top journal in a field should not be publishing such routine work. They should be publishing the best quality research, not just random things that happen to slip through the cracks.

I mean, sure, the referees should’ve caught the problems with that paper. But I blame the editors for even considering publication. Even without noticing the paper’s methodological flaws, it was nothing special.

And, once you decide to start publishing mediocre papers in your top journal, you’re asking for trouble. You’re encouraging more of the same.

To clarify, let me compare this with some other high-profile examples:

- Christakis and Fowler’s study of the contagion of obesity. This was published in top journals and was later found to have some serious methodological issues. But it’s not a mediocre work. They had a unique dataset, a new idea, and some new methods of data analysis. OK, they made some mistakes, but I can’t fault a leading journal for publishing this work. It has a lot of special strengths.

- Bem’s paper claiming to demonstrate ESP. OK, I wouldn’t have published this one. But I can see where the journal was coming from on this. If the results had held up, it would’ve been the scientific story of the decade, and the journal didn’t want to miss out. The editors didn’t show the best judgment here, but their decision was understandable.

- Kanazawa’s papers of schoolyard evolutionary biology. I’ve written about the mistakes here, and this work has a lot of similarities to the ovulation-and-voting study. The difference is that Kanazawa’s papers were published in a middling place—the Journal of Theoretical Biology—not in a top journal of their field. Don’t get me wrong, JTB is respectable, but it’s in the middle of the pack. It’s not expected that they are publishing the best of the best.

- Hamilton’s paper in the American Sociological Review, claiming that college students get worse grades if their parents pay. This paper had a gaping hole (not adjusting for the selection effect arising from less well-funded students dropping out) and I think it was a mistake for it to be published as is—but that’s just something the reviewers didn’t catch. On the plus side, Hamilton’s paper was thoughtful and had some in-depth analysis of large datasets. It had mistakes, but it had strengths too. It was not a turn-the-crank, run-an-online-survey-and-spit-out-p-values job.

My point is that, in all these cases of the publication of flawed work (and one could add the work of Mark Hauser and Bruno Frey as well), the published papers either had clear strengths or else were not published in top journals. When an interesting, exciting, but flawed paper (such as those by Bem, Hauser, etc) is published in a top journal, that’s too bad, but it’s understandable. When a possibly interesting paper (such as those by Kanazawa) is published in an OK journal, that makes sense too. It would not make sense to demand perfection. But when a mediocre paper (which also happens to have serious methodological flaws) is published in a top journal, there’s something seriously wrong going on. There are lots of things that can make a research paper special, and this paper had none of those things (unless anything combining voting and sex in an election year is considered special).

P.S. Let me emphasize that my goal here is not to pile on and slam the ovulation and voting paper. In fact, I’ve refrained linking to the paper here, just to give the authors a break. They did a little study that happened to be flawed. That’s no big deal. I’ve done lots of little studies that happened to be flawed, and sometimes my flawed work gets published. I’m not criticizing the authors for making some mistakes. I hope they can do better next time. I’m criticizing the journal for publishing a mediocre paper with little to offer. That’s not just a retrospective mistake; it seems like a problem with their policies that they would think that such an unremarkable paper could even be seriously considered for publication in the top journal of their field.

The post What happened that the journal Psychological Science published a paper with no identifiable strengths? appeared first on Statistical Modeling, Causal Inference, and Social Science.

I’m reminded of Nicholson Baker’s hilarious “From the Index of First Lines,” which is truly the poetic counterpart to Basbøll’s argument in prose:

The post Prose is paragraphs, prose is sentences appeared first on Statistical Modeling, Causal Inference, and Social Science.

I came across this graphic of vaccine-attributed decreases in mortality and was curious if you found it as unattractive and unintuitive as I did. Hope all is well with you!

My reply: All’s well with me. And yes, that’s one horrible graph. It has all the problems with a bad infographic with none of the virtues. Compared to this monstrosity, the typical USA Today graph is a stunning, beautiful masterpiece. I don’t think I want to soil this webpage with the image. In fact, I don’t even want to link to it.

The post uuuuuuuuuuuuugly appeared first on Statistical Modeling, Causal Inference, and Social Science.

Lee Sechrest sends along this article by Brian Haig and writes that it “presents what seems to me a useful perspective on much of what scientists/statisticians do and how science works, at least in the fields in which I work.” Here’s Haig’s abstract:

A broad theory of scientific method is sketched that has particular relevance for the behavioral sciences. This theory of method assembles a complex of specific strategies and methods that are used in the detection of empirical phenomena and the subsequent construction of explanatory theories. A characterization of the nature of phenomena is given, and the process of their detection is briefly described in terms of a multistage model of data analysis. The construction of explanatory theories is shown to involve their generation through abductive, or explanatory, reasoning, their development through analogical modeling, and their fuller appraisal in terms of judgments of the best of competing explanations. The nature and limits of this theory of method are discussed in the light of relevant developments in scientific methodology.

I found this very difficult to read and forwarded it to Cosma Shalizi, who writes:

Like a lot of what I read about abduction, it seems much more a theory (or sketch of a theory) of how scientists think, than of scientific method. Put another way, the H-D account of scientific method has always tended to “black-box” the issue of where hypotheses come from, in favor of what to do with them once you have them. I think this is usually helpful, but there’s no reason not to try to open up the black box, and study the origin of hypotheses; if there’s a role for abduction, it’s there, in explicating the “generate” part of generate-and-test. (In fact, if memory serves, Peirce later repented of his term “abduction”, and just called it “hypothesis” or “hypothesizing”.) If one could show, or even plausibly suggest, that certain modes of hypothesizing are systematically more reliable or fruitful than others, that would be extremely valuable. This paper in particular seems to have some odd confusions of levels between what are presumably fairly permanent parts of how scientists think (analogy), and current technological artifacts — I love bootstrapping, but it hardly belongs in the same category as a component of scientific method. (And as for stem-and-leaf plots…)

Sechrest wrote:

Too much research seems to be addressed to determining whether “it is,” or “it isn’t.” The more important question very often is “Why is (or isn’t) it?” To me, abduction seems more likely to occur in the aftermath of having seen something. Isaac Asimov once said The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ but ‘That’s funny…’

And that is when abduction begins, the attempt to identify explanations and reason toward the best one. For example, at least some drug trials begin with seemingly sensible expectations that a drug will work; those expectations are often wrong. Usually that is the end of the matter. But, to me, an important question may well be “Why didn’t the drug work as expected?” (A “why isn’t it? question) The abductive process cannot lead directly to a clear-cut answer, but it can get us closer. And that is what, I think, good scientists do. Ineffective scientists (and I have seen them many times) say, “Well, that didn’t work. Anybody got another idea?”

I have nothing to add to the above discussion, except to point to our recent discussion of the challenges of systematizing model building. As I see it, new ideas arise from anomalies in data with respect to existing theories.

The post Where do theories come from? appeared first on Statistical Modeling, Causal Inference, and Social Science.

First, Stephen Olivier pointed me to this post by Daniel Lakens, who wrote the following open call to statisticians:

You would think that if you are passionate about statistics, then you want to help people to calculate them correctly in any way you can. . . . you’d think some statisticians would be interested in helping a poor mathematically challenged psychologist out by offering some practical advice.

I’m the right person to ask this question, since I actually have written a lot of material that helps psychologists (and others) with their data analysis. But there clearly are communication difficulties, in that my work and that of other statisticians hasn’t reached Lakens. Sometimes the contributions of statisticians are made indirectly. For example, I wrote Bayesian Data Analysis, and then Kruschke wrote Doing Bayesian Data Analysis. Our statistics book made it possible for Kruschke to write his excellent book for psychologists. This is a reasonable division of labor.

That said, I’d like to do even more. So I will make some specific suggestions for data analysis in psychology right here in this post, in the context of my next story:

Dan Kahan sent me this note:

The most egregious instance of totally bogus methods I had the misfortune to feel obliged to call foul on involved an econometrics study that purported to find that changes in law that never happened increased homicides by “lowering the cost” of committing them…)

Actually, as you know, often times investigation of a “wtf?!” report like this discloses that the problem is in the news report & not in the study.

I think you agree that many of the “bad statistics/methods” problems & even the “nonreplicability” problem are rooted in the perpetuation of a set of mindless statistical protocols associated with ossified conception of NHT (one from which all the thought that it might have reflected was drained away & discarded decades ago).

But certainly another problem is the “wtf?!!!!!!” conception of psychology.  Its distinguishing feature is its supposed discovery of phenomena that are shocking bizarre & lack any coherent theory.

The alternative conception of psychology is the “everything is obvious – once you know the answer.”  The main point of empirical research isn’t to shock people. It’s to adjudicate disputes between competing plausible conjectures about what causes what we see.  More accounts of what is going are plausible than are true; without valid inference from observation, we will never separate the former from the sea of the latter & will drown in a sea of “just so” story telling.

I have zero confidence in “wtf?!!!” & am convinced that it is a steady stream of bogus, nonreplicable studies that hurt the reputation of psychology.

I have lots of confidence in EIO–OYKTA. It’s not nearly so sexy — which is good, b/c it removes the temptation to cut corners in all the familiar, petty ways that researchers do (usually by coaxing out a shy “p < 0.05″ to emerge  w/ one or another data-manipulative come-on line). But it is dealing with matters that reflect real, theorized, validated mechanisms of psychology (the issue in each case is — which one?!), and ones that are important enough for researchers to keep at it essentially forever, revising, correcting, improving our evolving understanding of what’s going on.

Kahan points to a much-mocked and criticized study by Kristina Durante, Ashley Arsena, Vladas Griskevicius, “The Fluctuating Female Vote: Politics, Religion, and the Ovulatory Cycle,” which was reported then retracted from CNN under the title, “Study looks at voting and hormones: Hormones may influence female voting choices.”

The relevance for the present discussion is that this paper was published in Psychological Science, a top journal in psychology. Here’s the abstract:

Each month many women experience an ovulatory cycle that regulates fertility. Whereas research finds that this cycle influences women’s mating preferences, we propose that it might also change women’s political and religious views. Building on theory suggesting that political and religious orientation are linked to reproductive goals, we tested how fertility influenced women’s politics, religiosity, and voting in the 2012 U.S. presidential election. In two studies with large and diverse samples, ovulation had drastically different effects on single versus married women. Ovulation led single women to become more liberal, less religious, and more likely to vote for Barack Obama. In contrast, ovulation led married women to become more conservative, more religious, and more likely to vote for Mitt Romney. In addition, ovulatory-induced changes in political orientation mediated women’s voting behavior. Overall, the ovulatory cycle not only influences women’s politics, but appears to do so differently for single versus married women.

I took a look at the paper, and what I found was a bunch of comparisons and p-values, some of which were statistically significant, and then lots of stories. The problem is that there are so many different things that could be compared, and all we see is some subset of the comparisons. Many of the reported effects seem much too large to be plausible. And there’s a casual use of causal language (for example, the words “influenced,” “effects,” and “induced” in the above abstract) to describe correlations.

Beyond all that, I found the claimed effects implausibly large. For example, they report that, among women in relationships, 40% in the ovulation period supported Romney, compared to 23% in the non-fertile part of their cycle. Given that surveys find vary few people switching their vote preferences during the campaign for any reason, I just don’t buy it. The authors might respond that they don’t care about the magnitude of the difference, just the sign, but (a) with a magnitude of this size, we’re talking noise noise noise, and (b) one could just as easily explain this as a differential nonresponse pattern: maybe liberal or conservative women in different parts of their cycle are more or less likely to participate in a survey. It would easy enough to come up with a story about that!

Anyway, my point is not to slam the work of Durante et al. They did a little study, wrote it up, and submitted it to one of the leading journals in their field. It’s not their fault the journal chose to publish it.

Also, let me emphasize that I’m not saying that their claims (regarding the effects of ovulation) are false. I’m just saying that the evidence from their paper isn’t as strong as they make it out to be.

A statistician offers helpful advice for psychology researchers

My real goal here is to address the question that was brought up at the beginning of this post: What recommendations can I, as a statistician, give to psychology researchers? Here are a few, presented in the context of the paper on ovulation and political attitudes:

1. Analyze all your data. For most of their analyses, the authors threw out all the data from participants who were PMS-ing or having their period. (“We also did not include women at the beginning of the ovulatory cycle (cycle days 1–6) or at the very end of the ovulatory cycle (cycle days 26–28) to avoid potential confounds due to premenstrual or menstrual symptoms.”) That’s a mistake. Instead of throwing out one-third of their data, they should’ve just included that other category in their analysis.

2. Present all your comparisons. The paper leads us through a hopscotch of comparisons and p-values. Better just to present everything. I have no idea if the researchers combed through everything and selected the best results, or if they simply made a bunch of somewhat arbitrary decisions throughout of what to look for.

For example, I would’ve liked to see a comparison of respondents in different parts of their cycle on variables such as birth year, party identification, marital status, etc etc. Just a whole damn table (even better would be a graph but, hey, I won’t get greedy here) showing these differences for every possible variable.

Instead, what do we get? Several pages full of averages, percentages, F tests, chi-squared tests, and p-values, all presented in paragraph form. Better to have all possible comparisons in one convenient table.

3. Make your data public. If the topic is worth studying, you should want others to be able to make rapid progress. If there’s some confidentiality restrictions, remove the respondents’ identifying information. Then post the data online.

4. And now some advice for journal editors. What’s the purpose of a top journal in a field such as psychology? I think they should be publishing the top work. This paper is not top work, by any standard. The researchers asked a few survey questions to a bunch of people on Mechanical Turk, then did who knows how many comparisons and significance tests, reported some subset of the results, then went on to story time. It’s not innovative data collection, it’s not great theory, it’s not great data analysis, it’s not a definitive data source, it’s nothing. What it is, is headline-bait that’s not obviously wrong. But is that the appropriate standard? It’s not obviously wrong to three referees, so publish it? Psychological Science is “the flagship journal of the Association for Psychological Science . . . the highest ranked empirical journal in psychology.”

As a statistician, my advice is: if a paper is nothing special, you don’t have to publish it in your flagship journal. In this as with the notorious Daryl Bem article, I feel that the journal almost seemed to feel an obligation to publish a dubious claim, just because some referees didn’t happen to find any flaws in the data collection or analysis. But if Psychological Science does not publish this article, it’s not censorship or suppression; the authors can feel free to submit it to a lesser journal. For the leading journal to have such low standards, this is bad news for the entire field. For one thing, it encourages future researchers to focus on this sort of sloppy work.

I’m hoping the above advice will make Stephen Olivier happy. I’m not just sitting there criticizing something, or telling someone to use R instead of SPSS, or lecturing psychologists about their lack of mathematical skills. I’m giving some very specific suggestions that you, the psychology researcher, can use in your next research project (or, if you’re a journal editor, in your next publication decision).

There is, of course, lots and lots of additional advice that I and other statisticians could give. The above is just a start. But I wanted to start somewhere, just to demonstrate to Olivier (and others) that this is indeed possible.

P.S. Blogger Echidne raised similar points last year.

The post How can statisticians help psychologists do their research better? appeared first on Statistical Modeling, Causal Inference, and Social Science.

The field of statistics continues to be divided into competing schools of thought. In theory one might imagine choosing the uniquely best method for each problem as it arises, but in practice we choose for ourselves (and recom- mend to others) default principles, models, and methods to be used in a wide variety of settings. This article briefly considers the informal criteria we use to decide what methods to use and what principles to apply in statistics problems.

And then I follow up with these sections:

Statistics: the science of defaults

Ways of knowing

The pluralist’s dilemma

And here’s the concluding paragraph:

Statistics is a young science in which progress is being made in many areas. Some methods in common use are many decades or even centuries old, but recent and current developments in nonparametric modeling, regularization, and multivariate analysis are central to state-of-the-art practice in many areas of applied statistics, ranging from psychometrics to genetics to predictive modeling in business and social science. Practitioners have a wide variety of statistical approaches to choose from, and researchers have many potential directions to study. A casual and introspective review suggests that there are many different criteria we use to decide that a statistical method is worthy of routine use. Those of us who lean on particular ways of knowing (which might include: performance on benchmark problems, success in new applications, insight into toy problems, optimality as shown by simulation studies or mathematical proofs, or success in the marketplace) should remain aware of the relevance of all these dimensions in the spread of default procedures.

Regular blog readers will recognize many of these themes, but I hope this particular presentation has some added value. And this is as good a place as any to thank my many correspondents who’ve helped contribute to the development and expression of these ideas.

The post How do we choose our default methods? appeared first on Statistical Modeling, Causal Inference, and Social Science.

Anyway, the full article with its 34 discussions just appeared in the journal. Here it is.

What surprised me, in reading the full discussion, was how supportive the commentary was. Given the topic of Pothos and Busemeyer’s article, I was expecting the discussions to range from gentle mockery to outright abuse. The discussion that Mike and I wrote was moderately encouraging, and I was expecting this to fall on the extreme positive end of the spectrum.

Actually, though, most of the discussions were positive, and only a couple were purely negative (those would be “Quantum models of cognition as Orwellian newspeak” by Michael Lee and Wolf Vanpaemel, and “Physics envy: Trying to fit a square peg into a round hole,” by James Shenteau and David Weiss). We expressed some vague skepticism but it’s hard for me to be really negative about the idea, given that classical probability theory is not actually correct, and we do indeed live in a quantum world (otherwise all our tables and chairs would fall apart, for one thing). I certainly see no logical reason why our models of probability and uncertainty should be restricted to the “Boltzmannian” simplification.

The post Does quantum uncertainty have a place in everyday applied statistics? appeared first on Statistical Modeling, Causal Inference, and Social Science.

Marginally specified priors for nonparametric Bayesian estimation

Prior specification for nonparametric Bayesian inference involves the difficult task of quantifying prior knowledge about a parameter of high, often infinite, dimension. Realistically, a statistician is unlikely to have informed opinions about all aspects of such a parameter, but may have real information about functionals of the parameter, such the population mean or variance. This article proposes a new framework for nonparametric Bayes inference in which the prior distribution for a possibly infinite-dimensional parameter is decomposed into two parts: an informative prior on a finite set of functionals, and a nonparametric conditional prior for the parameter given the functionals. Such priors can be easily constructed from standard nonparametric prior distributions in common use, and inherit the large support of the standard priors upon which they are based. Additionally, posterior approximations under these informative priors can generally be made via minor adjustments to existing Markov chain approximation algorithms for standard nonparametric prior distributions. We illustrate the use of such priors in the context of multivariate density estimation using Dirichlet process mixture models, and in the modeling of high-dimensional sparse contingency tables.

This seems very important to me, also I love the idea of Hoff and Dunson on the same paper, sort of like one of those 70′s supergroups.

The post Reputations changeable, situations tolerable appeared first on Statistical Modeling, Causal Inference, and Social Science.

So here goes.

Jarno Vanhatalo, Jaakko Riihimäki, Jouni Hartikainen, Pasi Jylänki, Ville Tolvanen, and Aki Vehtari write:

The GPstuff toolbox is a versatile collection of Gaussian process models and computational tools required for Bayesian inference. The tools include, among others, various inference methods, sparse approximations and model assessment methods.

We can actually now fit Gaussian processes in Stan. But for big problems (or even moderately-sized problems), full Bayes can be slow. GPstuff uses EP, which is faster. At some point we’d like to implement EP in Stan. (Right now we’re working with Dave Blei to implement VB.)

GPstuff really works. I saw Aki use it to fit a nonparametric version of the Bangladesh well-switching example in ARM. He was sitting in his office and just whipped up the model and fit it.

The post GPstuff: Bayesian Modeling with Gaussian Processes appeared first on Statistical Modeling, Causal Inference, and Social Science.

I thought you’d all like to know that Stan was used and referenced in a peer-reviewed Rapid Communications paper on influenza. Thank you for this excellent modelling language and sampler, which made it possible to carry out this work quickly!

I haven’t actually read the paper, but I’m happy to see Stan getting around like that.

The post Stan! appeared first on Statistical Modeling, Causal Inference, and Social Science.

David Jinkins writes:

The objective of this paper is to measure the relative importance of conspicous consumption to Americans and Chinese. To this end, I estimate the parameters of a utility function borrowed from recent theoretical work using American and Chinese data. The main parameter of interest governs the amount that individuals care about peer group beliefs regarding their welfare. Using survey data on the visibility of different good categories along with household budget surveys, I find that Chinese consumers care twice as much as American consumers about the beliefs of their peer group.

I came across this draft research manuscript by following the links back after Jinkins commented on our blog. The framing of the paper is a bit more foundation-y and a bit less statistic-y than I’d prefer, but I guess that’s just the way they do things in economics, compared to statistics or (some) political science. In any case, I wanted to point you to this paper, partly to let you know that I do read the blog comments and even sometimes follow the links, and also because it’s unusually well-written, not just in its first paragraph but all the way through. He’s got to work a bit on his presentation of results—I see some ugly tables there—but I think that’s much easier to learn than it is to learn how to write.

The post A Structural Comparison of Conspicuous Consumption in China and the United States appeared first on Statistical Modeling, Causal Inference, and Social Science.

Poderomedia Foundation and PinLatam are launching OpenDataLatinoamerica.org, a regional data repository to free data and use it on Hackathons and other activities by HacksHackers chapters and other organizations.

We are doing this because the road to the future of news has been littered with lost datasets. A day or so after every hackathon and meeting where a group has come together to analyze, compare and understand a particular set of data, someone tries to remember where the successful files were stored. Too often, no one is certain. Therefore with Mariano Blejman we realized that we need a central repository where you can share the data that you have proved to be reliable: OpenData Latinoamerica, which we are leading as ICFJ Knight International Journalism Fellows.

If you work in Latin America or Central America your organization can take part in OpenDataLatinoamerica.org. To apply, go to the website and answer a simple form agreeing to meet the standard criteria for open data. Once the application is approved, you will receive an account to start running and managing open data, becoming part of the community.

The post OpenData Latinoamerica appeared first on Statistical Modeling, Causal Inference, and Social Science.

Following up on this post by Noah Smith on economics in science fiction, Mark Palko writes on economics in crime fiction.

Just as almost all science fiction is ultimately about politics, one could say that just about all crime fiction is about economics.

But if I had to pick one crime novelist with an economics focus, I’d pick George V. Higgins. In one of his novels, his character Jerry Kennedy had a riff on the difference between guys who get a salary and guys who have to work for every dollar. But, really, almost all his novels are full of economics.

The post Crime novels for economists appeared first on Statistical Modeling, Causal Inference, and Social Science.

Tom Salvesen asks, is this the worst info-graphic of the year?

I say, no. Nobody really cares about these numbers. It’s an amusing feature. The alternative would not be a better display of these data, the alternative would be some photo or cartoon. They’re just having fun. I wouldn’t give it any design awards but it’s fine, it is what it is.

The post Actually, I have no problem with this graph appeared first on Statistical Modeling, Causal Inference, and Social Science.

The “best” picture of 2012 was Argo. At least that’s the film that won the Oscar for best picture. According to the Oscars, the decision to give this award to Argo was made by the nearly 6,000 voting members of the Academy of Motion Picture Arts and Sciences. . . . In other words, this choice is made by the “experts.” There is, though, another group that we could have listened to on Sunday night. That group would be the people who actually spend money to go to the movies. . . . According to that group, Marvel’s the Avengers was the “best” picture in 2012. With domestic revenues in excess of $600 million, this filmed earned nearly $200 million more than any other picture. And when we look at world-wide revenues, this film brought in more than $1.5 billion. . . . Despite what seems like a clear endorsement by the customers of this industry, the Avengers was ignored by the Oscars. Perhaps this is just because I am an economist, but this strikes me as odd. Movies are not a product made just for the members the academy. These ventures are primarily made for the general public. And yet, when it comes time to decide which picture is “best,” the opinion of the general public seems to be ignored. Essentially the Oscars are an industry statement to their customers that says: “We don’t think our customers are smart enough to tell us which of our products are good. So we created a ceremony to correct our customers.”

He keeps going along those lines for awhile and concludes:

One would hope the Academy would at least pay a bit more attention to the people paying the bills. Not only does it seem wrong (at least to this economist) to argue that movies many people like are simply not that good, focusing on the box office would seem to make good financial sense for the Oscars as well. A recent Slate article argued that the Oscars’ telecast tends to have higher ratings when more commercially successful films are nominated for best picture. So in the future, maybe voters for the Oscars will pay a bit more attention to their customers. These customers may not be thought of as “movie experts.” But these are the people who pay the bills, and therefore, ultimately it is their opinion that should matter to this industry.

What strikes me about this discussion is the mix of descriptive and normative that seems so characteristic of pop-microeconomics. (I should emphasize here that I’m not using “pop” in any sort of derogatory way. I’m speaking of serious economic writing that is intended for a popular audience.)

1. On one hand, you have the purely descriptive perspective: economist as person-from-Mars, looking at human society objectively, the way a scientist studies cell cultures in a test tube. Consumer sovereignty is what it’s all about, with a slightly offended tone that anyone could think otherwise. Who are you, smartypants, to think you know better than the average ticket-buyer, etc. I’m reminded of the perhaps-apocryphal story of the “some academics” who “conclude that bookmakers simply aren’t very smart.”

2. At the same time, we’re given a moral lesson. The Avengers is the best movie because it made more money. It is “the people who pay the bills” whose “opinion that should matter to this industry.”

The difficulty, of course, is that lesson 2 gets blurred if it is folded into lesson 1.

Berri’s argument is that moviemakers should not be paternalistically ignoring the attitudes of their customers in giving awards. But this argument dissolves if you take one step back and consider moviemakers as independent business operators. In that case, their business decisions (to do the Oscars however they want) should be given as much respect as that of moviegoers to choose which movies to watch.

As far as I’m concerned, the Academy can do whatever they want. What’s interesting to me here is to see how the economist’s explicitly non-normative ideology (his implication that the “best” picture must be the one with most revenue, and that any other criteria would be disrespectful of moviegoers) so quickly becomes normative (that it’s “wrong . . . to argue that movies many people like are simply not that good”). To me, it’s a strange mixture of idealism and cynicism. The man from Mars has become a scold.

The post The recursion of pop-econ appeared first on Statistical Modeling, Causal Inference, and Social Science.

The one thing you might want to add, to fit with your pragmatic perspective, is to point out that these different methods are optimal under different assumptions about the data. However, these assumptions are never true (even in the rare cases where you have a believable prior, it won’t really follow the functional form assumed by bayesglm; even in the rare cases where you have a real loss function, it won’t really follow the mathematical form assumed by lasso etc), but these methods can still be useful and be given the interpretation of regularized estimates.

Another thing that someone might naively think is that regularization is fine but “unbiased” is somehow the most honest. In practice, if you stick to “unbiased” methods such as least squares, you’ll restrict the number of variables you can include in your model. So in reality you suffer from omitted-variable bias. So there is not safe home base. It’s not like the user can simply do unregularized regression and then think of regularization as a frill. The practitioner who uses unregularized regression has already essentially made a compromise with the devil by restricting the number of predictors in the model to a “manageable” level (whatever that means).

The post Same old same old appeared first on Statistical Modeling, Causal Inference, and Social Science.

Part 1: The story of an unsuccessful discussion

The first time I had a discussion article was the result of an unfortunate circumstance. I had a research idea that resulted in an article with Don Rubin on monitoring the mixing of Markov chain simulations. I new the idea was great, but back then we worked pretty slowly so it was awhile before we had a final version to submit to a journal. (In retrospect I wish I’d just submitted the draft version as it was.) In the meantime I presented the paper at a conference. Our idea was very well received (I had a sheet of paper so people could write their names and addresses to get preprints, and we got either 50 or 150 (I can’t remember which, I guess it must have been 50) requests), but there was one person who came up later and tried to shoot down our idea. The shooter-down, Charlie Geyer, has done some great work but in this case he was confused, I think in retrospect because we did not have a clear discussion of the different inferential goals that arose in the sorts of calculations he was doing (inference for normalizing constants of distributions) and which I was doing (inference for parameters in fitted models). In any case, the result was that our new and exciting method was surrounded by an air of controversy. In some ways that was a good thing: I became well known in the field right away, perhaps more than I deserved at the time (in the sense that most of my papers up to then and for the next few years were on applied topics; it was awhile before I published other major papers on statistical theory, methods, and computation). But overall I’d rather have been moderately known for an excellent piece of research than very well known for being part of a controversy. I didn’t seek out controversy; it arose because someone else criticized our work without seeing the big picture, and at the time neither he nor I nor my collaborator had the correct synthesis of my work and his criticism.

(Again, the synthesis was that he was trying to get precise answers for hard problems and was in a position where he needed to have a good understanding of the complex distributions he was simulating from, whereas I was working on a method to apply routinely in relatively easy (but nontrivial!) settings. For Charlie’s problems, my method would not suffice because he wouldn’t be satisfied until he was directly convinced that the Markov chain was exploring all the space. For my problems, Charlie’s approach (to run a million simulations and work really hard to understand the computation for a particular model) wasn’t a practical solution. His approach to applied statistics was to handcraft big battleships to solve large problems, one at a time. I wanted to fit lots of small and medium-sized models (along with the occasional big one), fast.)

Anyway, this “different methods for different goals” conversation never occurred, hence I left that meeting with an unpleasant feeling that our method was controversial, not fully accepted, and not fully understood. So I got it into my head that our article should be published as a discussion, so that Geyer and others could comment and we could respond.

But we never had that discussion, not in those words. Neither Charlie nor I nor Don Rubin was aware enough of the sociological context, as it were, so we ended up talking past each other.

In retrospect, that particular discussion did not work so well.

Here’s another example from about the same time, the Ising model. Here’s one chain from the Gibbs sampler. After 2000 iterations, it looks like it’s settled down to convergence (here we’re plotting the log probability density, which is a commonly used summary for this sort of distribution).

But then look at the second plot: the first 500 iterations. If we’d only seen these, we might have been tempted to declare victory too early!

At this point, the naive take-home point might be that 500 iterations was not enough but we’re safe with 2000. But no! Even that the last bit of those 2000 looks as stationary and clean as can be, if we start from a different point and run for 2000, we get something different:

This one looks stationary too! But a careful comparison with the graphs above (even clearer when I displayed these on transparency sheets and overlaid them on the projector) reveals that the two “stationary” distributions are different. The chains haven’t mixed, the process hasn’t converged. R-hat reveals this right away (without even having to look at the graphs, but you can look at the graphs if you want).

As I wrote in our article in Bayesian Statistics 4,

This example shows that the Gibbs sampler can stay in a small subset of its space for a long time, without any evidence of this problematic behavior being provided by one simulated series of finite length. The simplest way to run into trouble is with a two-chambered space, in which the probability of switching chambers is very low, but the above graphs are especially disturbing because the probability density in the Ising model has a unimodal (in the sense that this means anything in a discrete distribution) and approximately Gaussian marginal distribution on the gross scale of interest. That is, the example is not pathological; the Gibbs sampler is just very slow. Rather than being a worst-case example, the Ising model is typical of the probability distributions for which iterative simulation methods were designed, and may be typical of many posterior distributions to which the Gibbs sampler is being applied.

So that was my perspective: start from one point and the chain looks fine; start from two points and you see the problem. But Charlie had a different attitude toward the Ising example. His take on it was: the Ising model is known to be difficult, no one but a fool would try to simulate it with 2000 iterations of a Gibbs sampler. There’s a huge literature on the Ising model already!

Charlie was interested in methods for solving large, well-understood problems one at a time. I was interested in methods that would be used for all sorts of problems by statisticians such as myself who, for applied reasons, bite off more in model than we can chew in computation and understanding. For Charlie with the Ising model, multiple sequences missed the point entirely, as he knew already that 2000 iterations of Gibbs wouldn’t do it. For me, though . . . as an applied guy I was just the kind of knucklehead who might apply Gibbs to this sort of problem (in my defense, Geman and Geman made a similar mistake in 1984, I’ve been told), so it was good to have a practical convergence check.

Again, I think that in our discussion and rejoinder, Don and I presented our method well, in the context of our applied purposes. But I think it would’ve worked better as a straight statistics article. Nothing much useful came out of the discussion because none of us cut through to the key difference in the sorts of problems we were working on.

Part 2: A successful discussion

In the years since then, I’ve realized that communication is more than being right (or, should I say, thinking that one is right). Statistical ideas (and, for that matter, mathematical and scientific ideas in general) are sometimes best understood through their limitations. It’s Lakatos’s old “proofs and refutations” story all over again.

Recently I was involved in a discussion that worked out well. It started a few years ago with a post of mine on the differences between the sorts of data visualizations that go viral on the web (using some examples that were celebrated by statistician/designer Nathan Yau), as compared to statistical graphics of the sort that we are trained to make. It seemed to me that many visualizations that are successful with general audiences feature unique or striking designs and puzzle-like configurations, whereas the most successful statistical graphics have more transparent formats that foreground data comparisons. Somewhere in between are the visualizations created by lab scientists, who generally try to follow statistical principles but usually (in my view) try too hard to display too much information on a single plot.

My posts, and various follow-ups, were disliked by many in the visualization community. They didn’t ever quite disagree with my claim that many successful visualizations involve puzzles, but they didn’t like what they perceived as my negative tone.

In attempt to engage the fields of statistics and visualization more directly, I wrote an article (with Antony Unwin) on the different goals and different looks of these two sorts of graphics. Like many of my favorite papers, this one took a lot of effort to get into a journal. But finally it was accepted in the Journal of Computational and Graphical Statistics, with discussion.

The discussants (Stephen Few, Robert Kosara, Paul Murrell, and Hadley Wickham; links to all four discussions are here on Kosara’s blog) politely agreed with us on some points and disagreed with us on others. And then it was time for us to write our rejoinder.

In composing the rejoinder I finally came upon a good framing of the problem. Before we’d spoken of statistical graphs and information visualization as having different goals and looking different. But that didn’t work. No matter how often I said that it could be a good thing that an infovis is puzzle-like, or no matter how often I said that as a statistician I would prefer graphing the data like This but I can understand how graphing it like That could attract more viewers . . . no matter how much I said this sort of thing, it was interpreted as a value judgment (and it didn’t help when I said that something “sucked,” even if I later modified that statement).

Anyway, my new framing, that I really like, is in terms of tradeoffs. Not “two cultures,” not “different goals, different looks,” but tradeoffs. So it’s not stat versus infographics; instead it’s any of us trying to construct a graph (or, better still, a grid of graphs) and recognizing that it’s not generally possible to satisfy all goals at once, so we have to think about what goals are most important in any given situation:

In the internet age, we should not have to choose between attractive graphs and informational graphs: it should be possible to display both, via interactive displays. But to follow this suggestion, one must first accept that not every beautiful graph is informative, and not every informative graph is beautiful.

Yes, it can sometimes be possible for a graph to be both beautiful and informative, as in Minard’s famous Napoleon-in-Russia map, or more recently the Baby Name Wizard, which we featured in our article. But such synergy is not always possible, and we believe that an approach to data graphics that focuses on celebrating such wonderful examples can mislead people by obscuring the tradeoffs between the goals of visual appeal to outsiders and statistical communication to experts.

So it’s not Us versus Them, it’s each of us choosing a different point along the efficient frontier for each problem we care about.

And I think the framing worked well. At least, it helped us communicate with Robert Kosara, one of our discussants. Here’s what Kosara wrote, after seeing our article, the discussions (including his), and our rejoinder:

There are many, many statements in that article [by Gelman and Unwin] that just ask to be debunked . . . I [Kosara] ended up writing a short response that mostly points to the big picture of what InfoVis really is, and that gives some examples of the many things they missed.

While the original article is rather infuriating, the rejoinder is a great example of why this kind of conversation is so valuable. Gelman and Unwin respond very thoughtfully to the comments, seem to have a much more accurate view of information visualization than they used to, and make some good points in response.

Great! A discussion that worked! This is how it’s supposed to go: not a point-scoring debate, not people talking past each other, but an honest and open discussion.

Reflections

Perhaps my extremely, extremely frustrating experience early in my career (detailed in Part 1 above) motivated me to think seriously about the Lakatosian attitude toward understanding and explaining ideas. If you compare Bayesian Data Analysis to other statistics books of that era, for example, I think we did a pretty good job (although maybe not good enough) of understanding the methods through their limitations. But even with all my experience and all my efforts, this can be difficult, as revealed by the years it took for us to finally process our ideas on graphics and visualization to the extent that we could communicate with experts in these fields.

The post A tale of two discussion papers appeared first on Statistical Modeling, Causal Inference, and Social Science.

Gary Marcus writes,

An algorithm that is good at chess won’t help parsing sentences, and one that parses sentences likely won’t be much help playing chess.

That is soooo true. I’m excellent at parsing sentences but I’m not so great at chess. And, worse than that, my chess ability seems to be declining from year to year.

Which reminds me: I recently read Frank Brady’s much lauded Endgame, a biography of Bobby Fischer. The first few chapters were great, not just the Cinderella story of his steps to the world championship, but also the background on his childhood and the stories of the games and tournaments that he lost along the way.

But after Fischer beats Spassky in 1972, the book just dies. Brady has chapter after chapter on Fisher’s life, his paranoia, his girlfriends, his travels. But, really, after the chess is over, it’s just sad and kind of boring. I’d much rather have had twice as much detail on the first part of the life and then had the post-1972 era compressed into a single chapter. I mean, sure, I respect that Brady wanted to tell the full life story, and I’m not telling him how he should’ve written his book, I’m just giving my reactions.

Also, I would’ve liked more information on the games: what was the amazing set of moves that Fischer did in the so-called Game of the Century, what happened in some of the games he lost, and so on. In an afterword, Brady writes that he decided not to include any games so as to make the book more accessible. What I wonder is, how many readers are there like me, who enjoy chess, could understand a diagram and some discussion of what these amazing plays were, even if we couldn’t follow an entire game written on the page or have the patience to play one out on the board. I wouldn’t have gotten much out of transcripts of chess games, but a few diagrams and discussions of key moments, that would’ve made the book a lot more interesting to me.

P.S. After Kasparov beat Karpov in the final game of their tournament—the game where both players knew that Kasparov had to win, that a draw wouldn’t be enough—I clipped the game out of the newspaper and later played it out with my dad. That was a game. To my ignorant eyes, there was no single point where I could spot a mistake by Karpov. Kasparov just gradually and imperceptibly got to a winning position. Amazing.

The post Of parsing and chess appeared first on Statistical Modeling, Causal Inference, and Social Science.

Ferguson reiterates that his remarks were “stupid.” The question then arises: He’s a smart guy, how did he end up saying such stupid things? Ferguson has a history of saying high-profile stupid things, and they always seem to be when he’s trying to make some sort of political point.

I’m still going with my theory that Ferguson misjudged his audience; he thought they’d appreciate an anti-Keynes remark, maybe he even thought they were the kind of crowd that would enjoy cracks about gay people who like ballet and poetry. No go.

Again, I’m not trying to nail the guy to the cross for this. We all make mistakes; in fact, we all make mistakes, of some sort, repeatedly. It’s just interesting to think about what made him say this stuff in the first place. He’s not any kind of “gay-basher” (in his words), but he still thought this sort of thing would work. So maybe he was trying a bit too hard to please the crowd. He should remember that, however entertaining he is as a speaker, however good he looks on TV, his ultimate qualifications come from his historical research.

Perhaps the case interests me so much because Ferguson, like me, is an academic researcher who likes to speak to popular audiences.

P.S. More here from Willam Black, including this amazing bit that Ferguson wrote about Paul Krugman:

His inability to debate a question without insulting his opponent suggests some kind of deep insecurity perhaps the result of a childhood trauma.

Black makes another good point:

Ferguson loves counterfactuals, so let’s try this counterfactual. What if Ferguson had made the obvious, stupid claim that people who are childless do not carry about future generations? We know that he despises Keynes and Krugman (in the same talk he archly referred to Krugman as his “arch-enemy”) and that both are childless. Ferguson could have claimed that Keynes and Krugman were indifferent to the lives of future generations because they were childless. How would the audience have reacted to such a claim?

The perhaps 25% of the audience who were childless would have stared at him like he was an idiot who had gone out of his way to insult them. The Americans in the audience would have thought first of themselves if they were childless, then of their relatives and close friends who were childless, then of George Washington, and finally of Jesus. . . .

P.P.S. For those who don’t get the title of this post, see here.

The post Like Casper the ghost, Niall Ferguson is not only white. He is also very, very adorable. appeared first on Statistical Modeling, Causal Inference, and Social Science.

Mark Palko writes:

Salmon is dismissive of the claim that there are fifty million over-the-air television viewers:

The 50 million number, by the way, should not be considered particularly reliable: it’s Aereo’s guess as to the number of people who ever watch free-to-air TV, even if they mainly watch cable or satellite. (Maybe they have a hut somewhere with an old rabbit-ear TV in it.)

And he strongly suggests the number is not only smaller but shrinking. By comparison, here’s a story from the broadcasting news site TV News Check from June of last year (if anyone has more recent numbers please let me know):

According to new research by GfK Media, the number of Americans now relying solely on over-the-air (OTA) television reception increased to almost 54 million, up from 46 million just a year ago. The recently completed survey also found that the demographics of broadcast-only households skew towards younger adults, minorities and lower-income families.

As Palko says, Salmon is usually a pretty careful reporter. And this one should be right up his alley. Here’s Palko again:

We’ve talked about how well over-the-air television compares to cable (for some people), how new and apparently successful businesses are springing up around OTA, and how the number of viewers getting their television through antennas appears to have been growing substantially since the introduction of digital. What we haven’t covered so far is the potential social impact of killing broadcast television.

It is almost axiomatic that, if you have a resource that is used in one way by people at the top of the economic ladder and in another way by people on the bottom and you “let the market decide” what to do with the resource, it will go with the people who have the money. . . .

This becomes particularly troubling when we’re talking about a publicly held resource. . . . What groups rely heavily on broadcast television? What groups would have the most difficulty finding alternatives?

People in the bottom one or two deciles are going to be in trouble. Even the lowest tier of cable would represent a significant monthly expense. People with limited residential security will be even worse off. People with limited income security will face a difficult choice: sign up for exorbitant no-contract plans or commit to a financial obligation they may not be able to fulfill. People with poor credit histories will have to come up with large deposits every time they move. . . .

Palko summarizes:

OTA [over-the-air television] is a promising technology supporting an innovative and growing industry, serving important economic and social roles.

The technology is doing fine in the marketplace. It’s lobbyists who are likely to kill it.

I wonder what Salmon’s take is on this. Is Palko missing something, or does he just happen to be sharing a perspective that is different from that of NYC-based financial journalists?

P.S. Let me emphasize that this post is not some sort of trolling of Felix Salmon. I’m a big fan of his quantitatively sophisticated reporting, which is why it’s interesting if he’s getting something wrong.

P.P.S. There’s some dispute about that 54 million number. Salmon points to this news article by Michael Grotticelli:

Free, over-the-air television viewing of broadcast TV signals are now watched by only 9 percent of the U.S. population — down from 16 percent in 2003, according to Nielsen, the major TV and radio rating service. . . .

The Nielsen numbers are certain to cause a dispute with the NAB, which has insisted the amount of over-the-air viewing is increasing in an era of cord-cutting. Last summer, the NAB produced a survey by Knowledge Networks citing about 18 percent as “broadcast exclusive” households. That total was 54 million Americans — up from 46 million in 2011.

So, one claim is that 9% watch any over-the-air TV, the other is that 18% only watch over-the-air TV. That’s a big gap.

The post Is Felix Salmon wrong on free TV? appeared first on Statistical Modeling, Causal Inference, and Social Science.

Social science research has been getting pretty bad press recently, what with the Excel buccaneers who didn’t know how to handle data with different numbers of observations per country, and the psychologist who published dozens of papers based on fabricated data, and the Evilicious guy who wouldn’t let people review his data tapes, etc etc. And that’s not even considering Dr. Anil Potti.

On the other hand, the revelation of all these problems can be taken as evidence that things are getting better. Psychology researcher Gary Marcus writes:

There is something positive that has come out of the crisis of replicability—something vitally important for all experimental sciences. For years, it was extremely difficult to publish a direct replication, or a failure to replicate an experiment, in a good journal. . . . Now, happily, the scientific culture has changed. . . . The Reproducibility Project, from the Center for Open Science is now underway . . .

And sociologist Fabio Rojas writes:

People may sneer at the social sciences, but they hold up as well. Recently, a well known study in economics was found to be in error. People may laugh because it was an Excel error, but there’s a deeper point. There was data, it could be obtained, and it could be replicated. Fixing errors and looking for mistakes is the hallmark of science. . . .

I agree with Marcus and Rojas that attention to problems of replication is a good thing. It’s bad that people are running incompetent analysis or faking data all over the place, but it’s good that they’re getting caught. And, to the extent that scientific practices are improving to help detect error and fraud, and to reduce the incentives for publishing erroneous and fradulent results in the first place, that’s good too.

But I worry about a sense of complacency. I think we should be careful not to overstate the importance of our first steps. We may be going in the right direction but we have a lot further to go. Here are some examples:

1. Marcus writes of the new culture of publishing replications. I assume he’d support the ready publications of corrections, too. But we’re not there yet, as this story indicates:

Recently I sent a letter to the editor to a major social science journal pointing out a problem in an article they’d published, they refused to publish my letter, not because of any argument that I was incorrect, but because they judged my letter to not be in the top 10% of submissions to the journal. I’m sure my letter was indeed not in the top 10% of submissions, but the journal’s attitude presents a serious problem, if the bar to publication of a correction is so high. That’s a disincentive for the journal to publish corrections, a disincentive for outsiders such as myself to write corrections, and a disincentive for researchers to be careful in the first place. Just to be clear: I’m not complaining how I was treated here; rather, I’m griping about the system in which a known error can stand uncorrected in a top journal, just because nobody managed to send in a correction that’s in the top 10% of journal submissions.

2. Rojas writes of the notorious Reinhardt and Rogoff study that, “There was data, it could be obtained, and it could be replicated.” Not so fast:

It was over two years before those economists shared the data that allowed people to find the problems in their study. If the system really worked, people wouldn’t have had to struggle for years to try to replicate an unreplicable analysis.

And, remember, the problem with that paper was not just a silly computer error. Reinhardt and Rogoff also made serious mistakes handling their time-series cross-sectional data.

3. Marcus writes in a confident tone about progress in methodology: “just last week, Uri Simonsohn [and Leif Nelson and Joseph Simmons] released a paper on coping with the famous file-drawer problem, in which failed studies have historically been underreported.” I think Uri Simonsohn is great, but I agree with the recent paper by Christopher Ferguson and Moritz Heene that the so-called file-drawer problem is not a little technical issue that can be easily cleaned up; rather, it’s fundamental to our current practice of statistically-based science.

And there’s pushback. Biostatisticians Leah Jager and Jeffrey Leek wrote a paper, which I strongly disagree with, called “Empirical estimates suggest most published medical research is true.” I won’t go into the details here—my take on their work is that they’re applying a method that can make sense in the context of a single large study but which won’t generally work with meta-analysis—my point is that there remains a constituency for arguments that science is basically OK already.

I respect the view of Marcus, Rojas, Jager, Leek, and others that the current environment of criticism has in some ways gone too far. All those people do serious, respected research, and those of us who do serious research know how difficult it can be to publish in good journals, how hard we work—out of necessity—to consider all possible alternative explanations for any results we find, how carefully we document the steps of our data collection and analysis, and so forth. But many problems still remain.

Thomas Basbøll analogizes the difficulties of publishing scientific criticism to problems with the subprime mortgage market before the crash. He quotes Michael Lewis:

To sell a stock or bond short you need to borrow it, and [the bonds they were interested in] were tiny and impossible to find. You could buy them or not buy them but you couldn’t bet explicitly against them; the market for subprime mortages simply had no place for people in it who took a dim view of them. You might know with certainty that the entire mortgage bond market was doomed, but you could do nothing about it.

And now here’s Basbøll:

I had a shock of recognition when I read that. I’ve been trying to “bet against” a number of stories that have been told in the organization studies literature for years now, and the thing I’m learning is that there’s no place in the literature for people who take a dim view of them. There isn’t really a genre (in the area of management studies) of papers that only points out errors in other people’s work. You have to make a “contribution” too. In a sense, you can buy the stories people are telling you or not buy them but you can’t criticize them.

This got me thinking about the difference between faith and knowledge. Knowledge, it seems to me, is a belief held in a critical environment. Faith, we might say, is a belief held in an “evangelical” environment. The mortgage bond market was an evangelical environment in which to hold beliefs about housing prices, default rates, and credit ratings on CDOs. There was no simple way to critique the “good news” . . .

Eventually, as Lewis reports, people were able to bet against the subprime mortgage market, but it wasn’t easy. And the fact that some investors, with great difficulty, were able to do it, doesn’t mean the financial system is A-OK.

Basbøll’s analogy may be going too far, but I agree with his general point that the existence of a few cases of exposure should not make us complacent. Marcus’s suggestions on cleaning up science are good ones, and we have a ways to go before they are generally implemented.

P.S. Coincidentally, Jeff Leek posted something today on the same topic, but with a slightly different perspective (he refers to “the current over-pessimism about science”). Leek argues, reasonably enough, “that people are using a few high-profile cases to hyperventilate about the real, solvable, and recognized problems in the scientific process” and he worries that “the rational reasonable problems we have, with enough hyperbole, will make it look like the scientific process ‘sky is falling’” and lend support to political attacks on science more generally. I think Jeff and I should be able to agree to the following:

- Science is hard, we all make mistakes, the system has problems but all human systems have problems, in working to fix these problems we shouldn’t thrown the research baby out with the bathwater that is the changing rules of scientific communication.

- We’re not there yet, we still live in a world in which it’s easier to publish and hype a elaborate flawed claim than to report a simple correction, a world in which data sharing is far from the norm, and where social and statistical biases lead to systematic overreporting of dramatic claims and systematic overestimation of effect sizes.

Leek is making the valid point that the sort of doomsaying that has been needed to draw attention to problems in scientific communication and to motivate improvements, can also be used, in guilt-by-association sense, to disparage good science. And, even in popular culture, my impression is that things aren’t as bad as they used to be. Sure, vaccine deniers and global warming deniers and all the other deniers are out there, but it’s not like the 70s when people were buying millions of copies of Chariots of the Gods, The Jupiter Effect, and The Bermuda Triangle, right?

The post Against optimism about social science appeared first on Statistical Modeling, Causal Inference, and Social Science.

P.S. Marcus updates with a glass-is-half-full take.

The post Cleaning up science appeared first on Statistical Modeling, Causal Inference, and Social Science.

I have too much of a technical bent to be the ideal reader for this sort of book, but it seems like an excellent gift for a non-technical reader who nonetheless enjoys math. (I assume that such people are out there, just as there are people like me who can’t read music but still enjoy reading about the subject.)

The book is organized by topic. My own preference would have been chronological and with more old stuff. I particularly enjoyed the material from many decades ago, such as the news report on one of the early computers. This must have been a fun book to compile.

The post The New York Times Book of Mathematics appeared first on Statistical Modeling, Causal Inference, and Social Science.

We had some interesting comments on our recent reflections on Niall Ferguson’s ill-chosen remarks in which he attributed Keynes’s economic views (I don’t actually know exactly what Keyesianism is, but I think a key part is for the government to run surpluses during economic booms and deficits during recessions) to the Keynes being gay and marrying a ballerina and talking about poetry. The general idea, I think, is that people without kids don’t care so much about the future, and this motivated Keynes’s party-all-the-time attitude, which might have worked just fine for Eddie Murphy’s girlfriend in the 1980s and in San Francisco bathhouses of the 1970s but, according to Ferguson, is not the ticket for preserving today’s American empire.

Some of the more robust defenders of Ferguson may have been disappointed by his followup remarks: “I should not have suggested . . . that Keynes was indifferent to the long run because he had no children, nor that he had no children because he was gay. This was doubly stupid. . . . My disagreements with Keynes’s economic philosophy have never had anything to do with his sexual orientation. It is simply false to suggest, as I did, that his approach to economic policy was inspired by any aspect of his personal life.” It’s tough to try to defend a statement that was disowned by the person saying it.

But the question then arises: What’s so horrible about what Ferguson said? After all, it’s not unreasonable to think that someone’s personal circumstances will affect their political attitudes and their views on economic policy. And certainly no one doubts that Keynes’s background as an upper-class British backgrounds was relevant for understanding his views.

So what was up?

I think the problem was that Ferguson did not seem to be engaging with an open mind but rather seemed to be just trying to throw mud at Keynes as if he (Ferguson) were operating a political campaign rather than engaging in academic inquiry. If Ferguson were to give a talk all about the connections between the personal circumstances and political beliefs of historical economists, I don’t think people would have a problem with it. He could mention Keynes’s sex life, Adam Smith’s pets, and anything else that might be relevant. Ferguson is a historian (or, as I assume he would say, an historian), and he’d be highly qualified to do that sort of thing. But to just throw culture-war words at his audience in an attempt to vaguely disparage Keynes, that’s just stupid. Again, nobody’s perfect, and Ferguson himself recognized the stupidity of his remarks. I don’t think this will, or should, deter historians from connecting the lives and ideas of famous thinkers. But it doesn’t look like that’s what was being done here; rather, Ferguson was doin some insinsuatin as a way to discredit a political opponent, which works better on the campaign trail than when coming from a scholar.

Regarding Ferguson himself, I hold with my view from last year that he is the victim of the paradox of influence:

No, I think what Ferguson is looking for (as am I, in my scholarly domain) is influence. He wants to make a difference. And one thing about being paid $50K for a lecture is that you can assume that whoever is paying you really wants to hear what you have to say.

The paradox, though, is that Ferguson gets and keeps the big-money audience is by telling them not what he (Ferguson) wants to say—not by giving them his unique insights and understanding—but rather by telling his audience what they want to hear. . . .

The paradox is that the anticipated influence becomes valueless if you end up saying whatever it takes to keep it.

In this case, I think Ferguson went too far when he threw in some crowd-pleasing anti-Keynes remarks that didn’t please his crowd so much. Ferguson took his bad-boy stance a bit too far.

As usual in such cases, I see a gap between what this guy is doing and what he has the potential to do. The Kenyes-is-a-poof remarks would’ve been unremarkable had they been made by a comedian or a politician or talk-show host. A historian can and should do better, and Ferguson himself recognized this by characterizing his remarks as “stupid.”

P.S. Yes, I know I’m violating the title of this post from a couple years ago. On the plus side, I’m posting this on a weekend, which is pretty much the time of minimum readership. So, with luck, nobody will notice what I’m doing here.

The post One more thought on Hoover historian Niall Ferguson’s thing about Keynes being gay and marrying a ballerina and talking about poetry appeared first on Statistical Modeling, Causal Inference, and Social Science.

Things are going much better now — it’s interesting, it feels like with both of my models, parameters are slow to converge or get “stuck” and have trouble mixing when the model is somehow misspecified.

See here for a statement of the folk theorem.

The post The Folk Theorem of Statistical Computing appeared first on Statistical Modeling, Causal Inference, and Social Science.

History professor (or, as the news reports call him, “Harvard historian”) Niall Ferguson got in trouble when speaking at a conference of financial advisors. Tom Kostigen reports:

Ferguson responded to a question about Keynes’ famous philosophy of self-interest versus the economic philosophy of Edmund Burke, who believed there was a social contract among the living, as well as the dead. Ferguson asked the audience how many children Keynes had. He explained that Keynes had none because he was a homosexual and was married to a ballerina, with whom he likely talked of “poetry” rather than procreated. . . . Ferguson . . . says it’s only logical that Keynes would take this selfish worldview because he was an “effete” member of society. . . .

Throughout his remarks, Ferguson referred to his “friends” in high places. They should all be embarrassed and ashamed of such a connection to such small-minded thinking. Ferguson says U.S. laws and institutions have become degenerate.

According to Henry Blodget, “Dan Jamieson at Investment News, also reported the remarks.”

It will be interesting to see what Ferguson’s next steps are. I’m guessing he’ll eventually get to the full groveling apology, but other options include affirmation (“Yes, Keynes was a poof who didn’t know jack about econ!”), explanation (“Statistics show that gay and childless people invest less in the future”), flat-out denial (“I never said such a thing”), a mumbling quasi-denial (“I don’t recall saying such a thing . . . must have been misquoted”), an appeal to academic freedom (“No, I’m not embarrassed to speculate in a politically incorrect fashion”), or, of course, the non-apology apology (“I’m sorry that people chose to be offended over my remarks”).

In short, he can go in two directions: (1) cut his losses or (2) aim to be a cult hero of political correctness in the Larry Summers mold (although I doubt that Larry Summers himself is going to endorse this particular anti-gay move).

In some ways, this is a higher-profile version of the choice that Ferguson had to make awhile ago, whether to follow his counterparts such as Jeffrey Frankel and John Yoo into embarrassing hackery, or stay on the academic straight and narrow (so to speak). A couple years ago, the choice was to keep doing academic research/writing/learning or to make provocative speeches saying silly things to rich people for big bucks. Now that Ferguson is upping the ante by antagonizing entirely new groups of people, he’ll have to decide where to go next.

The good news, though, is I think this whole story is a sign of improving standards of public discourse, at least when it comes to outspoken professors. Back in the old days, there were respected academics who were Stalinists, Fascists, you name it. Nowadays when you take an extreme position, you’re expected to defend it, otherwise you don’t get taken seriously any more. (For example, academic policy advocates such as Krugman and Mankiw piss off lots of people, but they defend their controversial statements.)

P.S. From his webpage:

Niall Ferguson, MA, D.Phil., is the Laurence A. Tisch Professor of History at Harvard University. He is a resident faculty member of the Minda de Gunzburg Center for European Studies. He is also a Senior Research Fellow of Jesus College, Oxford University, and a Senior Fellow of the Hoover Institution, Stanford University.

But he’s always referred to as “Harvard historian” or “Harvard professor” Niall Ferguson. Doesn’t seem so fair to Oxford and Stanford. I propose that future news articles refer to Ferguson as “Jesus historian” or “Hoover historian.” These have a good ring to them, no?

P.P.S. Ferguson chose the first option, a complete apology:

I should not have suggested – in an off-the-cuff response that was not part of my presentation – that Keynes was indifferent to the long run because he had no children, nor that he had no children because he was gay. This was doubly stupid. First, it is obvious that people who do not have children also care about future generations. Second, I had forgotten that Keynes’s wife Lydia miscarried.

My disagreements with Keynes’s economic philosophy have never had anything to do with his sexual orientation. It is simply false to suggest, as I did, that his approach to economic policy was inspired by any aspect of his personal life. As those who know me and my work are well aware, I detest all prejudice, sexual or otherwise.

This retraction seems like a reasonable move to me. After all, everybody makes mistakes.

As I wrote in comments, I’m guessing that, in the heat of the moment, Ferguson thought that some Keynes-bashing would be popular with a business crowd. What happened is, he misjudged his audience. He threw them the wrong flavor of red meat.

P.P.P.S. More here.

The post Jesus historian Niall Ferguson and the improving standards of public discourse appeared first on Statistical Modeling, Causal Inference, and Social Science.

The hype surrounding Big Data and Data Science is at a fever pitch with promises to solve the world’s business and social problems, large and small. How accurate or misleading is this message? How is it helping or damaging people, and which people? What opportunities exist for data nerds and entrepreneurs that examine the larger issues with a skeptical view?

This Meetup focuses on mathematical, ethical, and business aspects of data from a skeptical perspective. Guest speakers will discuss the misuse of and best practices with data, common mistakes people make with data and ways to avoid them, how to deal with intentional gaming and politics surrounding mathematical modeling, and taking into account the feedback loops and wider consequences of modeling. We will take deep dives into models in the fields of Data Science, statistics, financial engineering, and economics.

This is an independent forum and open to anyone sharing an interest in the larger use of data. Technical aspects will be discussed, but attendees do not need to have a technical background.

The post NYC Data Skeptics Meetup appeared first on Statistical Modeling, Causal Inference, and Social Science.

Forget the interpretation for a moment, and the political spin, but haven’t we reached an interesting point when a journalist says things like:

When you do an RCT with more than 12,000 people in it, and your defense of your hypothesis is that maybe the study just didn’t have enough power, what you’re actually saying is “the beneficial effects are probably pretty small”.

and

A good Bayesian—and aren’t most of us are supposed to be good Bayesians these days?—should be updating in light of this new information. Given this result, what is the likelihood that Obamacare will have a positive impact on the average health of Americans? Every one of us, for or against, should be revising that probability downwards. I’m not saying that you have to revise it to zero; I certainly haven’t. But however high it was yesterday, it should be somewhat lower today.

This is indeed an excellent news article. Also this sensible understanding of statistical significance and effect sizes:

But that doesn’t mean Medicaid has no effect on health. It means that Medicaid had no statistically significant effect on three major health markers during a two-year study. Those are related, but not the same. And in fact, all three markers moved in the right direction. They just weren’t big enough to rule out the possibility that this was just random noise in the underlying data. I’d say this suggests that it’s more likely than not that there is some effect–but also, more likely than not that this effect is small.

The only flaw is this bit:

There was, on the other hand, a substantial decrease in reported depression. But this result is kind of weird, because it’s not coupled with a statistically significant increase in the use of anti-depressants. So it’s not clear exactly what effect Medicaid is having. I’m not throwing this out: depression’s a big problem, and this seems to be a big effect. I’m just not sure what to make of it. Does the mere fact of knowing you have Medicaid make you less depressed?

McArdle is forgetting that the difference between “significant” and “not significant” is not itself statistically significant. I have no idea whether the result is actually puzzling. I just think that she was leaping too quickly from “A is significant and B is not” to “A and B contradict.”

Also I’d prefer she’d talk with some public health experts rather than relying on sources such as, “as Josh Barro pointed out on Twitter.” I have nothing against Josh Barro, I just think it’s good if a journalist can go out and talk with people rather than just grabbing things off the twitter feed.

But these are minor points. Overall the article is excellent.

With regard to the larger questions, I agree with McArdle that ultimately the goals are health and economic security, not health insurance or even health care. She proposes replacing Medicaid with “free mental health clinics, or cash.” The challenge is that we seem to have worked ourselves into an expensive, paperwork-soaked health-care system, and it’s not clear to me that free mental health clinics or even cash would do the trick.

Other perspectives

I did some searching and found this post by Aaron Carroll. I agree with what Carroll wrote, except for the part where he says that he would not say that “p=0.07 is close to significant.” I have no problem with saying p=0.07 is close to significant. I think p-values are often more of a hindrance than a help, but if you’re going to use p=0.05 as a summary of evidence and call it “significant,” then, indded, 0.001 is “very significant,” 0.07 is “close to significant,” and so forth. McArdle was confused on some of these issues too, most notably by mixing statistical significance with a Bayesian attitude. I wouldn’t be so hard on either of these writers, though, as the field of statistics is itself in flux on these points. Every time I write a new article on the topic, my own thinking changes a bit.

I see some specific disagreements between McArdle and Carroll:

1. McArdle writes:

Katherine Baicker, a lead researcher on the Oregon study, noted back in 2011, “people who signed up are pretty sick”.

Carroll writes:

Most people who get health insurance are healthy. They’re not going to get “healthier”.

This seems like a factual (or, at least, a definitional) disagreement.

2. McArdle:

We heard that 150,000 uninsured people had died between 2000 and 2006. Or maybe more. With the implication that if we just passed this new law, we’d save a similar number of lives in the future. Which is one reason why the reaction to this study from Obamacare’s supporters has frankly been a bit disappointing.

Carroll:

This was Medicaid for something like 10,000 people in Oregon. The ACA was supposed to be a Medicaid expansion for 16,000,000 across the country. If 8 people’s lives in the study were saved in some way by the coverage, the total statistic holds.

(16,000,000/10,000)*8*7 = 90,000, so that’s in the ballpark of the claimed 150,000 in seven years. I’m guessing that McArdle’s would reply that there’s no evidence that 8 people’s lives were saved in the Oregon study. Thus, numbers such as 100,000 lives saved are possible, but other things are possible too.

The bottom line

What does this all mean in policy terms? McArdle describes Obamacare as “a $1 trillion program to treat mild depression.” I’m not sure where the trillion dollars comes from. A famous graph shows U.S. health care spending at $7000 per person per year, that’s a total of 2.1 trillion dollars a year. I’m assuming that the Obama plan would not increase this to 3.1 trillion! Maybe it is projected to increase annual spending to 2.3 trillion, which would correspond to an additional trillion over a five-year period? In any case, that sounds pretty expensive. Given that other countries with better outcomes spend half as much as we do, I’d hope a new health-care plan would reduce costs, not increase them. But that’s politics: the people who are currently getting these 2.1 trillion dollars don’t want to give up any of their share! The other half of McArdle’s quote (“mild depression”) sounds to me like a bit of rhetoric. If a policy will reduce mild depression, I assume it would have some eventual effect on severe depression too, no?

Beyond this, I can’t comment. I’m like many (I suspect, most) Americans who already have health insurance in that I don’t actually know what’s in that famous health-care bill. I mean, sure, I know there’s something about every American getting coverage, but I don’t know anything beyond that. So I’m in no position to say anything more on the topic. I’ll just link to Tyler Cowen, who, I assume, actually knows what’s in the law and has further comments on the issue.

Let me conclude where I began, with an appreciation of the high quality of statistical journalism today. In her news article, McArdle shows the sort of nuanced understanding of statistics and evidence that I don’t think was out there, twenty years ago. And she’s not the only one. Journalists as varied as Felix Salmon, Nate Silver, and Sharon Begley are all doing the good work, writing about newsworthy topics in a way that acknowledges uncertainty.

The post Setting aside the politics, the debate over the new health-care study reveals that we’re moving to a new high standard of statistical journalism appeared first on Statistical Modeling, Causal Inference, and Social Science.

I had no idea this sort of thing even existed:

I’m reminded of our discussion of Charles Murray’s recent book on social divisions among Americans. Murray talked about differences between upper and lower class, but I thought he was really talking more about differences between liberals and conservatives among the elite. (More discussion here.)

In this particular case, Murray’s story about irresponsible elites seems to fit pretty well. At the elite level, you have well-connected D.C. gun lobbyists opposing any restrictions on personal weapons. As Murray might put it, the elites (Phil Spector aside) may be able to handle their guns, but some lower-class Americans cannot—they do things like give real rifles to 5-year-olds (!). As Murray writes, it’s a combination of cultural ignorance and a permissive ideology: I assume the senators who voted against the recent gun control bill wouldn’t give live weapons to their kids (or live in neighborhoods in which kids have access to guns at home), but they don’t feel right about restricting the rights of others to do so.

P.S. After reading some comments, I thought it might help to clarify two points.

First, my main point here was, as is noted in the title of the post, this is a culture clash. In some parts of the country, people are giving real live guns to their 5-year-olds and teaching them the four rules of gun safety. Here where I live, I had no idea that there was such a thing as a company that sold children’s guns with cute little cartoon characters.

Second, I don’t mean to overrate Charles Murray’s ideas, which are more of a set of interesting speculations than an all-encompassing theory. (This is fine, speculations are important, I’m not knocking Murray here.) For example, consider elites’ reactions to proposed laws restricting cigarette smoking. If a group of elite Americans support such a law, Murray can argue that this represents their lack of understanding of ordinary American culture: elites mostly don’t smoke, and they can’t understand people who do. Conversely, if a group of elites oppose an anti-smoking law, Murray can argue that these elites are supporting a permissive, dangerous ideology that they do not themselves believe for their own lives. This is the tension that I noted in my review of Murray’s book, where elites are alternately urged to show understanding for the unfamiliar lifestyles of lower-class Americans, and to show moral leadership by criticizing the mistaken lifestyle choices of such people. Murray’s reasoning goes in these two opposite directions, so when I say that Murray’s story fits pretty well in this example, I can only really be referring to part of Murray’s argument, not all of it.

The post Culture clash appeared first on Statistical Modeling, Causal Inference, and Social Science.

Betsey Stevenson and Justin Wolfers have been inspired by the recent Reinhardt and Rogoff debacle to list “six ways to separate lies from statistics” in economics research:

1. “Focus on how robust a finding is, meaning that different ways of looking at the evidence point to the same conclusion.”

2. Don’t confuse statistical with practical significance.

3. “Be wary of scholars using high-powered statistical techniques as a bludgeon to silence critics who are not specialists.”

4. “Don’t fall into the trap of thinking about an empirical finding as ‘right’ or ‘wrong.’ At best, data provide an imperfect guide.”

5. “Don’t mistake correlation for causation.”

6. “Always ask ‘so what?’”

I like all these points, especially #4, which I think doesn’t get said enough. As I wrote a few months ago, high-profile social science research aims for proof, not for understanding—and that’s a problem.

My addition to the list

If you compare my title above to that of Stevenson and Wolfers, you’ll find two differences. First, I changed “lies” to “errors.” I have no idea who’s lying, and I’m much more comfortable talking about errors. Second, I think they missed an even better, more general way to find mistakes:

7. Make your data and analysis public.

This is the best approach, because now you can have lots of strangers checking your work for free! This advice is also particularly appropriate for Reinhardt and Rogoff because, according to various reports (see here and here), it was years before they made their data available to outsiders. Nearly three years ago (!), Dean Baker wrote a column entitled, “It Would Be Helpful if Rogoff and Reinhart Made Their Data Available.”

Perhaps “the risk of forced disclosure” (as Keith O’Rourke puts it) will motivate researchers to be more careful in the future.

Your additions?

I told Wolfers I was going to link to his list and add my own #7. He replied that we’re probably missing #8, 9, and 10. In the comments, feel free to add your favorite ways to separate errors from statistics. Phil already gave some here.

The post 7 ways to separate errors from statistics appeared first on Statistical Modeling, Causal Inference, and Social Science.

As Kaiser writes, “from a purely graphical perspective, the chart is well executed . . . they have 54 points, and the chart still doesn’t look too crammed . . .” But he also points out that the graph’s implicit claims (that tax rates can explain happiness or cause more happiness) are not supported.

Kaiser and I are not being picky-picky-picky here. Taken literally, the graph title says nothing about causation, but I think the phrasing implies it. Also, from a purely descriptive perspective, the graph is somewhat at war with its caption. The caption announces a relationship, but in the graph, the x and y variables have only a very weak correlation. The caption says that happiness and progressive tax rates go together, but the graph uses the U.S. as a baseline, and when you move from the U.S. point on the graph to the right-hand side (more progressive taxes), you see a lot more points below the line than above the line. Thus the visual impression of the graph is that more progressive taxes will lead to lower happiness—the opposite of the message from the caption.

What can be done here?

I don’t exactly think the graph is “bad data,” and, although the graph says little directly about causation, the data have some relevance to our understanding of policy debates over taxes. If nothing else, we learn that tax progressivity and average happiness some variation among countries. I think a start would be to reframe and put happiness on the x-axis and the tax system on the y-axis, which would allow us to see that, at any happiness level, there is a range of tax systems. with none of the very happiest countries having flat taxes.

Better still might be to make a line plot with three columns: First, a list of country names, in decreasing order from richest to poorest (using, for example, per-capita GDP (yes, I know, such data aren’t perfect!)), then a column showing tax progressivity (if that’s the measure they want to use), then a column showing average happiness.

The advantage of this pair of dotplots is that you get to see the spread in each of these variables with respect to a natural measure (how rich the country is), and there’s no implicit causal story getting in the way.

The post A graph at war with its caption. Also, how to visualize the same numbers without giving the display a misleading causal feel? appeared first on Statistical Modeling, Causal Inference, and Social Science.

I once saw the following on one of those inspirational-sayings-for-every-day desk calendars: “To have ideas is to gather flowers. To think is to weave them into garlands.” Similarly, writing—more generally, communication to oneself or others—forces logic and structure, which are central to science.

Dan Kahan saw what I wrote and responded by flipping it around: He pointed out that there is a science of science communication. As scientists, we should move beyond the naive view of communication as the direct imparting of facts and ideas. We should think more systematically about how communications are produced and how they are understood by their immediate and secondary recipients.

The science of science communication is still in its early stages, and I’m glad that people such as Kahan are working on it. Here’s something he wrote recently explicating his theory of cultural cognition:

The motivation behind this research has been to understand the science communication problem. The “science communication problem” (as I use this phrase) refers to the failure of valid, compelling, widely available science to quiet public controversy over risk and other policy relevant facts to which it directly speaks. The climate change debate is a conspicuous example, but there are many others, including (historically) the conflict over nuclear power safety, the continuing debate over the risks of HPV vaccine, and the never-ending dispute over the efficacy of gun control. . . . The research I will describe reflects the premise that making sense of these peculiar packages of types of people and sets of factual beliefs is the key to understanding—and solving—the science communication problem. The cultural cognition thesis posits that people’s group commitments are integral to the mental processes through which they apprehend risk. . . .

I think of Kahan as part of a loose network of constructive skeptics, along with various people including Thomas Basbøll, John Ioannidis, the guys at Retraction Watch, pissed-off scholars such as Stan Liebowitz, bloggers such as Felix Salmon, and a whole bunch of psychology researchers such as Wicherts, Wagenmakers, Simonsohn, Nosek, etc. This is not to represent a complete list but rather is intended to give a sense of the different aspect of this movement-without-a-name. 10 or 20 or 30 years ago, I don’t think such a movement existed. There were concerns about individual studies or research programs but not such a sense of a statistics-centered crisis in science as a whole.

The post “Tragedy of the science-communication commons” appeared first on Statistical Modeling, Causal Inference, and Social Science.

Earlier in the column, Easterbrook has a plug for Tim Tebow. I’d forgotten about Tim Tebow.

The post Giving credit where due appeared first on Statistical Modeling, Causal Inference, and Social Science.

Last summer my wife and I took a 3.5-month vacation that included a wide range of activities. When I got back, people would ask “what were the highlights or your trip?”, and I was somewhat at a loss: we had done so many things that were so different, many of which seemed really great…how could I pick? Someone said, wisely, that in six months or a year I’d be able to answer the question because some memories would be more vivid than others. They were right, and I was recently thinking back on our vacation and putting together a list of highlights — enjoyable in itself, but also worth doing to help plan future vacations.

One of the things we did was go to four evenings of track and field events at the London Olympics. After we got back, people would ask what we had seen at the Olympics. I would say “We saw Usain Bolt run the 200m, we saw the women’s 4x100m relay and the men’s 4×400, we saw the last events of the decathlon…lots of great stuff. But my favorite was the men’s 800m.”

Trying to figure out why that was one of my favorite events to watch, I looked up some facts and statistics about the race. Perhaps unexpectedly, I think that some of the things that made it great, as both an athletic contest and a spectacle, are reflected in the stats.

The 800m is not one of track’s marquee events: it’s much too long for an all-out sprint, so it doesn’t have the intensity of a 100m or 200m. It has even less of a following than most track events, which is saying something: I’m going to guess that most readers of this blog cannot name a single U.S. track athlete, and that only a tiny fraction can name a competitor in that 800m Olympics final.

Me, I’m more of a track fan than most — my wife and I have gotten interested, especially since discovering that during our fitness workouts we were sometimes sharing the track with Alysia Montano and  Magdalena Lewy-Boulet, and we spectate at track meets occasionally — but although I recognized a few of the names in the 800m Olympics final I didn’t really have any context for what I was about to see. Like most people in the stadium, I expect, I was looking forward to the highly-anticipated 200m final later in the evening and hadn’t even thought about the 800. That 200m final was exciting and electrifying, but to me it was no match for the 800.

When the race started…it’s hard to say how this could possibly be, but I could just tell that something special was happening. Maybe everybody in the stadium could tell.  One of the runners — David Rudisha of Kenya, the record-holder at the time — went to the front and was pushing hard, and everyone else was going hard too.  Of course.  It’s the Olympics! Finals! Of course they’re running hard!  Well, yes, but… it’s hard to explain…it just seemed more intense even than the other finals we had seen, which seems paradoxical because, this being the 800m, the runners weren’t running as fast.  I’ve tried to tell if you can see in the video what I’m talking about (the race starts at about 3:15), but I’m not really sure, you tell me. (One thing that you can see on the video is how easily these guys seem to cruise through the 400m mark in under 50 seconds, incredible).

So, here’s what happened:

Rudisha, the pre-race favorite, won the event, beating his own previous world record.

The times flashed up on the scoreboard, with indicators to say if the time was a World Record, National Record, Junior Record, Personal Record, or Season’s Best.  That column on the scoreboard looked like this:WR, NR, PB, PB, PB, NR, SB, PB  In other words, seven out of the eight runners had run their personal best time.

That’s all I knew at the time, and it’s only recently that I learned some other facts: All eight athletes went under 1:44, the first time that has ever happened; and the first place time was (obviously) the fastest first place in history, but also, the second place was the fastest second place in history, and third was the fastest third in history, and so on all the way down the line.

It obviously takes a special effort to set a personal record, and those records usually come when the athlete has something at stake: the chance to finish on the podium at their national championships, or to qualify for the Olympic team, or whatever.  For a guy who is rounding the final curve 20 meters behind the leaders, a guy who is going to finish in 8th place, to push himself to the fastest that he has ever run, that seems to me to be even more remarkable than someone running a personal best in order to set a new world record. I tip my hat.

So  I think it wasn’t the effort by the top finishers that made the race seem so special at the time — we had seen other records set[correction, this was the first WR we saw], we would see records set the following day, and they were great and exciting but they didn’t have the same magic for me. I think what made this special was the effort all up and down the line, every runner trying his hardest right to the end. Maybe the best 800m races to watch are the ones where the last-place finishers run their hardest, not where the first-place finishers run theirs. Just a thought.

This post is by Phil.

The post The Great Race appeared first on Statistical Modeling, Causal Inference, and Social Science.

I encourage you to check out our linked blogs. Here’s what they’re all about:

Cognitive and Behavioral Science

BPS Research Digest: I haven’t been following this one recently, but it has lots of good links, I should probably check it more often. There are a couple things that bother me, though. The blog is sponsored by the British Psychological Society, so this sounds pretty serious. But then they run things like advertising promotions sponsored by a textbook company and highlight iffy experimental claims. For example, in 2010 they ran a wholly uncritical post on the notorious Daryl Bem study that purported to find ESP. After being called on it in the comments, the blogger (Christian Jarrett) responded with, “The stats appear sound. . . . it’s a great study. Rigorously conducted” and even defended “the discussion of quantum physics in the paper.” To be fair, though, and as he points out in comments, Jarrett wrote of Bem’s study: “this isn’t proof of psi, far from it. Needs to be replicated. I like how Bem has used standard psychological tasks as a way to explore psi. Makes it easier for other labs to try to replicate.” Jarrett writes that he tries to “strike a balance between promotion and skepticism of new findings.” Fair enough.

Decision Science News: A mix of conference announcements and reports of new research. Here’s a typical example. I love this stuff; others might find it a bit technical. Also, this blog runs ads. I wonder how much the advertisers pay? I can’t imagine anyone would pay enough to a niche blogger to make the ads worth it. I mean, sure, if an advertiser offered me enough money for me to hire a postdoc, I’d do it, but I can only imagine we’re talking really small amounts of money. A topic of discussion for Decision Science News, perhaps?

Language Log: Not much needs to be said here. This one’s a classic blog with lots of statistical content, remains strong after all these years.

Seth Roberts: I disagree with him on climate change denial, Holocaust denial, etc. Still, he’s a pioneer of self-experimentation. I hope that the next generation of psychology or medical research involves an integration of informal experimentation with statistical controls.

The Hardest Science: Mostly revolves around reproducible research. It’s where I heard the story of the lamest, grudgingest, non-retraction retraction ever.

Cultural

Light Reading: She’s like me, she likes to write and has a lot of energy. I’m still wondering what she will think of Debutante Hill (I’ll lend her my copy).

Lists and Letters of Note: Great stuff but not much new material lately; he says he’s busy working on a book.

Love the Liberry: Amazingly enough, they keep coming up with good material.

Paperpools: Not much material lately. As it should be. We want Helen DeWitt to be writing novels, not blogging!

Research as a Second Language: Anti-charismatic self-help advice. The alternative to those omnipresent shouting, obnoxious internet gurus.

Streetsblog: Good stuff. Ideally this would all be in your daily newspaper. I don’t read it too often; if I did, I’d be too angry to think about anything else all day.

Sister Blogs

The Monkey Cage: Sometimes I simul-post, other times I’ll rant there and then link from here. (for example)

The Statistics Forum: I recently formulated the plan to fill it up with 365 stories. So far, though, I’ve only received a few. So maybe just a story a week? I’m not sure what to do with this blog. An official American Statistical Association blog seems like a good thing but I don’t really know what to do with it.

Social and Political Science

Chris Blattman: International development, politics, economics, and policy.

Fivethirtyeight: Nate does a good job. I like how he can focus on whatever question he’s answering without getting overwhelmed. Here’s a good recent example.

Lane Kenworthy is a completely serious and reasonable person, just as his name would suggest.

Marginal Revolution: You’ve heard of these guys.

Monthly Labor Review Precis: Direct links to research on things that matter. Good stuff.

Overcoming Bias: He recently wrote, “most people we know talk as if they hate, revile, and despise ads. They say ads are an evil destructive manipulative force that exists only because big bad firms run the world, and use ads to control us all.” I was surprised to hear that most people Robin Hanson knows talk that way, and this gives me a new perspective on why he writes the way he does. It’s gotta be frustrating, hanging around people who talk about big bad firms and evil destructive manipulative forces.

Rajiv Sethi: He only blogs a couple times a month, but he always has something interesting to say. (The opposite of this blog, I suppose.)

The Baby Name Wizard: The one and only, by the people who, among other things, debunked the myth that there’s something special about the word “orange.” But you can just skip directly to the Name Voyager.

U.S. Census Blog: Not the funnest thing out there to read, but it’s good that the people at the Census are doing this for us. When you need good data, the Census is there for you.

Statistics and Machine Learning

Bob Carpenter: He wrote Stan.

Chance News: The original statistics blog.

Christian Robert: People who used to do theoretical statistics, now do computational statistics. This is a good thing.

Cosma Shalizi: He has an odd retro style and enough combination of common sense and knowledge of philosophy that I asked him to collaborate on my paper that became this. His set of interests and frustrations seems to overlap a lot with mine, except that he doesn’t really ride a bike and I’m sure there are some big parts of his life that don’t match to anything in mine.

Deborah Mayo: I learned about her through Shalizi. Mayo believes in learning through model checking, just like Jaynes (and me). Her blog features long comment threads and contributions from the likes of Stephen Senn.

John Cook: Like Tyler Cowen, a guy who does a lot of things but is best known for his blogging. He throws in some applied math and numerical analysis along with the statistics.

Kaiser Fung: Fun to read and utterly sensible. Among many other things, he offered a good probabilistic summary of the Lance Armstrong story, well before it finally broke.

Larry Wasserman: His perspective on statistics is different from mine (for example, he defines p(a|b) = p(a,b)/p(b), whereas I define p(a,b)=p(a|b)p(b)), but it’s good that he can get his views out there. Research proceeds in many different ways, and if everyone agreed with me (or with any single perspective), the field of statistics would make a lot less progress.

Messy Matters: This one reads a bit more like a draft of a pop-science book than like a blog. The trouble is, there are already so many pop-science books about economics and data. They’ll have to come up with their own unique twist.

Nuit Blanche: Compressive sensing: that’s cool stuff! I’m impressed by these CS guys who can effortlessly throw around terabytes of data.

Observational Epidemiology: These guys are thoughtful and I admire the effort they put into their blogging. If they’d started blogging in 2003, they would’ve been on everyone’s blogroll.

Stats Blogs: A convenient compendium, with links back to the originals.

The Numbers Guy: Carl Bialik is one of the original data journalists. He, Falix Salmon, and Nate Silver have very similar profiles (as Bill James might say).

Visualization

Chartsnthings: This is the ultimate graphics blog. The New York Times graphics team presents some great data visualizations along with the stories behind them. I love this sort of insider’s perspective.

Eager Eyes: Graphics research.

Information Aesthetics: Seriously pretty.

Junk Charts: The nitty gritty. What to read if you want to make your own graphs better.

The post The blogroll appeared first on Statistical Modeling, Causal Inference, and Social Science.

There is a report by Martin Tingley and Peter Huybers in Nature on the unprecedented high temperatures at northern latitudes (Russia, Greenland, etc). What is more interesting is the authors are have used a straightforward hierarchical Bayes model, and for the first time (as far as I can remember) the results are reported with a probability attached to them (P>0.99), as opposed to the usual p-value<0.01 business. This might be a sign that editors of big time science journals are welcoming Bayesian approaches.

I agree. This is a good sign for statistical communication. Here are the key sentences from the abstract:

Here, using a hierarchical Bayesian analysis of instrumental, tree-ring, ice-core and lake-sediment records, we show that the magnitude and frequency of recent warm temperature extremes at high northern latitudes are unprecedented in the past 600 years. The summers of 2005, 2007, 2010 and 2011 were warmer than those of all prior years back to 1400 (probability P > 0.95), in terms of the spatial average. The summer of 2010 was the warmest in the previous 600 years in western Russia (P > 0.99) and probably the warmest in western Greenland and the Canadian Arctic as well (P > 0.90). These and other recent extremes greatly exceed those expected from a stationary climate, but can be understood as resulting from constant space–time variability about an increased mean temperature.

As with classical p-values, these probability statements depend on an assumed model, but I agree with Sam that the expression of direct probabilities is a huge step forward from traditional practice.

The post Plain old everyday Bayesianism! appeared first on Statistical Modeling, Causal Inference, and Social Science.

We are pleased to announce that Time-Sharing Experiments for the Social Sciences (TESS) was renewed for another round of funding by NSF starting last Fall. TESS allows researchers to submit proposals for experiments to be conducted on a nationally-representative, probability-based Internet platform, and successful proposals are fielded at no cost to investigators. More information about how TESS works and how to submit proposals is available at http://www.tessexperiments.org.

Additionally, we are pleased to announce the development of two new proposal mechanisms. TESS’s Short Studies Program (SSP) is accepting proposals for fielding very brief population-based survey experiments on a general population of at least 2000 adults. SSP recruits participants from within the U.S. using the same Internet-based platform as other TESS studies. More information about SSP and proposal requirements is available at http://www.tessexperiments.org/ssp.html.

TESS’s Special Competition for Young Investigators is accepting proposals from June 15th-September 15th. The competition is meant to enable younger scholars to field large-scale studies and is limited to graduate students and individuals who are no more than 3 years post-Ph.D. More information about the Special Competition and proposal requirements is available at http://www.tessexperiments.org/yic.html.

We are being assisted by a new team of over 65 Associate PIs and peer reviewers across the social sciences. More information about our APIs is available at http://www.tessexperiments.org/associatepi.html.

This seems cool to me, both for the obvious reason that it will reduce the intellectual overhead required to run a good social-science experiment, and indirectly because I assume the resulting experiments will be open with shared data.

The post Time-Sharing Experiments for the Social Sciences appeared first on Statistical Modeling, Causal Inference, and Social Science.

If you like continued fractions, I recommend you read the relevant parts of the classic Numerical Methods That Work. The details are probably obsolete but it’s fun reading (at least, if you think that sort of thing is fun to read).

I then looked up Acton in Wikipedia and was surprised to see he’s still alive. And he wrote a second book (published at the age of 77!). I wonder if it’s any good. It’s sobering to read Numerical Methods That Work: it’s so wonderful and so readable, yet in this modern era there’s really not much reason to read it. Perhaps William Goldman (hey, I checked: he’s still alive too!) or some equivalent could prepare a 50-page “good parts” version that could be still be useful as a basic textbook.

The post Continued fractions!! appeared first on Statistical Modeling, Causal Inference, and Social Science.

Publication bias remains a controversial issue in psychological science. . . . that the field often constructs arguments to block the publication and interpretation of null results and that null results may be further extinguished through questionable researcher practices. Given that science is dependent on the process of falsification, we argue that these problems reduce psychological science’s capability to have a proper mechanism for theory falsification, thus resulting in the promulgation of numerous “undead” theories that are ideologically popular but have little basis in fact.

They mention the infamous Daryl Bem article. It is pretty much only because Bem’s claims are (presumably) false that they got published in a major research journal. Had the claims been true—that is, had Bem run identical experiments, analyzed his data more carefully and objectively, and reported that the results were consistent with the null hypothesis—then the result would be entirely unpublishable. After all, you can’t publish an article in a top journal demonstrating that a study is consistent with there being no ESP. Everybody knows that ESP, to the extent it exists, has such small effects as to be essentially undetectable in any direct study. So here you have the extreme case of a field in which errors are the only thing that gets published.

It’s science as Slate magazine is reputed to be: if it’s true, it’s obvious so no need to publish. If it’s counterintuitive, go for it. (Just to be clear, I’m not saying the actual Slate magazine is like that; this is just its reputation.)

This is indeed disturbing and I applaud yet another publication on the topic. The authors go beyond previous research by Gregory Francis and Uri Simonsohn by focusing specifically on difficulties with meta-analyses that unsuccessfully try to overcome problems of publication bias.

There’s something called the fail-safe number (FSN) of Rosenthal (1979) and Rosenthal and Rubin (1978), “an early and still widely used attempt to estimate the number of unpublished studies, averaging null results, that are required to bring the meta-analytic mean Z value of effect sizes down to an insignificant level,” but,

The FSN treats the file drawer of unpublished studies as unbiased by assuming that their average Z value is zero. This wrong assumption appears mostly not to be recognized by researchers who use the FSN to demonstrate the stability of their results. . . . Without making this computational error, the FSN turns out to be a gross overestimate of the number of unpublished studies required to bring the mean Z value of published studies to an insignificant level. The FSN thus gives the meta-analytic researcher a false sense of security.

The false sense of security persists:

Although this fundamental flaw had been spotted early, the number of applications of the FSN has grown exponentially since its publication. Ironically, getting critiques of the FSN published was far from an easy task . . .

Problems with meta-analysis

Ferguson and Heene continue:

Meta-analyses should be more objective arbiters of review for a field than are narrative reviews, but we argue that this is not the case in practice. . . . The selection and interpretation of effect sizes from individual studies requires decisions that may be susceptible to researcher biases.

It is thus not surprising that we have seldom seen a meta-analysis resolve a controversial debate in a field. Typically, the antagonists simply decry the meta-analysis as fundamentally flawed or produce a competing meta-analysis of their own . . . meta-analyses may be used in such debates to essentially confound the process of replication and falsification.

Thus:

The average effect size may be largely meaningless and spurious due to the avoidance of null findings in the published literature. This aversion to the null is arguably one of the most pernicious and unscientific aspects of modern social science.

Let me interject here that, although I am in general agreement with Ferguson and Heene on these issues, I have a bit of “aversion to the null” myself. I think it’s important to separate the statistical from the scientific null hypothesis.

- The statistical null hypothesis is typically that a particular comparison is exactly zero in the population.

- The scientific null hypothesis is typically that a certain effect is nonexistent or, more generally, that the effect depends so much on situation as to be unreplicable in general.

I might well believe in the scientific null but not in the statistical null.

Virtually unkillable

Ferguson and Heene continue:

The aversion to the null and the persistence of publication bias and denial of the same, renders a situation in which psychological theories are virtually unkillable. Instead of rigid adherence to an objective process of replication and falsification, debates within psychology too easily degenerate into ideological snowball fights, the end result of which is to allow poor quality theories to survive indefinitely. Proponents of a theory may, in effect, reverse the burden of proof, insisting that their theory is true unless skeptics can prove it false (a fruitless invitation, as any falsifying data would certainly be rejected as flawed were it even able to pass through the null-aversive peer review process described above).

Indeed. We see this reversal of the burden of proof all the time. For example, after a data alignment error was uncovered in their research, Neil Anderson and Deniz Ones notoriously wrote: “When any call is made for the retraction of two peer-reviewed and published articles, the onus of proof is on the claimant and the duty of scientific care and caution is manifestly high. . . . Goldberg et al. do not and cannot provide irrefutable proof of the alleged clerical errors.. . . . We continue to stand by the findings and conclusions reported in our previous publications” Ugh! This bothered me so much when I saw it, it made me want to barf. At the time, I wrote that it’s unscientific behavior not to admit error. Unfortunately, for reasons discussed by Ferguson and Heene, much of the scientific enterprise seems to be set up to avoid admission of error. These are serious issues, and it’s interesting to me that we as a field haven’t been thinking much about them until recently.

The post “A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science’s Aversion to the Null” appeared first on Statistical Modeling, Causal Inference, and Social Science.

And some words:

People talk less about video games as they get older, and more about politics and the weather. Men typically talk more about sports and technology than women—and, somewhat surprisingly to me, they also talk more about movies, television and music. Women talk more about pets+animals, family+friends, relationships—and, at least after they reach child-bearing years, health. . . . Some of this is rather depressingly stereotypical. And most of it isn’t terribly surprising to anyone who’s known a reasonable diversity of people of different ages. But what to me is remarkable is how we can see everything laid out in such quantitative detail in the pictures above—kind of a signature of people’s thinking as they go through life.

Of course, the pictures above are all based on aggregate data, carefully anonymized. But if we start looking at individuals, we’ll see all sorts of other interesting things. . . .

Good stuff, and I like the flexible, open attitude. And great graphs. That’s why I’m posting this, in order to spread the word, to inspire others to do this sort of statistical exploration. Follow the link for lots more.

By the way . . .

I wonder who did the analysis, who made the graphs, and who wrote the text. No authors are listed. It’s posted on the Stephen Wolfram Blog, but Wolfram is known for contracting out his research. It’s certainly possible that he did all the statistical analysis, computing, graphics, and writing himself, I just have no idea. It’s funny: in academia, allocation of credit and attribution of authorship is huge. In industry, not so much. As an academic, I’d like to give credit to whoever made these pretty graphs, but perhaps from Wolfram’s perspective, whoever made the graphs is just doing a job, just like whoever sweeps the floors in the lab or whoever cleans the erasers in the classroom. In any case, I give Wolfram credit, no joke. Even if he didn’t do any of the work on this, it takes skill to hire the right people to do the job.

The post Fascinating graphs from facebook data appeared first on Statistical Modeling, Causal Inference, and Social Science.

Zhiqiang Tan writes:

I have created an R package to implement the full likelihood method in Kong et al. (2003). The method can be seen as a binless extension of so-called Weighted Histogram Analysis Method (UWHAM) widely used in physics and chemistry. The method has also been introduced to the physics literature and called the Multivariate Bennet Acceptance Ratio (MBAR) method. But a key point of my implementation is to compute the free energy estimates by minimizing a convex function, instead of solving nonlinear equations by the self-consistency or the Newton-Raphson algorithm.

The post It’s binless! A program for computing normalizing functions appeared first on Statistical Modeling, Causal Inference, and Social Science.

Brett Keller points us to this feel-good story of the day:

A Samurai sword-wielding Mormon bishop helped a neighbor woman escape a Tuesday morning attack by a man who had been stalking her.

Kent Hendrix woke up Tuesday to his teenage son pounding on his bedroom door and telling him somebody was being mugged in front of their house. The 47-year-old father of six rushed out the door and grabbed the weapon closest to him — a 29-inch high carbon steel Samurai sword. . . .

Hendrix, a pharmaceutical statistician, was one of several neighbors who came to the woman’s aid after she began yelling for help . . .

Too bad the whole “statistician” thing got buried in the middle of the article. Fair enough, though: I don’t know what it takes to become a Mormon bishop, but I assume it’s more effort than what it takes to learn statistics.

The post Samurai sword-wielding Mormon bishop pharmaceutical statistician stops mugger appeared first on Statistical Modeling, Causal Inference, and Social Science.

I’m also giving a shorter talk on the same topic in the Sustainable Development Seminar Series 4pm Monday 29 Apr in room 407 International Affairs Bldg (at 118th St. and Amsterdam Ave.).

It’s just a coincidence that I’m giving the same talk twice. I was asked at different times to speak for these groups. When someone asks me to speak, I let them pick from recent talks on the list.

The post My talk midtown this Friday noon (and at Columbia Monday afternoon) appeared first on Statistical Modeling, Causal Inference, and Social Science.

Is social media a valid indicator of political behavior? We answer this ques- tion using a random sample of 537,231,508 tweets from August 1 to November 1, 2010 and data from 406 competitive U.S. congressional elections provided by the Federal Election Commission. Our results show that the percentage of Republican-candidate name mentions correlates with the Republican vote margin in the subsequent election. This finding persists even when controlling for incumbency, district partisanship, media coverage of the race, time, and demographic variables such as the district’s racial and gender composi- tion. With over 500 million active users in 2012, Twitter now represents a new frontier for the study of human behavior. This research provides a framework for incorporating this emerging medium into the computational social science toolkit.

One charming thing about this paper—and I know this is going to sound patronizing but I don’t mean it to be—is that the authors (or, at least, whatever subset of the authors who did the statistical work) are amateurs. They analyze the outcome in terms of total votes rather than vote proportion, even while coding the predictor as a proportion. They present regression coefficients to 7 significant figures. They report that they have data from two different election cycles but present only one in the paper (but they do have the other in their blog post).

But that’s all ok. They pulled out some interesting data. And, as I often say, the most important aspect of a statistical analysis is not what you do with the data, it’s what data you use.

Tweets and votes

As to the result itself, I’m not quite sure what to do with it. Here’s the key graph:

More tweets, more votes, indeed.
Of course most congressional elections are predictable. But the elections that are between 40-60 and 60-40, maybe not so much. So let’s look at the data there . . . Not such a strong pattern (and for the 2012 data in the 40-60% range it looks even worse; any correlation is swamped by the noise). That’s fine, and it’s not unexpected, it’s not a criticism of the paper but it indicates that the real gain in this analysis is not for predicting votes.

I’m not so convinced that tweets will be so useful in predicting votes—most congressional elections are predictable, but perhaps the prediction tool could be more relevant in low-information or multicandidate elections where prediction is not so easy.

Instead, it might make sense to flip it around and predict twitter mentions given candidate popularity. That is, rotate the graph 90 degrees, and see how much variation there is in tweet shares for elections of different degrees of closeness. Also, while you’re at it, re-express vote share as vote proportion. And scale the size of each dot to the total number of tweets for the two candidates in the election.

Move away from trying to predict votes and move toward trying to understand tweets. DiGrazia et al. write, “the models show that social media matters . . .” No, not at all. They find a correlation between candidate popularity and social media mentions. No-name and fringe candidates get fewer mentions (on average) than competitive and dominant candidates. That’s fine, you can go with that.

Again, I fear the above sounds patronizing, but I don’t mean it to be. You gotta start somewhere, and you’re nowhwere without the data. As someone who was (originally) an outsider to the field of political science, I do think that researchers coming from other fields can offer valuable perspectives.

Sharing the data

What I want to know is, is this dataset publicly available? What would really make this go viral is if DiGrazia et al. post the data online. Then everyone will take a hack at it, and each of those people will cite them.

There’s been a lot of talk about reproducible research lately. In this case, they have a perfect incentive to make they data public: it will help them out, it will help out the research project, and it will be a great inspiration for others to follow in their footsteps. Releasing data as a publicity intensifier: that’s my new idea.

P.S. In the first version of this post I included a graph showing votes given tweet shares between 40% and 60%. I intended this to illustrate the difficulty of predicting close elections, but my graph really missed the point, because the x-axis represented close elections in tweet shares, not in votes. So I crossed that part out. If nothing else, I’ve demonstrated the difficulty of thinking about this sort of analysis!

The post The Tweets-Votes Curve appeared first on Statistical Modeling, Causal Inference, and Social Science.

Now here’s a foundation I (Bob) can get behind:

Foundation for Open Access Statistics (FOAS)

Their mission is to “promote free software, open access publishing, and reproducible research in statistics.” To me, that’s like supporting motherhood and apple pie!

FOAS spun out of and is partially designed to support the Journal of Statistical Software (aka JSS, aka JStatSoft). I adore JSS because it (a) is open access, (b) publishes systems papers on statistical software, (c) has fast reviewing turnaround times, and (d) is free for authors and readers. One of the next items on my to-do list is to write up the Stan modeling language and submit it to JSS.

As a not-for-profit with no visible source of income, they are quite sensibly asking for donations (don’t complain — it beats $3K author fees or not being able to read papers).

The post Foundation for Open Access Statistics appeared first on Statistical Modeling, Causal Inference, and Social Science.

Following up on our blog discussions a year ago, I published a review of Charles Murray’s recent book, “Coming Apart,” for the journal Statistics, Politics, and Policy. I invited Murray to publish a response, and he did so.

Here’s the abstract to my review:

This article examines some claims made in a recent popular book of political sociology, with the intent not being to debunk any claims but rather to connect some important social and policy positions to statistical data on income, social class, and political attitudes. The thesis of Charles Murray’s book is that America’s upper and lower classes have become increasingly separate, with elites living more disciplined, orderly lives (characterized by marriage, work, and stable families) while being largely unaware of the lifestyles of the majority of Americans. I argue that some of Murray’s conclusions are sensitive to particular choices of whom to label as elite or upper-class. From my analysis of survey data, I see the big culture war occurring within the upper class, whereas Murray focuses on differences in attitudes and lifestyles comparing rich to poor. Coming Apart is a lively contribution to current debates and complements more statistical analyses of political and social polarization.

Murray’s reply had no abstract, but here’s the text of the whole thing. I think the discussion was productive.

Before leaving the conversation, I just want to point to a bit in Murray’s article that I thought inadvertently illustrated Murray’s points about implicit assumptions and cultural divides. Near the end of his reply to my review, Murray writes:

When a society’s elite is confident that its own values are the ones that all of society ought to adopt, those values get communicated. They’re in the air—in the way journalists cover stories, editors write editorials, television networks choose the new season’s series, and screenwriters create plots. They are reflected in the way that members of the elite talk with their children, with their professional colleagues, and whenever those topics relating to their values come up in a public setting. In all of those settings, today’s new upper class tends to be obsessively nonjudgmental.

If you doubt it, try bringing up the issue of single women having babies at your next dinner party, and see how many of your companions are willing to say, even in a private gathering of friends, that it is morally wrong for a woman to bring a baby into the world knowing that it will not have a father, and morally wrong for a man to impregnate a woman knowing he will not be a father to the child. Fifty years ago, no one at the same kind of dinner party would have said that it was not morally wrong.

It is statistically highly likely that all of the biological children of the people at a dinner table of today’s upper-class adults have been born within wedlock. If there are childless never-married women at the table, it is likely that they have deliberately foregone having a baby, even though they might want one, because they have decided it is unfair to the child not to have a father. Put another way, it is likely that all of the people at the table have made moral evaluations and behaved accordingly. “Preach what you practice” simply means to stop being nonjudgmental in public about moral principles that you hold in private.

I have no idea what Murray’s personal view is on the matter, but from the above passage I take it that he thinks it is OK for people to think that “it is morally wrong for a woman to bring a baby into the world knowing that it will not have a father,” and that it is “obsessively nonjudgmental” to not think it is morally wrong for a woman etc.

I think there’s something generational going on here. I actually know several women who’ve knowingly brought fatherless babies into the world—and they and their kids seem to be doing just fine. Yes, I realize there are people who find this sort of thing objectionable, but I’d actually think it was kind of rude for them to say something like that over dinner—at least, it would be rude if one of our friends with children were around.

So, yes, I agree with Murray that there have been big changes over the past few decades—but these changes go deeper than he might realize. In the last two sentences of the quotation above, Murray implies that members of the upper class hold in private the principle that it is morally wrong for a woman to bring a baby into the world knowing that it will not have a father. That might be true among the upper class of Dallas or other conservative parts of the country, but where I live, not so much.

At this point, Murray could of course respond that I’m making his point for him, that I’m illustrating how distant I am, in my Manhattan lifestyle, from average Americans. And I could respond with some poll data showing liberal attitudes on social issues among Americans under 40. But really this particular discussion would not be needed, because my point is simpler than that. Murray thinks it’s obsessionally nonjudgmental to think it’s OK for a woman to have a baby with no father, and I think it’s an odd sort of retro throwback to believe that such a woman’s behavior is morally wrong. In this case it can be difficult to communicate in general, although perhaps it would be possible in special cases. (Murray might, for example, agree that my friends might be OK having brought babies into the world knowing they will not have a father. Perhaps he could consider it morally wrong in general but accept my friends as special cases, perhaps arguing that this sort of think can work in Morningside Heights but not in many other places in the country.)

P.S. There’s a bunch of discussion below but I think some of it misses my main point, which is the following. Murray wrote, “try bringing up the issue of single women having babies at your next dinner party, and see how many of your companions are willing to say, even in a private gathering of friends, that it is morally wrong for a woman to bring a baby into the world knowing that it will not have a father.” I agree that my friends wouldn’t be “willing to say” this, but for the simple reason that I think most of my friends don’t think it is wrong for a woman to do that. Times have changed. Then again, lots of Americans would agree with Murray on the moral condemnation. Attitudes vary by geography, by generation, by many factors.

The post Charles Murray’s “Coming Apart” and the measurement of social and political divisions appeared first on Statistical Modeling, Causal Inference, and Social Science.

Daniel Murell has more thoughts on Turing chess (last discussed here):

When I played with my brother, we had it that if you managed to lap someone while running around the house, then you got an additional move. This means that if you had the option to take the king on your additional move, you could, and doing so won you the game. He was fitter at the time so he slipped in two additional moves over the course of the game. I still won :) I am much better at him at chess though, so I’m sure he would have beaten me had we been more even.

W.r.t. dsquared’s comment and your response, I’m not overly concerned about the first move, because you can enforce that white must reach a halfway point or that some time interval elapse before black makes his first move.

This version though does have one significant weakness that is evident to me. If you wait a little for your opponent to return to make his second move in a row against you, you get your breath back. He couldn’t plan for this tactic since it was your decision to wait. So he’s probably not played his first move to check your king. If he does play is second move and start running and you are in the position to take his king in two moves you can easily do it since you have had a short rest. His only defense would be to see exactly what you are up to when he comes back for his second move, and wait for you to move before he does so he can respond and so that you don’t get two moves in a row. If you also don’t respond, then this give time for him to catch his breath back too :) and so he will now be harder to lap when he finally does move. If you can out sprint him naturally though then he still has a problem and you would both wait indefinitely if both playing optimally. Can you see any solution to this problem? I’m still trying to decide on a rule set that works.

I agree that it would be great to have a standard set of rules for this game, a set of rules that are non-arbitrary (that is, no clock, no tuning parameters other than the length of the circuit to be run) for which the game is playable and is a reasonable balance between chess and running skills.

P.S. A google search turns up this idea:

The rules are the same as standard chess but the chess clock is set up some distance away, say 10 meters, on a separate table.

That’s ok, I guess, but the use of the chess clock makes it less beautiful. I want a pure version where the running is its own clock.

The post Goal: Rules for Turing chess appeared first on Statistical Modeling, Causal Inference, and Social Science.