Solution to question 5

From yesterday:

5. Which of the following better describes changes in public opinion on most issues? (Choose only one.)

(a) Dynamic stability: On any given issue, average opinion remains stable but liberals and conservatives move back and forth in opposite directions (the “accordion model”)

(b) Uniform swing: Average opinion on an issue can move but the liberals and conservatives don’t move much relative to each other (the disribution of opinions is a “solid block of wood”)

(c) Compensating tradeoffs: When considering multiple survey questions on the same general topic, average opinion can move sharply to the left or right on individual questions while the average over all the questions remains stable (the “rubber band model”)

Solution: b. You can make an argument for option a over the long term, but if you have to pick just one of the three, you have to go with uniform swing.

Wegman: “It’s not reprinted 100 percent like you had it.”

Wikipedia guy: “No, you added another paragraph at the end and you changed the headline. . . . You even copied the typos that I’ve corrected on my website. It was taken verbatim and reprinted in your paper.”

The original author got a check for $500 but, unfortunately, no free subscription to “Wiley Interdisciplinary Reviews: Computational Statistics” (a $1400-$2800 value).

P.S. To those who think I’m being mean to Wegman: I haven’t yet heard that he’s apologized to the people whose work he copied without attribution, or to the people who spent their time tracking all this down, or to the U.S. Congress for misrepresenting his expertise in his official report.

Everyone makes mistakes, and just about everyone has ethical lapses at times. But when you get caught you’re supposed to make apology and restitution.

(a) Dynamic stability: On any given issue, average opinion remains stable but liberals and conservatives move back and forth in opposite directions (the “accordion model”)

(b) Uniform swing: Average opinion on an issue can move but the liberals and conservatives don’t move much relative to each other (the disribution of opinions is a “solid block of wood”)

(c) Compensating tradeoffs: When considering multiple survey questions on the same general topic, average opinion can move sharply to the left or right on individual questions while the average over all the questions remains stable (the “rubber band model”)

Solution to question 4

From yesterday:

4. Researchers have found that survey respondents overreport church attendance. Thus, naive estimates from surveys overstate the percentage of Americans who attend church regularly. Does this have a large impact on estimates of time trends in religious attendance?

Solution: Yes. See this article by Hadaway, Marler, and Chaves, who write, “We suspect that the actual attendance rate has declined since World War II, despite the fact that the survey rate remained basically stable.”

Cell Transplantation is a medical journal published by the Cognizant Communication Corporation of Putnam Valley, New York. In recent years, its impact factor has been growing rapidly. In 2006, it was 3.482 [I think he means "3.5"---ed.]. In 2010, it had almost doubled to 6.204.

When you look at which journals cite Cell Transplantation, two journals stand out noticeably: the Medical Science Monitor, and The Scientific World Journal. According to the JCR, neither of these journals cited Cell Transplantation until 2010.

Then, in 2010, a review article was published in the Medical Science Monitor citing 490 articles, 445 of which were to papers published in Cell Transplantation. All 445 citations pointed to papers published in 2008 or 2009 — the citation window from which the journal’s 2010 impact factor was derived. Of the remaining 45 citations, 44 cited the Medical Science Monitor, again, to papers published in 2008 and 2009.

Three of the four authors of this paper sit on the editorial board of Cell Transplantation. Two are associate editors, one is the founding editor. The fourth is the CEO of a medical communications company.

Further details follow, but I think you get the picture.

Davis continues:

The ease to which members of an editorial board were able to use a cartel of journals to influence their journal’s impact factor concerns me greatly because the cost to do so is very low, the rewards are astonishingly high, it is difficult to detect, and the practice can be facilitated very easily by overlapping editorial boards or through cooperative agreements between them. What’s more, editors can protect these “reviews” from peer review if they are labeled as “editorial material,” as some are. It’s the perfect strategy for gaming the system.

I’d think this sort of thing should be detectable using statistical analysis. This seems like the sort
of fight-fire-with-fire situation where there will be at least a partial
technological solution.

P.S. The discussion thread is worth reading too.

P.P.S. Lots more interesting stuff on that blog, for example this post by Kent Anderson disparaging open-access publishing.

Solution to question 3

From yesterday:

3. We discussed in class the best currently available method for estimating the proportion of military servicemembers who are gay. What is that method? (Recall the problems with the direct approach: there is no simple way to survey servicemembers at random, nor is it likely that they would answer such a question honestly.)

Solution: I was talking about the work of Gary Gates, combining an estimate of the percentage of gays in the population with an estimate of the probability that someone is in the military, given that he or she is gay.

I have mixed feelings about Hastie’s article. On one hand I do think his point is important. It’s not new to me, but presumably it’s new to many readers of bloomberg.com. I like Hastie’s book (with Robyn Dawes), Rational Choice in an Uncertain World, and I’m predisposed to like anything new that he writes.

On the other hand, there’s something about Hastie’s article that bothered me. It seemed a bit smug, as if he thinks he understands the world and wants to just explain it to the rest of us. That could be fine—after all, Hastie is a distinguished psychology researcher—but I wasn’t so clear that he’s so clear on what he’s saying. For example:

The human brain is designed to support two modes of thought: visual and narrative. These forms of thinking are universal across human societies throughout history, develop reliably early in individuals’ lives, and are associated with specialized regions of the brain.

Is that really true? How does math fit into this picture? Or music? Music has a sort of narrative structure but it doesn’t seem quite like a story, either.

Hastie continues:

What isn’t universal or natural is the kind of highly structured cognitive processes that underlie logical and mathematical thinking.

Not natural . . . really? Maybe math is not universal, but certainly it’s natural. I was doing it when I was 2 years old. And music, that does seem to be universal, no?

Later on:

The mathematics of causal reasoning has recently experienced a major change, with the widespread acceptance of Bayesian Causal Networks as a normative, rational model for causal induction and reasoning.

Ummm . . . maybe Hastie is a bit too accepting of this particular story! I think Bayesian inference is great—I wrote two books on the topic!—but I wouldn’t go so far as to call it “a normative, rational model for causal induction and reasoning.” But I suppose that if I feel able to opine about psychology, I can’t object to Hastie expressing his views on statistics.

Hastie continues with a famous example:

The legendary theorists of decision-making Amos Tversky and Daniel Kahneman illustrated [our desire for stories] with the following pair of judgment questions: One group of respondents was asked, “What is the probability that a massive flood will occur sometime in the next year and drown more than 1,000 Americans?” The typical estimate was low (less than 20 percent). But, when another comparable sample of respondents was asked, “What is the probability that an earthquake in California will be followed by a flood in the next year that drowns at least 1,000 Americans?” the estimates were significantly higher.

The irrationality is that the second question is about a much more specific event, an earthquake that would be only one of the several reasons for the flood referred to in the first question. It is logically impossible for the second probability to be higher than the first. But, because the second question provides a plausible scenario for the unlikely outcome in the first query, our innate preference for a good story trumps our logical thinking skills.

This story is a great example of the availability heuristic, but I don’t see how it demonstrates a problem with “our logical thinking skills.” When responding to the first question, many people have difficulty visualizing that massive flood. The second question gives a clue. But I don’t see the combination of responses (coming from different sets of people) as indicating irrationality. Most people are not flood experts. They answer the questions as best they can, and when you give more information they will use it.

I hate to get all Gerd Gigerenzer on you here, but what’s the point of saying that this “trumps our logical thinking skills”? I think Kahneman and Tversky did better, decades ago, by writing of “heuristics and biases.”

What’s the political message here?

The article under discussion concludes with:

So the next time you hear a good story about why the financial recession, or any other economically significant event, was caused by a single collection of bad actors — or how a simple linear narrative “explains” an important event — remember this: Just as we are wired to like a diet rich in fats and sugars, we have an appetite for simple, coherent narratives. Neither habit is good for our long-term health.

(Reid Hastie, a professor of behavioral science at the University of Chicago Booth School of Business, is a contributor to Business Class. The opinions expressed are his own.)

Aaahhhh, now I get the message: The financial crisis is nobody’s fault! Let’s put aside the politics of blame, let’s all work together etc etc. OK, fine. Does this apply to all catastrophes? If you know someone in a plane that crashed, are we allowed to check if the pilot was stoned before takeoff? If someone takes $100,000 from you on a fraudulent pretext, and you catch him, are you allowed to try to collect? Or is it only in the financial crisis that we should set aside all “good stories” and “simple linear narratives”?

I agree that our financial problems our complex, and I’m all for warning people about the simplicity of storytelling, but I’m also a bit suspicious of someone from the University of Chicago School of Business telling me not to think about stories of the financial crisis.

Getting quantitative

Also, I’m surprised that, when people estimate “the probability that a massive flood will occur sometime in the next year and drown more than 1,000 Americans” as less than 20%, Hastie characterizes that estimate as “low.” Even Katrina drowned only 387 people (according to this source which I found by googling Katrina drownings). If a 20% chance of this “massive flood” occurring in a one-year period is “low,” I’d be interested in what Hastie thinks is a more reasonable probability estimate.

Responding

Hastie’s article bothered me for two reasons. First, what does it mean to it describe “the kind of highly structured cognitive processes that underlie logical and mathematical thinking” as “unnatural.” I don’t quite get what “natural” means here.

Second, I see an implicit political message, which seems to be that we shouldn’t blame anyone for the financial crisis:

We know there was no single cause or event that set in motion the crisis and that the truth is complex and multicausal. So why do we keep seeking the easy answers? It may be that we are hard-wired to do so.

Or, as the guy said in Repo Man, “it’s society’s fault.”

I contacted bloomberg.com, the publishers of the above-linked article, but was told:

We typically don’t publish opeds responding to articles we’ve published, though we welcome letters to the editor. We also post corrections to pieces containing factual errors and would gladly review any objections you have to Mr. Hastie’s column.

Fair enough, but in this case I don’t think the problems would be resolved by a correction note. I’m more bothered by the totality of the piece. For example, the claim that logical reasoning is “unnatural” is not quite a “factual error” but it still seems wrong to me.

P.S. Someone who knows the judgment and decision making field better than I do writes:

I don’t think that Reid has a political agenda here. (He has only been at Chicago for a few years, and Chicago’s School of Business is not monolithic.) . . . To say that blame narratives are oversimplified is not the same as saying that nobody should be blamed; you may be reading the latter subtext into his text.

So maybe I was being unfair. Although I’d feel a little better about Hastie’s column if he’d clarified that, even though stories can be oversimplified, the “life is complicated” defense shouldn’t be used to get people off the hook.

Also, I’m still unhappy about the claim that logical and mathematical reasoning is “unnatural.” But this fits with the innumeracy of thinking there’s a greater-than-20%-chance of a major flood in any given year. I feel that, to Hastie, numbers are just words. Which is consistent with the idea that mathematical reasoning is unnatural to him.

Solution to question 2

From yesterday:

2. Which of the following are useful goals in a pilot study? (Indicate all that apply.)

(a) You can search for statistical significance, then from that decide what to look for in a confirmatory analysis of your full dataset.

(b) You can see if you find statistical significance in a pre-chosen comparison of interest.

(c) You can examine the direction (positive or negative, even if not statistically significant) of comparisons of interest.

(d) With a small sample size, you cannot hope to learn anything conclusive, but you can get a crude estimate of effect size and standard deviation which will be useful in a power analysis to help you decide how large your full study needs to be.

(e) You can talk with survey respondents and get a sense of how they perceived your questions.

(f) You get a chance to learn about practical difficulties with sampling, nonresponse, and question wording.

(g) You can check if your sample is approximately representative of your population.

Solution: e and f. The purpose of a pilot study is to test out the data collection. The sample size will be too small for a, b, c, d, and g. In some of their earliest work, Kahneman and Tversky documented the common misconception of researchers that data from a small pilot study should closely match the population.

The question would have clearer if I’d inserted the word “small” before “pilot” in the preamble.

See also here. He did Wacky Packs!

(a) You can search for statistical significance, then from that decide what to look for in a confirmatory analysis of your full dataset.

(b) You can see if you find statistical significance in a pre-chosen comparison of interest.

(c) You can examine the direction (positive or negative, even if not statistically significant) of comparisons of interest.

(d) With a small sample size, you cannot hope to learn anything conclusive, but you can get a crude estimate of effect size and standard deviation which will be useful in a power analysis to help you decide how large your full study needs to be.

(e) You can talk with survey respondents and get a sense of how they perceived your questions.

(f) You get a chance to learn about practical difficulties with sampling, nonresponse, and question wording.

(g) You can check if your sample is approximately representative of your population.

Solution to question 1

From yesterday:

1. Suppose that, in a survey of 1000 people in a state, 400 say they voted in a recent primary election. Actually, though, the voter turnout was only 30%. Give an estimate of the probability that a nonvoter will falsely state that he or she voted. (Assume that all voters honestly report that they voted.)

Solution: Draw the probability tree, you get that the proportion of people who say they voted is .3+.7p. Solve .3+.7p=.4, you get p=(.4-.3)/.7=.14, or 14%. I was also going to ask for the standard error (which you’d obtain by starting with the standard error for the “.4″ and propagating that through) but I decided to keep it simple. As it was, only about half the students got this question right. This is not a knock on the kids—I just didn’t teach this material well—I’m just letting you know to give a sense that this isn’t such an easy problem.

P.S. As some commenters note, Problem 1 isn’t so realistic. Commenter awm points out that “for the most part people aren’t lying and that the sorts of people who participate in surveys about elections are disproportionately the sort of people who vote.” My problem would’ve been cleaner if I’d also said to assume there was no nonresponse, and if I’d chosen a better example!

But maybe I’m wrong on this. Jay Livingston argues that black and white are colors whereas Black and White are races (or, as I would prefer to say, ethnic categories) and illustrates with this picture of a white person and a White person:

In conversation, I sometimes talk about pink people, brown people, and tan people, but that won’t work in a research paper.

P.S. I suspect Carp will argue that I’m being naive: meanings of words change across contexts and over time. To which I reply: Sure, but I still have to choose how to write these words!

P.S. The commenters are picking up some of the unintended “Hare and pineapple” ambiguity in my question!

Hartzing reports the following published claims:

Harvey (1996: 103): `The rate of failure of expatriate managers relocating overseas from United States based MNCs has been estimated to range between 25±40 per cent (Tung, 1982, 1988; Mendenhall and Oddou, 1985; Gray 1991; Wedersphan, 1992; Solomon, 1994; Dowling, Schuler and Welch, 1994; Swaak, 1995).’
Shay and Bruce (1997: 30): `Cross-industry studies have estimated US expatriate failure, defined as premature return from an overseas assignment, at between 25±40 per cent for developed countries (Baker and Ivancevich, 1971; Tung, 1981).’

Ashamalla (1998: 54): `According to a number of recent [emphasis added] studies, the rate of failure among American expatriates ranges from 25 to 40 per cent depending on the location of the assignment (Dumaine, Fortune, 1995; McDonald, 1993; Ralston et al., 1995).’

Hartzing writes that, despite the profusion of references which appear to show multiple confirmations, these claims all comes from a single publication from 1979 which itself gives no source for its numbers.

If you believe Hartzing on this (and I see no reason not to), this is not about one or two sloppy researchers; rather, it seems to be general practice for people to thrown in citations without reading the original articles, thus creating a statistical problem of increasing the apparent N by treating multiple instances of the same claim as if they were independent pieces of supporting evidence.

The course was offered in the political science department and covered a mix of statistical and political topics. Followers of our recent discussion on test questions won’t be surprised to learn that some of the questions are ambiguous. This wasn’t on purpose. I tried my best, but good questions are hard to write.

Question 1 will appear tomorrow.

Dear Dr. Gelman,

I am emailing to let you know that your accepted article for Economic Inquiry will be published in print in the forthcoming April 2012 Issue. You will be receiving hard copies of the journal from Wiley-Blackwell for distribution to yourself and the Co Authors.

Hmmm . . . Economic Inquiry . . . didn’t I publish something there once? A quick check turned up this paper from 2010.

I wonder what this new paper is. Did someone submit something with my name on it? I remember my surprise when, many years ago, I received a postcard asking for a reprint of my article in the Journal of Cerebral Blood Flow and Metabolism. I was sure they were looking for the wrong Andrew Gelman, but, no, it turned out that my coauthors had submitted that article all on their own.

In this case, though, there was no new article. Economic Inquiry was indeed talking about my 2010 paper, which appeared online two years ago but is coming out in print only this month.

P.S. To add insult to injury: I wrote this post in March but it’s not appearing until May because I have a long lead time for non-topical entries on this blog.

Some of my fragmentary work on varying treatment effects is here (Treatment Effects in Before-After Data) and here (Estimating Incumbency Advantage and Its Variation, as an Example of a Before–After Study).

To all:

Here’s the abstract of the work I’ve done this summer. It’s stored in the file,
/fs5/gelman/abstract.bell, and copies of the Figures 1-3 are on Trevor’s desk.
Any comments are of course appreciated; I’m at gelman@stat.berkeley.edu.

On the Routine Use of Markov Chains for Simulation

Andrew Gelman and Donald Rubin, 6 August 1990

corrected version: 8 August 1990

1. Simulation

In probability and statistics we can often specify multivariate distributions
many of whose properties we do not fully understand–perhaps, as in the
Ising model of statistical physics, we can write the joint density function, up
to a multiplicative constant that cannot be expressed in closed form.
For an example in statistics, consider the Normal random
effects model in the analysis of variance, which can be
easily placed in a Bayesian framework with a conjugate prior distribution.
All the conditional densities of the resulting posterior distribution
are simple, but marginal densities can only be written in integral form and
can only be calculated approximately. (For details, see Kinderman and Snell
(1980) or Pickard (1987) for the Ising model, and Lindley and Smith (1972)
for the Bayesian random effects model.)

In such cases, we may not even be able to compute marginal moments of the
difficult distribution, let alone more complicated and interesting summaries
that would help us understand a probability model or posterior inference.
When direct methods such as analytic or numerical integration of “nuisance”
parameters are not computationally feasible, we might try Monte Carlo simulation;
in the simplest form, we draw a finite set of independent random samples from our
distribution, and then calculate desired distributional summaries as functions of
the sampled points. The Monte Carlo method is quite general and powerful; it is
easy to calculate arbitrary quantities of interest
such as the expected long-distance correlation in the Ising model or a posterior
95% confidence region for the largest block effect in a random effects model.
Any aspect of the distribution can be approximated to any desired accuracy if
the number of independently sampled points is large enough.
Simulation also has the advantage of flexiblility: once a sample is drawn, it
can be used to learn about any number of different distributional summaries.

2. Markov chain methods

Drawing independent random samples is a wonderful tool that is unfortunately not
available for every distribution; in particular, the Ising model and random
effects posterior distributions mentioned above do not permit direct
simulation. Fortunately, a form of indirect simulation method exists for
any multivariate distribution if we can calculate its joint density
(up to a multiplicative constant) or if we can sample from all its univariate
conditional densities. The first of these methods was introduced by
Metropolis et al. (1953) in the Journal of Chemical Physics. Our work focuses
on a similar and slightly simpler method called the Gibbs sampler by Geman and
Geman (1984) in an article for the IEEE.

Let F(x) be our distribution; the Metropolis algorithm takes a starting (vector)
point x0 and constructs a series x1, x2, . . ., that is a sample from an
ergodic Markov chain whose stationary distribution is F(x). Computer
simulation of the series requires calculation of the density f(x) (up to a
constant). These samples xj are not independent; however, the stationary
distribution of the Markov chain is
correct, so if we take a long enough series, the set of values {x1, . . ., xn}
takes the place of the distribution just as an
independent random sample does (although of course an independent sample
carries more information than a Markov chain sample of the
same length).

The Gibbs sampler is a similar algorithm, which produces a Markov chain that
converges to the desired distribution, this time requiring draws from all the
univariate conditional densities at each iteration.

3. Have we converged yet?

Markov chain simulation methods are attractive for many problems because they
enable us to flexibly summarize intractable multivariate distributions by making
full use of the mathematical structure we do know, using a tool we think we
understand–Monte Carlo simulation. Unfortunately, using a sample of a Markov
chain to estimate a distribution raises an immediate question: how long a series
is needed? After one or two steps, we are almost certainly still too close to
the starting point to hope for unbiased summaries. Asymptotically, the chain is
stationary, and all is OK (with some loss of efficiency compared to independent
samples, as mentioned above).

To obtain a feeling for the practical difficulties, we ran the Gibbs sampler for
2000 steps to simulate a case of the Ising model. To give the minimum of details:
x is a vector of binary variables defined on a 100 by 100 lattice; each step of
the Gibbs sampler took on the order of 10,000 computations; and we summarize
each iterate xj by the sample correlation r on the lattice–a function r(x) that
lies between -1 and 1. Theoretical calculations (Pickard, 1987) show that
under our model–the Ising model with beta = 0.5–the marginal distribution
of r is approximately Gaussian with mean around 0.85 or 0.9 and standard
deviation around 0.01. We’d like to know whether the set {r(x1), . . .,
r(x2000)} from the simulated Markov chain can serve as a substitute for the
marginal distribution of r.

Figure 1 shows the values of r(xj), for j=1 to 2000. (r(x0) = 0, and the first
few values are cut off to improve resolution on the graph.) The Markov chain
seems to have “converged to stationarity” after the thousand or so steps required
to shake off its initial state. How do we know it has converged, though? Figure
2 zooms in of the first 500 steps of the series, whose apparent convergence we
know to be illusory. For comparison we ran the Gibbs
sampler again for 2000 steps, but this time starting at a point x0 for which
r(x0) = 1; Figure 3 displays the series r(xj), which again seems to have
converged nicely. To destroy all illusions about convergence, hold
Figures 1 and 3 up to the light. The two Markov chains have “converged” to
different distributions! We are, of course, still observing transient
behavior.

Interestingly, the means of the series in Figures 1 and 3 differ, but
the variances are roughly equal. We’re not sure why, but it seems to be a
general feature in these Markov chain simulations that the variance converges
before the mean.

All simulations and plots were done using the New S Language:
A Programming Environment for Data Analysis and Graphics.

4. The answer: parallel Markov chains

To restate the general problem: we wish to summarize an intractable
distribution F(x) by running the Gibbs sampler (or a similar method such as the
Metropolis algorithm) until the distribution of the set of Markov chain
iterates is close to F. As shown in the previous section, convergence seems
impossible to monitor from a single finite realization of the Markov chain;
consequently, we follow the implicit suggestion of Figures 1 and 3 and track
several parallel sample paths.

Consider m independent runs of the Gibbs sampler, each of length n, starting
from m different initial states x10, . . ., xm0:

series 1: x11, . . ., x1n
. . .
series m: xm1, . . ., xmn.

Again, we focus attention on a univariate summary, say r(x); we want to
use the observed simulations rij to determine whether the series of r’s are
close to convergence after n steps.
To understand our method, consider the set of series as m blocks, each with
n observations, in the one-way analysis of variance layout (that is, ignore the
time ordering in the series). We will work with the total sum of squares
(with (mn-1) degrees of freedom) and the “within” sum of squares (with
(m-1)(n-1) degrees of freedom).

First assume for simplicity that the starting points of the simulated series
are themselves independent random samples from F(x). (Of course, if this
condition could be obtained in practice,
a Markov chain simulation method would not be
needed.) With independent starting points, all values of any series
are independent of all the values of any other series, and the unconditional
variance of any point rij is just the marginal variance var r under the
distribution F. We can then estimate var r, given the “data matrix” (rij),
by [total SS - (within SS)/m] / ((m-1)n). (Algebraic derivations appear
in the longer version of this article.) Given the assumption of initial
independence, this “between” estimate of variance
(not the same as the usual “between” estimate in ANOVA) is unbiased for finite
series of any length.

In contrast, the estimated variance within the series, (within SS) / ((m-1)(n-1)),
has expectation var r only in the limit as n -> infinity.
For finite series, the expected within mean square increases with n, assuming,
as is likely, that the random variables r(x1), r(x2), . . ., from the Markov
chain are positively correlated. The discrepancy between the two estimates
of var r suggests a test: declare the Markov chain to have converged when
the within mean square is close to the variance estimate between series, with
confidence intervals derived from classical ANOVA theory. Because of the
dependence within blocks, the degrees of freedom of the between and within
estimates are less than (m-1)n and m(n-1), respectively. We can
approximately correct for this information loss (once again, details will be
provided in the longer article).

Once we are close enough to convergence to be satisfied, the variance estimates
and degrees of freedom corrections alluded to above allow us to estimate the
marginal summaries E r, var r, and Normal-theory confidence intervals for our
Monte Carlo approximations. We can run the series longer if more precision
is desired, and can repeat the process to study the marginal distributions of
other parameters (without, of course, having to simulate any new series of x’s).

In practice, the starting points of the parallel series can never be sampled
independently with distribution F(x); the simulated series are thus no longer
stationary for any finite n, formally invalidating the above analysis. We
currently have two strategies designed to make the independence assumption
approximately true. First, we try to pick starting values that are far apart and,
if anything, more dispersed than independent random samples. The m parallel series
should then start far apart and grow closer as they approach stationarity, as in
Figures 1 and 3; since the variance between series declines with n, the
comparison-of-variances test should be conservative. Second, we reduce the
effect of the starting values by crudely throwing away the the first half
of each simulated series until approximate convergence has been reached.
Once again, Figures 1 and 3 illustrate how a few early steps
of the Markov chain can greatly distort estimates of means and variances
within series. We hope that the conservative strategies of starting with
dispersed points and throwing away early simulations will yield confidence
regions that are wider than those obtained by the ideal method, but that
still have good coverage properties.

The idea of comparing parallel simulations is not new; for
example, Fosdick (1959) applied the Metropolis algorithm to the Ising model
by simulating four series independently, from each of two different starting
points. Approximate convergence was declared when the two groups of series
became indistinguishable on the scale of a prechosen error bound.

5. Some references

Ehrman, J. R., Fosdick, L. D., and Handscomb, D. C. (1960).
Computation of order parameters in an Ising lattice by the Monte Carlo method.
{\em Journal of Mathematical Physics} {\bf 1} 547–558.

Fosdick, L. D. (1959). Calculation of order parameters in a binary
alloy by the Monte Carlo method. {\em Physical Review} {\bf 116}, 565–573.

Geman, S., and Geman, D. (1984). Stochastic relaxation, Gibbs
distributions, and the Bayesian restoration of images. {\em IEEE Transactions
on Pattern Analysis and Machine Intelligence} {\bf 6}, 721–741.

Hammersley, J. M., and Handscomb, D. C. (1964), chapter 9. {\em Monte Carlo
Methods}. London: Chapman and Hall.

Kinderman, R., and Snell, J. L. (1980). {\em Markov Random Fields and
their Applications}. Providence, R.I.: American Mathematical Society.

Lindley, D. V., and Smith, A. F. M. (1972). Bayes estimates for the linear
model. {\em Journal of the Royal Statistical Society B} {\bf 34}, 1–41.

Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and
Teller, E. (1953). Equation of state calculations by fast computing machines.
{\em Journal of Chemical Physics} {\bf 21}, 1087–1092.

Pickard, D. K. (1987). Inference for discrete Markov fields: the
simplest nontrivial case. {\em Journal of the American Statistical Association}
{\bf 82} 90–96.

Ripley, B. D. (1981). {\em Spatial Statistics}, p. 16ff. New York: Wiley.

Tanner, M. A., and Wong, W. H. (1987). The calculation of posterior
distributions by data augmentation. {\em Journal of the American Statistical
Association} {\bf 82}, 528–550.

I wrote the article but properly listed Rubin as coauthor, as the idea came about after many long phone conversations. I encountered the idea of between-within comparison in the 1959 paper by Fosdick (see above citations); I can’t remember how I found that paper but it must have been from a literature search, going backward from more recent sources. Anyway, when I brought up this idea, Rubin picked up on it right away, as it was close to methods he had developed for inference from multiple imputations. Once we had that connection, the idea was there. And I’d credit Rubin’s influence for my goal of estimating a potential scale reduction factor—that is, a numerical measure of lack of mixing—rather than formulating the problem as a hypothesis test.

The published article appeared over two years later in the journal Statistical Science, in a much expanded version.

In some ways, I prefer this short paper to the full version. I like the snappy style, and I like the clarity about what we believe and what we don’t know. I regret not submitting some version of the above article to a journal immediately, right then in Aug 1990. On the other hand, editors and reviewers for statistics journals can be very stuffy, and an article such as above with a concept but no theoretical derivations probably would’ve been shot down over and over and over. Maybe it just took two years to put in enough blah blah blah to make it publishable.

The above is more like a blog post than a journal article. It contains the key idea with no messing around.

P.S. You’ll notice above that I wrote, “Any comments are of course appreciated.” And you probably won’t be surprised to hear that I got no comments. It took me a long time to realize that most people don’t want to comment on things. When we were getting close to finishing the first edition of Bayesian Data Analysis back in 1994, I printed out copies and gave them to lots of prominent statisticians I knew, but very few gave any comments at all. It’s not about me; people just don’t like to read and make comments. We get some comments on the blog, but when you consider the number of comments and the number of readers, you’ll realize that most people don’t comment here either.

The conventional wisdom that’s developed over the past few decades — based on early research — has said parents are less happy, more depressed and have less-satisfying marriages than their childless counterparts.

But now, two new studies presented as part of the Population Association of America’s annual meeting suggest that earlier findings in several studies weren’t so clear-cut and may, in fact, be flawed. The newer analyses presented this week use analytical methods based on data from almost 130,000 adults around the globe — including more than 52,000 parents — and the conclusions aren’t so grim. They say that parents today may indeed be happier than non-parents and that parental happiness levels — while they do drop — don’t dip below the levels they were before having children. . . .

The other study, of some 120,000 adults from two nationally representative surveys between 1972-2008, finds that parents were indeed less happy than non-parents in the decade 1985-95, but parents from 1995 to 2008 were happier. . . .

This is consistent with my observation that happiness research is a mess. Don’t get me wrong: I think happiness is very much worth studying. But various seemingly well-known results don’t seem so clear when they are studied more carefully. This is not the first time that Happiness Scholar #2 comes to an opposite conclusion as Happiness Scholar #1.

The larger point, perhaps, is that it that “stylized facts” (the social-science term for generally-accepted findings) can mislead. Sometimes the interaction is bigger than the main effect.

Here’s a recent paper by Mikko Myrskylä, one of the researchers mentioned in the above article:

The literature on fertility and happiness has neglected comparative analysis. We investigate the fertility/happiness association using data from the World Values Surveys for 86 countries. We find that globally, happiness decreases with the number of children. This association, however, is strongly modified by individual and contextual factors. Most importantly, we find that the association between happiness and fertility evolves from negative to neutral to positive above age 40, and is strongest among those who are likely to benefit most from upward intergenerational transfers. In addition, analyses by welfare regime show that the negative fertility/ happiness association for younger adults is weakest in countries with high public support for families, and the positive association above age 40 is strongest in countries where old-age support depends mostly on the family.

The news article ends with the following comment:

“The first child increases happiness quite a lot. The second child a little. The third not at all,” says Myrskylä.

As a statistician, I hate hate hate hate hate when people ignore variability and present results deterministically. The above statement might be an accurate summary of average patterns but is certainly not true in every case!

Part of the story is that I do research for a living so I resent people who devalue research through misattribution or fraud, in the same way that rich people don’t like counterfeiters.

What really bugs me, though, is when cheaters get caught and still don’t admit it. People like Hauser, Wegman, Fischer, and Weick get under my skin because they have the chutzpah to just deny deny deny. The grainy time-stamped videotape with their hand in the cookie jar is right there, and they’ll still talk around the problem. Makes me want to scream.

This happens all the time. All. Over. The. Place.

Everybody makes mistakes, and just about everybody does things they shouldn’t, every now and then. But to not apologize when you’re caught, that to me just seems evil, showing a disrespect not just for the people involved but for the very concept of truth. (As you can see, I wouldn’t make a good criminal defense lawyer.)

Anyway, here’s the latest story. There’s always an outrage-of-the-week, and I don’t mean to make a big deal about this particular scandal. It’s just another example of what I consider the disgraceful pattern of people refusing to admit error.

Part 1: The mistake

The following question was on a New York State eighth grade reading exam:

The Hare and the Pineapple

by Daniel Pinkwater

In olden times, the animals of the forest could speak English just like you and me. One day, a pineapple challenged a hare to a race.

(I forgot to mention, fruits and vegetables were able to speak too.)

A hare is like a rabbit, only skinnier and faster. This particular hare was known to be the fastest animal in the forest.

“You, a pineapple have the nerve to challenge me, a hare, to a race,” the hare asked the pineapple. “This must be some sort of joke.”

“No,” said the pineapple. “I want to race you. Twenty-six miles, and may the best animal win.”

“You aren’t even an animal!” the hare said. “You’re a tropical fruit!”

“Well, you know what I mean,” the pineapple said.

The animals of the forest thought it was very strange that tropical fruit should want to race a very fast animal.

“The pineapple has some trick up its sleeve,” a moose said.

Pineapples don’t have sleeves, an owl said

“Well, you know what I mean,” the moose said. “If a pineapple challenges a hare to a race, it must be that the pineapple knows some secret trick that will allow it to win.”

“The pineapple probably expects us to root for the hare and then look like fools when it loses,” said a crow. “Then the pineapple will win the race because the hare is overconfident and takes a nap, or gets lost, or something.”

The animals agreed that this made sense. There was no reason a pineapple should challenge a hare unless it had a clever plan of some sort. So the animals, wanting to back a winner, all cheered for the pineapple.

When the race began, the hare sprinted forward and was out of sight in less than a minute. The pineapple just sat there, never moving an inch.

The animals crowded around watching to see how the pineapple was going to cleverly beat the hare. Two hours later when the hare cross the finish line, the pineapple was still sitting still and hadn’t moved an inch.

The animals ate the pineapple.

MORAL: Pineapples don’t have sleeves

Several questions follow, including:

The animals ate the pineapple most likely because they were

A Hungry
B Excited
C Annoyed
D Amused

Which animal spoke the wisest words?

A The hare
B The moose
C The crow
D The owl

Nope, I don’t know how to answer these either.

Part 2: Exposure

The story hits the Daily News and other media outlets. To its credit, the state education commission takes the question off the test. More on the story here.

Part 3: No admission of error

From Time magazine, a memo from Jon S. Twing of Pearson, the company that made the test with the silly questions:

Pearson is confident that the NYS Grades 3-8 English Language Arts (ELA) and Mathematics assessments have been developed to support valid and reliable interpretations of scores for their intended uses. The “Hare and the Pineapple” passage and associated items were placed on the Grade 8 ELA test after the NYSfield test data associated with the multiple choice items and the feedback from the “final eyes” committee determined that this was an appropriate passage and set of items to include on the test. . . .

“The Hare and the Pineapple” passage is intended to measure NYS Standard “interpretation of character traits, motivations, and behavior” and “eliciting supporting detail”. . . .

Blah blah blah . . . But now the good part. Jon S. Twing writes:

There have been two items of the set of six that have been challenged by NY teachers and students as the test was under way April 17-19, 2012 -Item 7 and Item 8. The correct answers and rationales to Item 7 and Item 8 are as follows:

• Item 7: The correct answer is C. The question regarding the animals’ possible motivation for eating the pineapple requires a reader to infer the correct answer from clues conveyed in the text. While all of the options are plausible motivations, the most likely answer is that the animals were annoyed. Paragraph 13 indicates that the animals support the pineapple to win the race because they assume the pineapple has a clever plan. However, the pineapple never moves during the race. From these clues and events, a reader can infer that the animals are annoyed. The text does not support the inference that the animals are motivated by hunger, excitement, or amusement.

• Item 8: The correct answer is D. The question regarding the wisest animal requires the reader to apply close analytic reading skills to determine which of the choices represents the wisest animal based on clues given in the text. The moose and the crow are the two animals that present the incorrect idea that the pineapple has a clever plan to win the race. This idea is proven false when the hare wins the race. The hare is presented as incredulous that a pineapple would challenge him to a race, but overconfidently agrees to race a pineapple.

Huh? The hare “overconfidently agrees to race a pineapple”??? What’s overconfident about that? A pineapple can’t run at all!

Twing continues:

Finally, the owl declares that “Pineapples don’t have sleeves,” which is a factually accurate statement. This statement is also presented as the moral of the story, allowing a careful reader to infer that the owl is the wisest animal.

Sorry, Jon S. Twing, but that doesn’t make sense at all. The moose makes it perfectly clear that “a trick up the sleeve” is just a figure of speech. The owl is wise for taking a figure of speech literally? In that case, why not kill the joke entirely with “hares can’t talk”? Grrrrrr.

You gotta remember that the people taking this exam are 8th graders who are trying to not get tricked themselves.

And, by the way, I have a feeling “the moral of the story” is a joke, in the style of James Thurber’s Fables for Our Time.

Twing wraps things up with a bunch of words and numbers suggesting that these questions have good psychometric properties, i.e. that students who got other questions right tended to get these right also. That’s interesting: even an impossible-to-answer question has some patterns which some students are able to match. But that doesn’t make these good exam questions.

The psychometrics are relevant but I just think it’s embarrassing that this poor guy is in the position of claiming, first, that the animals are annoyed, when there is no evidence at all of that in the quoted passage, and, second, all that business of pineapples and sleeves.

Why not just say: Hey, it’s not easy to write a test! We made a mistake! Sorry!

But nooooooo, he can’t do that, he’s gotta defend defend defend defend defend. OK, I’m not calling him a Wegman—there’s no reason to suspect that any botched wikipedia copying went down—but that makes the whole thing even worse, in a way. These people didn’t even do anything intentionally wrong. They just made a mistake. But they still won’t admit it. “The correct answer is C,” indeed.

Here I am, railing against universal human nature again. That’s what blogging is for, I suppose.

Here’s my favorite recent item, a letter sent to the Seattle Bureau of Prohibition in 1931:

Dear Sir:

My husband is in the habit of buying a quart of wiskey every other day from a Chinese bootlegger named Chin Waugh living at 317-16th near Alder street.

We need this money for household expenses. Will you please have his place raided? He keeps a supply planted in the garden and a smaller quantity under the back steps for quick delivery. If you make the raid at 9:30 any morning you will be sure to get the goods and Chin also as he leaves the house at 10 o’clock and may clean up before he goes.

Thanking you in advance,

I remain
yours truly,

Mrs. Hillyer

But this one is still my favorite.

Few truly insignificant traits receive as much attention as left-handedness. In just the last couple of generations, an orientation once associated with menace has become associated with leadership, creativity, even athletic prowess. Presidents Gerald R. Ford, George Bush, Bill Clinton and Barack Obama were born left-handed (as was Ronald Reagan, though he learned to write with his right hand). Folklore has it that southpaws are unusually common in art and architecture schools. Left-handed athletes like Tim Tebow and Randy Johnson are celebrated.

Sounds interesting so far. Then we get several paragraphs of history of how people got things wrong (authoritarians of past generations who forced lefties to use their right hands, silly “blank slate” ideologues, etc.).

What about the science? Smits writes:

Left-handers have been redefined as creative, broad-minded, natural leaders. Meanwhile, other studies continue to identify all sorts of negative associations with left-handers — clumsiness, propensity to die prematurely, higher breast cancer rates and greater vulnerability to suicide.

After reviewing hundreds of such studies for a book on left-handers, I [Smits] found that the evidence of positive qualities associated with left-handedness was anecdotal at best, while the scores of studies associating left-handedness with all manner of afflictions were generally too unreliable to have any practical consequence.

I’d have to see Smits’s book to judge (and, before you get on my case for commenting on a book that I haven’t read, please reflect that Smits chose to publish his article in the Times, and I think it’s expected that many people will write a newspaper article without reading the corresponding book. Certainly, when I wrote op-eds about Red State Blue State, I wanted these to stand on their own for the benefit of the vast majority of newspaper readers who were not reading the book), but I’m skeptical of his skepticism. The studies I’ve seen of handedness do have potential problems, so I wouldn’t object to labeling as “speculative” such claims such as “mathematicians are more likely to be left-handed” or “left-handers live shorter lives than right-handers.” At the same time, such claims are not scientifically implausible, and they do seem supported to some extent by the data.

I have not looked at the research on handedness recently, so I’m not sure whether Smits’s skepticism reflects new information or whether it is just a statement that the claimed findings about left-handers have not been proved. If the latter, I think it would be better to say that it’s not clear to what extent left-handers are different from the majority, rather than to say with such certainty that “lefties aren’t special at all.”

P.S. I’m right-handed. I don’t have any personal stake in all this; it’s a topic I got interested in awhile ago when Seth and I taught our class on left-handedness. At the time we recognized the weakness of the studies on the topic, but we also recognized weaknesses in sweeping arguments that attempted to dismiss the findings by arguing for selection effects etc. My impression at the time was that there was some evidence of interesting and important systematic differences between lefties and righties, but that it was also possible we were seeing nothing more than a bunch of statistical artifacts. My conclusion was that skepticism was warranted but it would be going too far to be certain that nothing was going on.

You can only accept capital punishment if you’re willing to have innocent people executed every now and then

The politics of America’s increasing economic inequality

How many zombies do you know?’ Using indirect survey methods to measure alien attacks and outbreaks of the undead, Arxiv preprint arXiv:1003.6087, 2010
I hope you would find interesting the following paper, recently posted on arXiv:
Aliens on Earth. Are reports of close encounters correct?, arXiv:1203.6805

This is soooooo much better than getting links to bad graphs or to papers on sex ratios!

I didn’t like this analysis because it is equivalent to fitting a step function. There is a tendency for people to interpret the (arbitrary) tercile boundaries as being meaningful thresholds even though the underlying dose-response relation has to be continuous. I’d prefer to start with a linear model and then add nonlinearity from there with a spline or whatever.

At this point I stepped back and thought: Hey, the divide-into-three analysis does not literally assume a step function. It doesn’t assume anything at all; it’s just a data summary! People discretize input variables all the time! So why am I complaining?

I justify my complaints on two levels. First on the grounds of interpretation: my applied colleagues really were interpreting the three-category model in terms of thresholds. The three categories were: “0 to A”, “A to B”, and “B to infinity”. And somebody really was saying something about the effect of exposure A or exposure B. Which just ain’t right.

My second issue is statistical efficiency. You can say that the categorical-input model is nothing but a summary, an estimate of averages—but by binning like this, you lose statistical efficiency. And you become the slave to “statistical significance”; there’s the temptation to butcher your analysis and throw away tons of information, just so you can get a single clean, statistically significant result.

P.S. The more categories you have, the less of a concern it is to discretize. And sometimes your data come in discrete form (see here, for example).

• Microsoft Research New York City with John Langford (machine learning), Duncan Watts (networks), and Dave Pennock (algorithmic economics).
• eBay technology center focusing on data – led by Chris Dixon, the co-founder of the recommendation engine company Hunch, which has recently been acquired by eBay.

New York already has Facebook’s engineering unit, Twitter’s East Coast headquarters, and Google’s second-largest engineering office.

The data community here is on an upswing, and it might be one of the best places to be if you’re into applied statistics, machine learning or data analysis.

Post by Aleks Jakulin.

P.S. (from Andrew): The formerly-Yahoo-now-Microsoft researchers have a more-or-less formal connection to Columbia, through the Applied Statistics Center, where some of them will be organizing occasional mini-conferences and workshops!

(Please click through and read the whole thing.)

Some discussion of revision of the Nuts paper, some conversations about parameterizations of categorical-data models, plans for the R interface, blah blah blah.

But also, I had two exciting new ideas!

Google Translate for code

Wouldn’t it be great if Google Translate could work on computer languages? I suggested this and somebody said that it might be a problem because code isn’t always translatable. But that doesn’t worry so much. Google Translate for human languages isn’t perfect either but it’s a useful guide. If I want to write a message to someone in French or Spanish or Dutch, I wouldn’t just write it in English and run it through Translate. What I do is try my best to write it in the desired language, but I can try out some tricky words or phrases in the translator. Or, if I start by translating, I go back and forth to make sure it all makes sense.

An R help-list bot

We were talking about how to build a Stan community that will be helpful to a diverse range of users without taking up too much of our time, and that’s when I came up with a brilliant idea. Let’s take a successful existing help group—for example, the R-help mailing list—then make a database of the helpful bits of advice of a distinguished and frequent contributor to the list. The bot would be easy: whatever the question is that comes in, just send back a random tip. I have a feeling that advice such as “PLEASE do, and not send HTML” and “My guess is that this is a Mac-specific question (e.g. you are using the R.app GUI), so please consider if this is the appropriate list” and “The posting guide was not followed” and “Please use the R-devel list to comment on current development versions” would work pretty well for almost any question (maybe after some global sub of Stan for R).

It’s sort of like when we were kids and had this book, Bennett Cerf’s Book of Riddles—it was a gray with a picture of a big red rock-eater on the cover. We started out by just reading through the riddles one at a time but we had more fun after inventing a game where we’d open the book and read a riddle, then open again at random to give the answer. For example:
Q: What’s big, red and eats rocks?
A: Two cats stuck in a tree!

Recently I was disturbed (but, I’m sorry to say, not surprised) to see Seth post the following:

Predictions of climate models versus reality. I [Seth] have only seen careful prediction-vs-reality comparisons made by AGW [anthropogenic global warming] skeptics. Those who believe humans are dangerously warming the planet appear to be silent on this subject.

In response, Phil commented:

Funny, on the day you [Seth] made your post saying that you haven’t seen comparisons between models and predictions except by skeptics, the top entry on RealClimate, the single most prominent global-warming-related blog that is not run by skeptics, was “Evaluating a 1981 temperature projection.”

Pretty amazing, huh? On its face it would seem surprising to claim that the majority of leading climate scientists don’t do “careful prediction-vs-reality comparisons,” and indeed on the very day of Seth’s post, there is such a comparison right there on the first place you might look for what the climate scientists are doing!

How did Seth miss it?

A clue comes from the sources on which Seth relies. His link above points to the webpage of the Von Mises Institute, a political advocacy organization. Other Seth links to global warming stories have come from an climate change skeptic blog, a right-leaning politics blog, another climate change skeptic blog, another advocacy organization, one more climate change skeptic blog, a letter to the Wall Street Journal, yet another climate change skeptic blog, an op-ed in Forbes magazine, a lecture by science writer and political activist Matt Ridley, another blog that seems to specialize in climate change skepticism, conservative political columnist Jeff Jacoby, a Wall Street Journal op-ed, still one more climate change skeptic blog, a conservative religious magazine, . . . ummm, you get the idea.

If these are your sources, you can get a distorted view of the opinions and arguments of “those who believe humans are dangerously warming the planet”! Any one or two or three of the above sources might be informative, but it doesn’t make sense to only look there. (Quick: glance again at the list of sources in the above paragraph.)

I’m not saying that Seth has any sort of duty to read the scientific literature—he’s trained as a psychologist, not as a physicist—nor does he need to read RealClimate (even if only as a supplement to the Von Mises Institute page, Wall Street journal op-eds, and so on), but it seems pretty silly for him to be so sure of himself on the science, given that his selection of politically-loaded sources.

I’m not an expert on climate science either (I’m currently involved in a research project using tree-ring data to estimate historical climate, but my expertise in this project is in the statistics, not the biology or physics), so I asked Phil what his thoughts were on the particular article that Seth linked to (by David M. W. Evans, described as “a mathematician and engineer, with six university degrees including a PhD from Stanford University in electrical engineering”).

Phil responded:

If you look at http://en.wikipedia.org/wiki/File:Satellite_Temperatures.png the temperature anomaly was higher in 1988 than in any other year until 1997…and was lower in the past couple of years than at most times in the past decade. The paper Seth blogged about looks at the change in temperatures since 1988 and says hey, there hasn’t been any increase whereas this modeler guy said the temperature would be way higher by now. If he had chosen 1985 or 1989 as the starting point instead, it would look different. Of course he’d say he chose 1988 because that’s when Hansen testified in Congress, so fair enough in a way…but this is a bit like the housing data that you blogged about, why not show the earlier data too? And the reason is the same as in the housing data: it’s because if you show the earlier data it undermines the story you are trying to tell. 1988 was an unusually hot year, by a lot.

Phil then turns to the RealClimate post (by Geert Jan van Oldenborgh and Rein Haarsma) mentioned earlier:

They compare a forecast from 1981 to data. This uses a land-ocean index rather than tropospheric temperatures…perhaps that’s an example of this guy cherry-picking too (the land-ocean temperature increase has been more rapid than the satellite-derived tropospheric temperature increase for reasons someone else might know but I don’t). Anyway this one shows a totally different story of course. Actually this guy cheats a little by starting his baseline too high I think, though that’s just based on a visual impression.

One thing that people who do these things don’t seem to understand is that the uncertainties in climate models (like low/medium/high curves) are supposed to represent the uncertainty in the trend, not the uncertainty for an individual year. It’s sort of like showing uncertainties in a regression model with one of those bow-tie curves: that doesn’t mean that you’re predicting that the individual data points will fall within the upper and lower curve, it means that the true line is probably somewhere in there. Similarly, people say “the past five years (or whatever) have been below the predicted trend line so the trend line must be too high,” apparently not realizing (or deliberately ignoring) that there is a lot of year-to-year variation, plus autocorrelation on the scale of a decade or so. It’s a true statement that global mean temperature was as high in 1988 as it was last year; it’s false to say that there’s not an upward trend in temperatures.

I think, though, that if Seth continues to search for information from his usual sources, he will remain convinced that there is no man-made global warming and that vast majority of scientists who study these problems are not interested in checking their models, have not ever thought about the ice age, etc etc.

I think this is important not only for the followers of Seth Roberts but more generally in that it illustrates the traps that people can fall into when seeking out confirmation of their beliefs. Seth is better-equipped than most people to read about scientific evidence, yet he is stuck, not only in holding a scientific view which I find implausible (after all, I might be wrong) but in not understanding that Geert Jan van Oldenborgh, Rein Haarsma, etc etc etc are doing serious science. It’s sad, and it’s scary.

P.S. Phil provides a good summary in this comment.

Good point.

P.S. As discussed in the linked thread, the great statistician R. A. Fisher was notorious for minimizing the risks of smoking. How does this connect to Fisher’s name, one might ask?

What gives a text presence is our commitment to asserting facts. We have to face the possibility that we may be wrong about them resolutely, and we do this by writing about them as though we are right.

This and an earlier remark by Basbøll are closely related in my mind to predictive model checking and to Bayesian statistics: we make strong assumptions and then engage the data and the assumptions in a dialogue: assumptions + data -> inference, and we can then compare the inference to the data which can reveal problems with our model (or problems with the data, but that’s really problems with the model too, in this case problems with the model for the data).

I like the idea that a condition for a story to be useful is that we put some belief into it. (One doesn’t put belief into a joke.) And also the converse, that thnking hard about a story and believing it can be the precondition to ultimately rejecting it because its implications don’t make sense. It’s like in chess: the way to refute a move is to consider making the move (which is as irrevocable in a chess context as believing a story is, in the context of basing a social-scientific theory on it.)

I’m also reminded of the advice from Pólya (or somebody like that) about solving math problems. If the question is, “Is statement A true?”, you can try to prove A or find a counterexample. But it’s hard to do both at the same time! Better to take a guess and go from there: if you try and try to prove it and fail, this may give insight into where to find a counterexample (in those folds of the problem that make A so hard to prove); conversely, if you can’t find a counterexample no matter how hard you look, you can try to systematize that search, thus perhaps leading to a proof that no counterexample exists.

I [Behlendorf] am replicating some previous research using OLS [he's talking about what we call "linear regression"---ed.] to regress a logged rate (to reduce skew) of Y on a number of predictors (Xs). Y is the count of a phenomena divided by the population of the unit of the analysis. The problem that I am encountering is that Y is composite count of a number of distinct phenomena [A+B+C], and these phenomena are not uniformly distributed across the sample. Most of the research in this area has conducted regressions either with Y or with individual phenomena [A or B or C] as the dependent variable. Yet it seems that if [A, B, C] are not uniformly distributed across the sample of units in the same proportion, then the use of Y would be biased, since as a count of [A+B+C] divided by the population, it would treat as equivalent units both [2+0.5+1.5] and [4+0+0].

My goal is trying to find a methodology which allows a researcher to regress Y on a number of Xs, but which accounts for the uneven variation in the distributions of the individual phenomena [A+B+C] that constitute Y. I have thought that it could be treated within a Structural Equation Model as multiple dependent variables, or through a process of joint estimation, but in essence I know the latent factor (Y) that one usually does not know when trying to measure through some sort of SEM or Rasch Model. I have also considered weighting [A,B,C] by converting them into percentages of the total count of each phenomena within the sample (i.e. (A1/sum A(1-100)) + (B1/sum B(1-100)) + (C1/sum C(1-100))), but the result lacks interpretational quality as to the overall relationship between Xs and Y.

My reply:

First off, the reason for logging is to model a multiplicative relationship using an additive model. Skewness is typically irrelevant (see the discussion of regression assumptions in chapter 3 or 4 of ARM). No big deal here, I just wanted to get that out of the way. Also, if y is a count, you might want to use an overdispersed Poisson regression as discussed in chapter 6.

My main question is, if you have a, b, and c, why not just model them separately? Is it a sample size issue, that by combining a,b,c into y, you get more stable estimates? If so, that’s ok, and you could always try weighted averages if that makes sense in your application.

Via Yalda Afshar, a 2005 paper by Hans-Hermann Dubben and Hans-Peter Beck-Bornholdt:

Publication bias is a well known phenomenon in clinical literature, in which positive results have a better chance of being published, are published earlier, and are published in journals with higher impact factors. Conclusions exclusively based on published studies, therefore, can be misleading. Selective under-reporting of research might be more widespread and more likely to have adverse consequences for patients than publication of deliberately falsified data. We investigated whether there is preferential publication of positive papers on publication bias.

They conclude, “We found no evidence of publication bias in reports on publication bias.” But of course that’s the sort of finding regarding publication bias of findings on publication bias that you’d expect would get published. What we really need is a careful meta-analysis to estimate the level of publication bias in studies of publication bias of studies of publication bias.

I see you have a blog post about turing chess . . . I’ve seen another reference to it but am unable to find a definitive source. Do you know of a source where I could find out about the history of the idea?

My reply:

You mean the run-around-the-house thing? I don’t know where it comes from. It’s a well known story, if you google Turing chess run around the house you can find lots of references but I don’t know the definitive source. I can blog and see if anything comes up!

I’ve never actually played the game. I’ll try it outdoors sometime, perhaps. When I last posted on the topic, we had a fun discussion, revealing that the rules are not as clear as one might think. It makes me wonder if anyone’s thought hard about it and come up with a good set of “official rules.”

Any thoughts?

Psychologists perform experiments on Canadian undergraduate psychology students and draws conclusions that (they believe) apply to humans in general; they publish in Science. A drug company decides to embark on additional trials that will cost tens of millions of dollars based on the results of a careful double-blind study….whose patients are all volunteers from two hospitals. A movie studio holds 9 screenings of a new movie for volunteer viewers and, based on their survey responses, decides to spend another $8 million to re-shoot the ending.  A researcher interested in the effect of ventilation on worker performance conducts a months-long study in which ventilation levels are varied and worker performance is monitored…in a single building.

In almost all fields of research, most studies are based on convenience samples, or on random samples from a larger population that is itself a convenience sample. The paragraph above gives just a few examples.  The benefits of carefully conducted randomized trials are well known, but so are the costs and impediments. Lucky people studying some natural phenomena like solar output or earthquakes can deal with complete datasets, but for most data analysts and applied statisticians the fact that your data are not a random sample from your population of interest is so commonplace that it usually goes without saying. This does not mean that it goes without thinking, of course: most researchers, and all good ones, think about the extent to which their results might or might not be applicable to a wider population and try to frame their conclusions accordingly. But most or all researchers are willing to extrapolate their results to wider populations to some degree. The movie studio reshoots their ending not because they want to please their 9 test audiences, but because they think that the response of those 9 test audiences tells them something about millions of other viewers, even though those 9 audiences were not selected according to a careful, randomized sampling scheme.

If you think everything I’ve said so far is so obvious as to be boring, so did I, but I was proven wrong. Read on.

I’m currently working with time series data on building electricity consumption. People want to be able to answer questions like “I changed something about my large commercial building on March 1; how much energy am I saving,” and one way to do that is to fit a statistical model using data from before March 1, use it to predict the energy use after March 1, and compare the predictions to what actually happened. There are other uses for these models too.

There are quite a few companies that offer energy modeling programs. Typically, a company develops its own proprietary tool. Everyone thinks their package is better than anyone else’s, or at least they say so, but there’s usually no evidence.  A few months ago a company approached us to ask if we will compare the accuracy of their tool to some standard methods and allow them to publicize the results if they want to. (I work at a government research lab and they correctly see us as a disinterested party.)  We said sure. Our chosen approach is cross-validation. We compiled electricity data from a few dozen large commercial buildings — the population of interest — blanked out big chunks of data, and gave the resulting dataset to the company. Their task is to make predictions for the missing time periods and give them to us, and we will compare their predictions to reality. We’re doing the same with some standard prediction models. The data we’ve given them are a convenience sample. There’s no practical way to get data from a random sample of buildings in the country or even from a single electric utility, in part because of privacy issues (the utilities can’t give out the data without permission of the building owners). Our data are a grab bag from different sources, and certainly not representative of the broad population of commercial buildings in many ways. Still, what else can one do? We want to find out if these guys have a model that outperforms standard models, and this is a way to find the answer.

To my surprise, our postdoc strongly disagrees. He asserts that since we don’t have a sampling plan, just a haphazard collection of building data, we can’t say anything at all. He says “sure, of course you can say which method performs better for these 52 buildings, but you cannot say a single thing about a 53rd building. Nothing.”  At first I thought he meant that we should be cautious about making firm claims, and I certainly agree with that, but that’s not it: he really thinks that it is wrong to draw any conclusions whatsoever from our results. He thinks it’s wrong (incorrect, and borderline immoral) to say that our results are even suggestive. I offered a wager: if we find that one method performs substantially better than the others on average, I will bet you dinner that it will perform better for a 53rd building. He said sure, fine for wagering over dinner, but it is scientifically indefensible to make any statement at all about which of the methods performs better in general on the basis of anything we might find using our dataset.

To me this has some parallels to a recent post about a theoretical statistician whose work is useless in practice.  I am as aware of the problems of biased datasets as anyone — I once looked into the issue of bias in indoor radon measurement datasets, and the bias turned out to be absolutely enormous — but in most fields of research if you think you can’t learn anything about the wider world unless that wider world is inside your sampling frame, you may as well quit now because you are never going to get the world into your sampling frame.

This is an interesting problem because it is sort of outside the realm of statistics, and into some sort of meta-statistical area. How can you judge whether your results can be extrapolated to the “real world,” if you can’t get a real-world sample to compare to? (And if you could get a sample to compare to, you would, and then this problem wouldn’t come up).

I would welcome thoughtful commentary on this subject.

Sixty-two percent of U.S. employees say it’s not likely they or a family member will be diagnosed with a serious illness like cancer, a survey indicates.

The Aflac WorkForces Report, a survey of nearly 1,900 benefits decision-makers and more than 6,100 U.S. workers, also indicated 55 percent said they were not very or not at all likely to be diagnosed with a chronic illness, such as heart disease or diabetes.

Here are some actual statistics:

The American Cancer Society, Cancer Facts & Figures 2012, said 1-in-3 women and 1-in-2 men will be diagnosed with cancer at some point in their lives, and the National Safety Council, Injury Facts 2011 edition, says more than 38.9 million injuries occur in a year requiring medical treatment.

The American Heart Association, Heart Disease & Stroke Statistics 2012, said 1-in-6 U.S. deaths were caused by coronary heart disease, Tillman said.

And some details on the survey:

The survey conducted in January and February by Research Now. The first 3,151 worker interviews were nationally representative, while the remaining 3,000 interviews were conducted among the Top 30 designated market areas.

Did these people really say they that neither they nor a family member will have a serious illness? Is this for real? What were they thinking? I’m used to seeing wacky survey findings, but this one is ridiculous.

You have to click the image and play with it to appreciate it. The methodology isn’t yet published – but I can see how this could be very illuminating. The dynamic clustering aspect hasn’t been researched much – one of the notable pieces is the Blei and Lafferty dynamic topic model of Science.

I did a static analysis of the US Senate back in 2005 with Wray Buntine and coauthors. Some additional visualizations and the source code are here. We did a dynamic analysis of US Supreme Court on this blog but there’s also a paper.

My knowledge on this topic is out of date, however. Who has been doing good work in this area? I’ll organize the links.

[added 4/29/12, via Edo Airoldi]: Visualizing the Evolution of Community Structures in
Dynamic Social Networks by Khairi Reda et al (2011) [PDF].

[added 4/29/12, via Allen Riddell] Joint Analysis of Time-Evolving Binary Matrices and Associated Documents by Eric Wang et al (2010) [PDF] [Video]

My (now deceased) collaborator and guru in all things inference, Sam Roweis, used to emphasize to me that we should evaluate models in the data space — not the parameter space — because models are always effectively “effective” and not really, fundamentally true. Or, in other words, models should be compared in the space of their predictions, not in the space of their parameters (the parameters didn’t really “exist” at all for Sam). In that spirit, when we estimate the effectiveness of a MCMC method or tuning — by autocorrelation time or ESJD or anything else — shouldn’t we be looking at the changes in the model predictions over time, rather than the changes in the parameters over time? That is, the autocorrelation time should be the autocorrelation time in what the model (at the walker position) predicts for the data, and the ESJD should be the expected squared jump distance in what the model predicts for the data? This might resolve the concern I expressed a few months ago to you that the ESJD is not affine-invariant, and etc. Thoughts?

Hogg continues with an example:

Imagine you have a three-planet model for some radial velocity data. In the naivest implementation, you have a three-factorial exact degeneracy from swapping planets, but the modes are very well separated in parameter space: Your autocorrelation time in the parameters is essentially infinite (because you will never switch from one permutation of the planets to another, realistically), but in the predictions the autocorrelation time is finite and fine.

My reply:

It depends on the context. Sometimes we have a redundant parameterization in which the individual parameters are not identified, but predictions are well-identified. For a simple example, suppose you have a model, y ~ N (a+b, 1), with a uniform prior distribution on (a,b). Then your data don’t tell you anything about a or b, but you can get good inference for a+b and good predictions for new data from the same model. On the other hand, if you want to make a prediction for new data z ~ N(a,1), you’re out of luck.

More generally, one problem I have with the hard-line predictivist stance—the idea that models and parameters are mere fictions whereas predictions are real—is that models and parameters can be thought of as bridges between the data of yesterday and the data of tomorrow. Consider the speed of light. It’s not just part of a prediction for some particular measurement. It’s also a universal constant. For a more humble example, consider our discussion of physiologically-based pharmacokinetics models in Section 4.3 of my article with Bois and Jiang. In a Bayesian model, good parameterization can be important, as it is typically through the parameters that we put in prior information. In many ways, the parameterization represents a key source of prior information.

Darrell Huff, author of the wildly popular (and aptly named) How to Lie With Statistics, was paid to testify before Congress in the 1950s and then again in the 1960s, with the assigned task of ridiculing any notion of a cigarette-disease link. On March 22, 1965, Huff testified at hearings on cigarette labeling and advertising, accusing the recent Surgeon General’s report of myriad failures and “fallacies.” Huff peppered his attack with with amusing asides and anecdotes, lampooning spurious correlations like that between the size of Dutch families and the number of storks nesting on rooftops–which proves not that storks bring babies but rather that people with large families tend to have larger houses (which therefore attract more storks).

This was all a surprise to me, and I suspect to other statisticians as well. For example, Huff’s activities with the cigarette companies are not mentioned on his Wikipedia page (as of 17 Apr 2012), nor are they mentioned in an article on Huff by probabilist J. Michael Steele from 2005.

“How to Lie with Smoking Statistics”

Darrell Huff is best known for How to Lie with Statistics but he wrote or cowrote several other books, including Pictures by Pete (1944), The Dog that Came True (1946), How to Take a Chance (1959), Score: The Strategy of Taking Tests (1961), Cycles in Your Life (1964), How to be the Parent of a Successful Creative Child (1968), Twenty Careers of Tomorrow (1945), and How to Lower Your Food Bills (1963).

It appears that in the late 1960s he was also working on a book called “How to Lie with Smoking Statistics,” which the publisher saw “high likelihood of proceeding into print.”

In November 1965, a letter was sent to Huff as follows:

Here‘s a letter from 1967 where Huff asks the tobacco dudes for another $1500 to keep writing.

And here‘s a letter from mid-1968 from Huff’s publisher, Macmillan:

But publication “as soon as possible” never seems to have occurred.

What happened? In one of these documents, William Kloepfer, vice president for public relations for the Tobacco Institute, wrote of the manuscript, “Frankly, this mass of verbiage needs drastic editing before it will directly address itself to the needs of our industry.”

After glancing at a couple of sections from the draft, I gotta say that William Kloepfer had a point: “mass of verbiage” is a pretty good description! Huff’s book chapter reads like a bad sitcom where the writers were too lazy to put together enough material and they just milk the same couple of jokes over and over again.

In retrospect, I think Huff really dodged a bullet on this one. If “How to Lie with Smoking Statistics” had come out, I expect it would’ve destroyed his reputation—remember, we’re talking 1969 here, that’s five years after the Surgeon General’s report—and taken a big bite out of the later sales and reputation of his 1954 bestseller.

How sincere was Huff? Did he tank his book for strategic reasons?

I wanted to call this post, How to Lie With “How to Lie With Statistics,” but to be fair I have no reason to believe that Huff was lying or intentionally deceiving in his testimony. He may well have simply been misleading himself in analogizing research on the effects of smoking to silly things like studies of storks and babies. And if he was sincere in his views, I can hardly fault him for collecting some money for his efforts.

On the other hand, this document makes me think that Huff may have seen his role as producing talking points in support of a predetermined conclusion. I guess we’ll never know if he really wanted to publish How to Lie with Smoking Statistics. Maybe he intentionally sabotaged it because he sensed it would ruin his reputation, whereas it was possible for him to keep the consulting and testimony under the radar.

Rojas: “What does your research tell us about a sample of, say, a few hundred cases?”

Statistician: “That’s not important. My result works as n–> 00.”

Rojas: “Sure, that’s a fine mathematical result, but I have to estimate the model with, like, totally finite data. I need inference, not limits. Maybe the estimate doesn’t work out so well for small n.”

Statistician: “Sure, but if you have a few million cases, it’ll work in the limit.”

Rojas: “Whoa. Have you ever collected, like, real world network data? A million cases is hard to get.”

The conversation continues in this frustrating vein. Rojas writes:

This illustrates a fundamental issue in statistics (and other sciences). One you formalize a model and work mathematically, you are tempted to focus on what is mathematically interesting instead of the underlying problem motivating the science. . . .

We have the same issue in statistics. “Statistics” can mean “the mathematics of distributions and other functions arising in statistical models.” Or it can mean the traditional problems of statistics like inference, measurement, model estimation, sampling, data collection/management, forecasting, and description. The problem for a guy like me (a social scientist with real data) is that the label “statistician” often denotes someone who is actually a mathematician who happens to be interested in distributions. . . . What I really want is a nuts and bolts person to help me solve problems.

My first reaction—actually, my main reaction—is that Rojas hangs out with the wrong sort of statistician. Following the links, I see that Rojas works at Indiana University, which features a large statistics department. I suspect he had the misfortune to encounter “a mathematician who happens to be interested in distributions” and he didn’t realize he could shop around among the many statisticians in that department who work on applied social research.

On the other hand, it’s a bad sign that Rojas reports having this conversation multiple times. I thought that statisticians nowadays know they’re supposed to be helpful on real problems. That “n -> infinity” thing seems so old-fashioned! I’d like to believe that Rojas was just having some bad luck, but maybe there’s more of this bad stuff going on than I realized. Or maybe it was just a communication problem?

It’s hard for me to imagine a statistician in 2012 telling a sociologist, “if you have a few million cases, it’ll work in the limit,” except as a joke, as an ironic comment on the limitations of some of our theory. But perhaps that just reflects the poverty of my imagination.

In my profession as a risk manager I encountered this graph:

I can’t figure out what kind of regression this is, would you be so kind to enlighten me?
The points represent (maturity,yield) of bonds.

My reply: That’s a fun problem, reverse-engineering a curve fit! My first guess is lowess, although it seems too flat and asympoty on the right side of the graph to be lowess. Maybe a Gaussian process? Looks too smooth to be a spline. I guess I’ll go with my original guess, on the theory that lowess is the most accessible smoother out there, and if someone fit something much more complicated they’d make more of a big deal about it. On the other hand, if the curve is an automatic output of some software (Excel? Stata?) then it could be just about anything.

Does anyone have any ideas?

I conducted an experiment in which subjects where asked to estimate the probability of a certain event given a number of information (like a wheater forecaster or a stockmarket trader). These probability estimates are the dependent variable of my experiment. My goal is to model the data with a (hierarchical) Bayesian regression. A linear equation with all the presented information (quantified as log odds) defines the mu of a normal likelihood. The tau as precision is another free parameter.

y[r] ~ dnorm( mu[r] , tau[ subj[r] ] )
mu[r] <- b0[ subj[r] ] + b1[ subj[r] ] * x1[r] + b2[ subj[r] ] * x2[r] + b3[ subj[r] ] * x3[r]

My problem is that I do not believe that the normal is the correct probability distribution to model probability data (… because the error is limited). However, until now nobody was able to tell me how I can correctly model probability data.

My reply: You can take the logit of the data before analyzing them. That is assuming there are no stated probabilities of 0 and 1. You could round these to 0.01 and 0.99. In any case you should graph the data and fitted model to see if there are problems.

In my [Dyson's] career as a scientist, I twice had the good fortune to be a personal friend of a famous dissident. One dissident, Sir Arthur Eddington, was an insider like Thomson and Tait. The other, Immanuel Velikovsky, was an outsider like Carter. Both of them were tragic figures, intellectually brilliant and morally courageous, with the same fatal flaw as Carter. Both of them were possessed by fantasies that people with ordinary common sense could recognize as nonsense. I made it clear to both that I did not believe their fantasies, but I admired them as human beings and as imaginative artists. I admired them most of all for their stubborn refusal to remain silent. With the whole world against them, they remained true to their beliefs. I could not pretend to agree with them, but I could give them my moral support.

Dyson first writs about Eddington. I agree with Peter Woit that “this sympathy for a great physicist who headed down a wrong path in his later years is easy to understand, but the case of Velikovsky is less so. Velikovsky was a well-known author of crackpot best-sellers starting in the 1950s . . . and a neighbor of Dyson’s in Princeton.”

Woit quotes what Dyson “wrote as a proposed blurb for Velikovsky in 1977″:

First, as a scientist, I [Dyson] disagree profoundly with many of the statements in your books. Second, as your friend, I disagree even more profoundly with those scientists who have tried to silence your voice. To me, you are no reincarnation of Copernicus or Galileo. You are a prophet in the tradition of William Blake, a man reviled and ridiculed by his contemporaries but now recognized as one of the greatest of English poets. A hundred and seventy years ago, Blake wrote: “The Enquiry in England is not whether a Man has Talents and Genius, but whether he is Passive and Polite and a Virtuous Ass and obedient to Noblemen’s Opinions in Art and Science. If he is, he is a Good Man. If not, he must be starved.” So you stand in good company. Blake, a buffoon to his enemies and an embarrassment to his friends, saw Earth and Heaven more clearly than any of them. Your poetic visions are as large as his and as deeply rooted in human experience. I am proud to be numbered among your friends.

Now back in 2012, Dyson writes:

Science is a creative interaction of observation with imagination. “Physics at the Fringe” is what happens when imagination loses touch with observation. Imagination by itself can still enlarge our vision when observation fails. The mythologies of Carter and Velikovsky fail to be science, but they are works of art and high imagining. As William Blake told us long ago, “You never know what is enough unless you know what is more than enough.”

I’m with Woit (I think) here. I don’t see the appeal of bad science. I don’t think such voices should be silenced (as Dyson puts it), but it’s probably a good thing to keep these theories off the nonfiction shelves. I see the appeal of poetry and literature and philosophy and all sorts of things that aren’t science, and I recognize that poets, philosophers, etc., can motivate themselves by all sorts of wacked-out theories. Look at Philip K. Dick. His visions are part of who he was, and I wouldn’t trade Valis for anything, but without the art the visions aren’t so exciting. My impression is that the problem with crackpot scientific theories is not that they are not beautiful but that they lead to no scientific progress or understanding. To put it another way: as a poet, Velikovsky does not have much to offer. It is only if his theories point toward scientific understanding that they have value. In contrast, Blake was an artists whose visions are appealing without any necessity for them to correspond to scientific reality.

To me, a good analogy would be with the fascinating “outsider art” done by schizophrenics, where an entire canvas is covered with tiny scribbles relating to the nature of the universe. These artworks can be just amazing and I don’t think anyone should try to silence their voices. But I don’t think it does anybody any favors to call this science.

I think my view is shared by most scientists. Dyson gets some attention here partly from his eminence and partly because of his contrary views. That’s fine—he’s done enough good work that he’s earned the right to have his ideas broadcast—but it all seems a bit odd to me. Whatever people think of William Blake’s scientific ideas now, he is admired as a poet and artist. Velikovsky is more of a historical footnote in the annals of past bestsellers, a pop-culture artifact who belongs with the Chariots of the Gods guy, the Jupiter Effect guy, the Bible Code guy, the people who made the Search for Noah’s Ark movie, etc etc. I don’t think anyone will be reading his books for the pleasure of his prose.

ESPN is looking for a statistician to join the HR department as a Research Analyst. The job will consist of analytical research and producing statistics about the people that work at ESPN. Topics of interest will include productivity, efficiency, and retention of employees, among other items. In addition to data mining and producing reports, we also field surveys and analyze results.

The position is located at the headquarters in Bristol, Connecticut, the same campus where nearly all ESPN shows are produced. ESPN is a Disney company, so discounts and free admission to Disney parks are available for employees. Flexible work arrangements are available, along with working in the New York City office part-time if desired.

The role is a relatively new function and will have a high impact very quickly on helping the business function. Statistical software, text books, and any other resource needed to get the job done will be provided.

The link for the application is below. Any interested candidates with questions can contact Michael.J.Springer@espn.com or the recruiter Amy.McManus@espn.com.

24 years ago (1988, Journal of Economics Perspectives) I [Poirier] noted cognitive dissonance among some economists who treat the agents in their theoretical framework as Bayesians, but then analyze the data (even in the same paper!) as a frequentist. Recently, I have found similar cases in cognitive science. I suspect other disciplines exhibit such behavior. Do you know of any examples in political science?

My reply:

I don’t know of any such examples in political science. Game theoretic models are popular in poli sci, but I haven’t seen much in the way of models of Bayesian decision making.

Here are two references (not in political science) that might be helpful.

1. I have argued that the utility model (popular in economics and political science as a way of providing “microfoundations” for analyses of aggregate behavior) is actually more of a bit of folk-psychology that should not be taken seriously. To me, it is silly that many economists and political scientists give this model such prominence. Utility theory can be a helpful normative model in many situations, but I don’t think it should be anything close to foundational as

2. Are you familiar with the work of Josh Tenenbaum? He is a cognitive scientist at MIT who has been working on Bayesian models for human reasoning and also Bayesian methods for fitting such models given data from psychological experiments. See here and here.

Getting back to Poirier’s original point, it could make complete sense to me to use non-Bayesian inference to learn about Bayesian agents, as long as you believe that (a) people’s behavior can be reasonably approximated by Bayesian decision rules, and (b) from a normative standpoint, non-Bayesian inference is to be preferred. It seems that many economists believe both a and b, so I don’t necessarily see any cognitive dissonance in using non-Bayesian statistical inference while modeling behavior as Bayesian.

The funny thing is, I believe not-A and not-B, so my preference would be to use Bayesian inference for non-Bayesian models of behavior.

What really makes it work, I think, is that it goes slowly enough. 2 minutes and 45 seconds is enough time for me, as a viewer, to feel like I’m living through each stage of development. If the video were sped up to go from 0 to 12 in only 30 seconds, that would be cool in its own way but would give up the sense of local stability that is characteristic of development.

We have recently had a discussion (here and here) of Karl Weick, a prominent scholar of business management who plagiarized a story and then went on to draw different lessons from the pilfered anecdote in several different publications published over many years.

Setting aside an issues of plagiarism and rulebreaking, I argue that, by hiding the source of the story and changing its form, Weick and his management-science audience are losing their ability to get anything out of it beyond empty confirmation.

A full discussion follows.

1. The lost Hungarian soldiers

Thomas Basbøll (who has the unusual (to me) job of “writing consultant” at the Copenhagen Business School) has been writing in different places about the a story that has been making the rounds over the past few decades among organizational sociologists and management consultants. The story started with a discovery that of plagiarism by the eminent scholar Karl Weick but then moved toward a more general exploration of storytelling and belief. (I learned about this example via an email from Basbøll (who had become aware of my interest in plagiarism); it turns out we also have a common interest in the bases of scientific and scholarly ideas.)

From Basbøll’s latest and most historical telling (linked from here, via Basbøll’s blog), supplemented by Wikipedia, I summarize what happened in time order (which is somewhat ahistorical in that it does not represent the order in which Basbøll, and perhaps Weick, learned about these events):

1916: Albert Szent-Györgyi, a medical student in Budapest, serves in World War 1.

1930: Working in Szeged, Hungary, Szent-Györgyi and his colleagues discover vitamin C. In the next several decades, he continues to make research contributions and becomes a prominent scientist, eventually moving to the U.S. after World War 2. He dies in 1986.

1972: Medical researcher Oscar Hechter reports the following in the proceedings of a “an international conference on cell membrane structure,” published in 1972:

Let me close by sharing with you a story told me by Albert Szent-Györgyi. A small group of Hungarian troops were camped in the Alps during the First World War. Their commander, a young lieutenant, decided to send out a small group of men on a scouting mission. Shortly after the scouting group left it began to snow, and it snowed steadily for two days. The scouting squad did not return, and the young officer, something of an intellectual and an idealist, suffered a paroxysm of guilt over having sent his men to their death. In his torment he questioned not only his decision to send out the scouting mission, but also the war itself and his own role in it. He was a man tormented.

Suddenly, unexpectedly, on the third day the long-overdue scouting squad returned. There was great joy, great relief in the camp, and the young commander questioned his men eagerly. “Where were you?” he asked. “How did you survive, how did you find your way back?” The sergeant who had led the scouts replied, “We were lost in the snow and we had given up hope, had resigned ourselves to die. Then one of the men found a map in his pocket. With its help we knew we could find our way back. We made camp, waited for the snow to stop, and then as soon as we could travel we returned here.” The young commander asked to see this wonderful map. It was a map not of the Alps but of the Pyrenees!

The moral of the story, as given by Hechter and by Bernard Pullman at another symposium a year later, is that the map gave the soldiers the confidence to make good decisions. Basbøll writes:

The map of the Pyrenees is like a controversial paper or a tentative model study in science. Whether or not it turns out to be an accurate representation of the “territory” is less important than the stimulus it may provide to further research, argue both Hechter and Pullman.

1977: Immunologist Miroslav Holub publishes a poem (of the prosy, non-rhyming sort) telling the lost-soldiers story (again, crediting Szent-Györgyi) in the Times Literary Supplement, translated from the Czech. Holub may have actually attended the meeting reported on by Hechter.

1982: Robert Swieringa and Karl Weick publish an article including a nearly word-for-word transcription of Holub’s poem, but not using quotation marks or acknowledging Holub at all, presenting the story as “an incident that happened,” and placing the event (implausibly) in Switzerland.

“Sometime in the mid-1980s”: Weick tells the story to Bob Engel, a “top Wall Street executive” who “was about to take leadership of a new strategic planning group at Morgan Guaranty.” The moral of the story: “When you are lost, any old map will do.”

1995: Weick writes:

What is interesting about Engel’s twist to the story is that he has described a situation that most leaders face. Followers are often lost and even the leader is not sure where to go. All the leaders know is that the plan or the map they have in front of them are not sufficient to get them out. What the leader has to do, when faced with this situation, is instill some confidence in people, get them moving some general direction, and be sure they look closely at cues created by their actions so that they learn where they were and get some better idea of where they are and where they want to be.

2005: Barbara Czarniawska reports on a 1998 talk:

“If any old map will do to help you find your way out of the Alps,” Weick had said, “then surely any old story will do to help you find your way out of puzzles in the human condition.”

As Basbøll notes, the moral of the story keeps changing! In the earliest known tellings (by Hechter and Pullman), the only clear role of the map was to calm the soldiers os they could find their way back to camp on their own. By 1998 (or 2005), the map has an actual instrumental use. What is creepy (to me) is that the story has changed from a story about people using an external device (the map) as a way to calm themselves and make a reasoned course of action, to a parable about savvy managers (“leaders”) who can manipulate their underlings, to the conclusion that “any old story will do,” a claim that makes the original events (or non-events) irrelevant and would seem to me to encourage a form of scholarship that does nothing but confirm people’s preconceptions.

This is where we come to the point about anecdotes that I discussed in the three paragraphs at the start of the post.

2006: Basbøll and Henrik Graham uncover the Weick plagiarism. Various scholars engage in discussion of the case over the next several years.

Full circle

Going beyond questions of plagiarism and scholarly ethics, the Szent-Györgyi/Holub/Weick story is relevant for our discussion because it sheds (anecdotal) light on the relevance of anecdotal reasoning and the nature of evidence. Weick’s misrepresentation of Holub’s texts suggests how such flexible uses of stories can lend themselves to flexible conclusions. Reliance on anecdotes is already iffy because of selection problems, but when the stories can be altered, the concepts of confirmation and falsification are turned on their head. (Weick and his ilk may very well argue that his stories are not evidence but merely entry points for involving the audience to think about his deeper principles, but then this just pushes the question back one step: What, then, is the evidence for those principles, and what is the role of anecdotes in the translation of principles to the outside world?)

2. Which leaf on which tree are you talking about?

I was giving a talk the other day to a group of statistics graduate students, on the subject of connections between teaching and research, and I mentioned one of my favorite sayings, “God is in every leaf of every tree.” What this means is that if you study any problem carefully and seriously enough, you will come to interesting statistical research problems.

If you’ve been reading this far, the above paragraph should sound familiar. The meta-statistical principle that “God is in every leaf of every tree” is very close to Weick’s “any old story will do to help you find your way out of puzzles in the human condition.”

Now, though, let me explore the differences as well as the similarities between the two quotes. In the leaf/tree scenario, it is important—crucial—that the statistician or scientist look carefully at the particular leaf in question. By carefully trying to resolve the contradictions involved in a single sampling problem, or a single causal inference, or a single dataset, or (in Holub’s field of immunology), a single medical patient, we might take ourselves along the path to a general solution—or at least a recognition and better understanding of a problem with the current scientific formulation. But you have to study the actual leaf! Studying an abstract leaf, or a secondhand story about a leaf, won’t do the trick. Constructing the leaf from a research hypothesis, or altering the leaf to fit the hypotheses, won’t give you as much (although you can still learn something, for example if you take care to note how many changes you needed to make to keep your audience happy).

Similarly, the history of the various misrepresentations and changes to Szent-Györgyi’s story, as related and interpreted by Basbøll, give some sense of the various uses that the story has been put. Each telling comes with its own “moral” or message.

3. The methodological attribution problem

One of my meta-principle of statistics, which in my published discussion of an article of Brad Efron I call the “methodological attribution problem”:

The many useful contributions of a good statistical consultant, or collaborator, will often be attributed to the statistician’s methods or philosophy rather than to the artful efforts of the statistician himself or herself. Don Rubin has told me that scientists are fundamentally Bayesian (even if they do not realize it), in that they interpret uncertainty intervals Bayesianly. Brad Efron has talked vividly about how his scientific collaborators find permutation tests and p-values to be the most convincing form of evidence. Judea Pearl assures me that graphical models describe how people really think about causality. And so on. I am sure that all these accomplished researchers, and many more, are describing their experiences accurately. Rubin wielding a posterior distribution is a powerful thing, as is Efron with a permutation test or Pearl with a graphical model, and I believe that (a) all three can be helping people solve real scientific problems, and (b) it is natural for their collaborators to attribute some of these researchers’ creativity to their methods.
The result is that each of us tends to come away from a collaboration or consulting experience with the warm feeling that our methods really work, and that they represent how scientists really think. In stating this, I am not trying to espouse some sort of empty pluralism—the claim that, for example, we would be doing just as well if we were all using fuzzy sets, or correspondence analysis, or some other obscure statistical method. There is certainly a reason that methodological advances are made, and this reason is typically that existing methods have their failings. Nonetheless, I think we all have to be careful about attributing too much from our collaborators’ and clients’ satisfaction with our methods.

As suggested in my discussion, I came to this meta-principle through my indirect experiences of hearing various researchers talk about the efficacy of their statistical methods. Theoreticians and methodologists often have extreme confidence in their approaches, and applied researchers often have what seems to me to be oddly strong opinions about statistical methods.

Playing tennis without a map

As the saying goes, research is when you don’t know what you’re doing. We don’t have maps; much of research can’t be automated. In the spirit of Weick’s writings, let me say that statistical models and scientific theories can play useful roles even when far off from the truth.

I discussed this a few days ago in the context of the two-edged nature of belief. On the one hand, belief is powerful. By conditioning on assumptions, we can rule out alternatives and move quickly and surely. But belief is risky, especially since all of our beliefs, if stated precisely enough, our false. The resolution is that we can use the strength and power of beliefs to better study their limitations.

From a statistical (and philosophy-of-science) perspective, strong assuptions play two roles: First, with strong assumps we can (often) make strong and precise inferences. The likelihood function is a powerful thing. Second, strong assumptions are strongly checkable and falsifiable. We take our models seriously, work with them as if we believe them unquestioningly, then use the leverage from this simulation of belief to check model fit and explore discrepancies between inferences and data.

4. Building up a store of anecdotes

One difference between academic statistics and the academic study of management (as embodied by Karl Weick) is that applied statisticians such as myself live in the world of “tabletop experiments” (as they say in physics) whereas Weick (and, I assume, other scholars of his field) do “big science.” What I mean is that I work on lots of little problems and a few big ones, whereas Weick has a couple of main areas of focus. I have built up a huge store of personal anecdotes about statistics, whereas Weick must largely rely on the anecdotes of others. I’m David Sedaris, he’s Jay Leno (not the best analogy here; what would be a better example of a storyteller who uses recycled stories but tells them well?). Weick represents storytelling as an important part of his style, which puts him at a particular dependence on interpreting events that did not happen to him.

I’m not saying that I’m better than Weick in this dimension; we’re just different. A historian of the Middle Ages, for example, would have no directly relevant personal experience at all. That’s just the way it is!

Getting back to the comparison: I really do use anecdotes as evidence, as well as to illustrate existing principles. My hundreds of statistical experiences have been important in the development of my ideas. This shows up even in my published papers and in how I evaluate the methodological contributions of others: I typically trust an idea to the extent that it helps solve what seems to me to be a real applied problem.

Weick is in a different situation: he uses stories to grab his audience and perhaps to inspire him to come up with new theories or modification of existing theories. For Weick, anecdotes play the role of the map in his some of his interpretations of Szent-Györgyi’s story: It’s ok if you get the details and sourcing wrong, all that matters that it motivates you to move forward. As noted, that approach would not work in statistical research (or, I suspect, in medical research) because it would deprive us of the opportunity to learn from anecdotes’ specific features. Maybe it’s ok in Weick’s area of management research, as long as the stories and their alterations are accurately sourced.

5. Putting it all together

I have discussed several themes relating to the use of anecdotes in scientific learning. It’s easy to laugh at anecdotes, but I recognize that they are crucial in my meta-statistical understanding. That is, much of my judgment about what methods to use, comes from my own personal experiences. I suspect that many people without my breadth of statistical experience are still relying on it indirectly through my books and articles.

In other fields, though, anecdotes play a different role, and their truth or falsity, even their sources, do not seem to be so important. But I am make this judgment based on anecdotal evidence. All of this is also related, I believe, to out recent discussions of the problems of scientific publications, statistical significance, and so forth.

In particular, all of this is related to model checking. Anecdotes with accurate sourcing can reveal problems in a model, whereas when you start altering an anecdote and hiding its source, you lose a key opportunity for learning. This relates to the much-discussed selection problems in quantitative research.

6. The mysterious move to Switzerland

One of the few things that Weick adds along with his plagiarism of the Holub poem is to place the action, identified only as Hungarian soldiers in the Alps, in Switzerland. What were Hungarian troops doing in Switzerland, one might ask? My guess, following Nick Cox’s comment on our earlier discussion, is simple ignorance: Weick is American, and when we hear about the Alps, we automatically think “the Swiss Alps.” When we think about World War 1, we think about western Europe. Perhaps Weick in 1982 did not realize that not all the Alps are in Switzerland, nor was he primed to reflect upon the location of the theater of the war involving Austria-Hungary and Italy. This small error is irrelevant to the use of the story as a management parable—Weick could just as well have used a clearly fictional example such as that of Winnie the Pooh or the Sneeches or Yoruga la Tortuga (I imagine that the sort of managers who would hire a sociologist in the first place would love the morals of those stories!)—but it demonstrates the risks of copying a story without attribution. The deadly combination of word-for-word quotation and gratuitous error (as in the notorious case where Ed Wegman copied 2^n and it came out as 2n) reveals an ignorance of the underlying material, leading outsiders such as myself to question the scholar’s competence as well as his integrity.

I agree. The graphs are corporate standard, neither pretty or innovative enough to qualify as infographics, not informational enough to be good statistical data displays.

Some examples include the above exploding pie chart, which, as Kaiser notes, is not merely ugly and ridiculously difficult to read (given that it is conveying only nine data points) but also invites suspicion of its numbers, and pages and pages of graphs that could be better compressed into a compact displays (see pages 25-65 of the report). Yes, this is all better than tables of numbers, but I don’t see that much thought went into displaying patterns of information or telling a story. It’s more graph-as-data-dump.

To be fair, the report does have some a clean scatterplot (on page 65). But, overall, the graphs are not well-integrated with the messages in the text.

I feel a little bit bad about this, because I’m involved with the Earth Institute. I should be their graphics adviser! I’m actually surprised they didn’t ask me for advice on this. I gave a talk to the Earth Institute postdocs a year or so ago, so they should know I like graphs!

P.S. Nathan Yau reproduces the report’s Figure 11, which is just as bad as the exploding pie chart shown above.