Have you seen this? The debate is interesting in general, but I think you would be especially interested in the empirical claims about issue polls and their relationship to electoral outcomes.

I [Dorman] have a concern that none of them address, the role of low propensity/engagement voters. My understanding is that in 2024 the Trump campaign acquired some secret sauce that enabled it to score *very* big among the low-pro’s. Whether that was engineered in some way or just pure luck (or some combination), we don’t know yet, but it played a crucial role. What interests me is that working class designation, generally by education, likely correlates tightly with being low engagement, to the extent that they would be almost impossible to disentangle in observational samples. Yet we hear a lot of pontificating based on the belief that the D’s lost a large chunk of the working class and have to do A, B or C to get it back. Maybe. But perhaps it’s about the distinctive characteristics of the low-pro’s, how they form their fluid political beliefs and what motivates them to get out and vote. And lurking in the background is the possibility that, already by mid-2024, the tech titans working for Trump had obtained backdoor access to social media accounts, allowing them to test content in low-pro samples in real time. If I had that capacity and I were them, that’s what I would have done.

The above-linked forum reminds me of this discussion of mine, “The WAR war and the electoral benefits of running more moderate candidates for political office.” The forum has a zillion discussants. I’m surprised they didn’t ask me for a comment!

Peter adds:

Normally, when we talk about measurement and construct error, we have a pretty clear idea in mind of what we want to be measuring, and then we compare our actual proxies to them. But with this debate about issue polling, it’s nearly the other way around. We *don’t* have a clear idea of what a political opinion or belief is, where it comes from or how it interacts with behavior, but we go out and get respondents to check a Likert box about an issue we’ve defined for them.

My hunch is that social media data, which we’re not supposed to have, may come the closest to getting observations on beliefs — their formation, their trajectories, their interactions with other dimensions of social and political life — that we’re likely to get. If you control the platform you can even conduct experiments. That’s why I suspect that political operations that include Palantir et al. are probably at a huge advantage.

I don’t know how much that they can use that is more effective than what can be found in the voter file. But maybe.

The other thing is that the linked forum focuses on strategies for the Democrats. But the same ideas apply to the Republican party. Most legislative districts are so close that there’s essentially no practical electoral benefit from moderation. But there is the argument that if you run a lot of extremists all over the country, that this will make things harder for your party’s moderates in the few swing states and swing districts out there.

The field of evolutionary psychology has often been the target of allegations that its hypotheses are unfalsifiable. . . .

We . . . present evidence of several evolutionary psychological hypotheses that have generated specific, falsifiable predictions; undergone multiple empirical tests; and been refuted. Specifically, we discuss the evidentiary status of (a) the ovulatory shift in mate preferences (dual-mating) hypothesis, (b) the mate deprivation hypothesis of rape, and (c) the kin altruism hypothesis for the evolution of male homosexuality. We contrast these with the wide range of evolutionary psychology hypotheses whose specific predictions have been robustly supported by empirical data.

. . . many academics continue to perceive evolutionary psychology as unfalsifiable, despite evidence to the contrary, a misconception that has also permeated mainstream culture. . . . By illustrating that hypotheses generated by evolutionary psychology can be directly tested with appropriate scientific rigor, we dispel these pervasive misconceptions and highlight the field’s heuristic potential for generating valuable insights into human behavior.

It’s interesting to see this article. I have to admit I have a generally negative view of evolutionary psychology, a view that probably isn’t so fair. The problem is that I’ve been exposed to a lot of bad evolutionary psychology . . . David Weakliem and I wrote a whole article about it.

Also some blog posts. Let’s search on “evolutionary psychology” and see what comes up:

2025:  I have a horrible feeling sometimes that heavily promoted crap research on space aliens, cold showers, mind-body healing, schoolyard evolutionary psychology, extra-sensory perception, magic golf balls, air rage, himmicanes, subliminal smiley faces, etc etc etc, has softened the ground so that the seeds of more evil trees could then be planted and take root.

2023:  Hey—here’s some ridiculous evolutionary psychology for you, along with some really bad data analysis.

2022:  Evolutionary psychology in economics: Mo papers, mo problems

And some things I’ve written:

2025:  I don’t mean this in a sinister way, necessarily, the idea that crafty researchers are using their networking powers to insert bad work into the literature—even though, yes, I think this happens all the time—but rather in more general terms, that it takes a lot of active effort to keep these bad ideas in embodied cognition, evolutionary psychology, subliminal effects, etc., continuing in the scientific literature. In the same way that the Loch Ness monster, UFOs as space aliens, and various political conspiracy theories require constant refreshing to stay in the news. Yes, people hold all sorts of silly beliefs—a third of Americans say they believe in ghosts!—but passive belief is not enough. Well-connected people need to be actively promoting them, or they drift to the back of the collective consciousness.

2025: The [erroneous] belief that if a general phenomenon (e.g., “nudge,” or “evolutionary psychology,” or “social priming”) is real, that it will show up in the same direction no matter how it is studied (for nudging this could be investment decisions or food choice or risky behaviors; for evolutionary psychology this could be the sex of babies or religious behavior or voting behavior; for social priming this could be walking speed or hormone levels or political attitudes; etc.).

2025:  I also don’t see any substantive theory behind the claims, other than that it seems very intuitive to people that (1) the sex of a baby should be predictable and (2) sex ratios should vary a lot.

I think the basis for that first intuition is gender essentialism: men and women are so clearly different that it just stands to reason that they should somehow be externally distinguishable (in the pre-ultrasound era) even in the womb, and that the sex of the baby should be influenced by various ways as might be explained by gender essentialism.

I think the basis for the second intuition is what Tversky and Kahneman called “the law of small numbers”: everybody knows someone with several siblings, all of the same sex. People don’t have a sense of the huge sample size that would be needed to learn anything useful from such data, even setting aside selection and recall issues. Also, gender essentialism: it just stands to reason that parents of boys should be more masculine, in some way, than parents of girls. This bit of intuition can be backed up by endless speculation of the sort that is called evolutionary psychology and which plays well in Freakonomicsland.

2024:  In some settings, I fear that “replication” is serving a shiny object to distract people from the central issues of measurement, and I think that’s what’s going on here. The authors of this study were working with some vague ideas of evolutionary psychology, and they seem to be working under the assumption that, if you’re interested in theory X, that the way to science is to gather some data that have some indirect connection to X and then compute some statistical analysis in order to make an up-or-down decision (“statistically significant / not significant” or “replicated / not replicated”).

Again, that’s not enuf! Science isn’t just about theory, data, analysis, and conclusions. It’s also about measurement. It’s quantitative. And some measurements and designs are just too noisy to be useful.

2012:  Schoolyard Evolutionary Biology: The idea that, because of evolution, all people are equivalent to all other people, except that all boys are different from all girls.

OK, so that’s some attitude.

That said, I don’t recall ever characterizing the hypotheses coming from evolutionary psychology as unfalsifiable.

Let me put it this way: I think that evolutionary psychology can produce falsifiable predictions (there I agree with Costello et al.); I also think it is often employed in an unfalsifiable way, either by motivating flexible stories that can explain any possible data pattern, or though flexible data analysis (“forking paths”) that can fit any theory.

Stepping back, I don’t think evolutionary psychology, as a whole, is falsifiable. But that’s ok. I don’t think that any science is falsifiable as a whole. I don’t think chemistry or physics are falsifiable either. Evolutionary psychology, like chemistry and physics, is a framework under which falsifiable theories can be developed. Which is fine; indeed, Costello et al. carefully write, “Evolutionary psychology hypotheses are testable and falsifiable,” not that evolutionary psychology as a whole is testable and falsifiable. I would just amend their statement to, “Some evolutionary psychology hypotheses are testable and falsifiable.”

But then you get to the specific cases in the paper, the several evolutionary psychological hypotheses that they say were “refuted,” and . . . I don’t know!

What they describe as “refutations” look to me more like unsuccessful replications. Or, more specifically, results that are not statistically significant. But a not-statistically-significant finding is, at most, a refutation of the hypothesis that the average effect is large (relative to the standard error of estimation); it does not refute the hypothesis that there is a nonzero effect, or that there is an effect in a specified direction.

This is not to say that data can’t inform theorizing in this area. I just think that the framework of “falsification” isn’t so useful here, as it doesn’t address three key factors in the experimental human sciences:

1. The uncertainty involved in learning from statistical evidence in the presence of variation or noisy measurements;

2. The often shaky connection between underlying scientific theories and the particular statistical models being tested;

3. Variation in effects, which can be positive in some settings and negative in others, making it difficult to pin down exactly what is the hypothesis being tested.

For better or worse, I think we need to give up on strict falsificationism and move to a more Lakatosian perspective in which theories are informed by data. And all this presupposes that we’re doing serious science and not screwing around with experiments that are essentially pure noise.

Again, these are issues with the human sciences in general, nothing special about evolutionary psychology here.

A core principle in medicine is non-maleficence, often equated with “do no harm.” A formal definition of harm based on counterfactual reasoning has been proposed and popularized. This notion of harm has been promoted in simple settings with binary treatments and outcomes. Here, we highlight a problem with this definition in settings involving multiple treatment options. Illustrated by an example with three tuberculosis treatments (say, A, B, and C), we demonstrate that the counterfactual definition of harm can produce intransitive results: B is less harmful than A, C is less harmful than B, yet C is more harmful than A when compared pairwise. . . . In contrast, an interventionist definition of harm based on expected utility forgoes counterfactual comparisons and ensures transitive treatment rankings.

I agree, and this is related to what we wrote in our Russian roulette paper. I’ve always been uncomfortable with utility functions that depend on outcomes that have never occurred. I recognize that counterfactual utilities can make sense psychologically (this includes concepts such as uncertainty aversion and regret), but not normatively, not for what I’ve referred to as institutional decision analysis. This new paper by Sawant and Stensrud goes beyond our Russian roulette paper by introducing a new example and considering continuous outcomes.

Also related is this paper by Stensrud with Aaron Sarvet, “Rejoinder to ‘”Harm” in personalized medicine–an alternative perspective.'”

Harvey Olden writes:

I’ve developed an election prediction model based on a transformer architecture; I thought you might find it interesting. The model will update its information automatically in the run-up to the midterm elections; you can also click on a state/district to see what information the model is leaning on to make its predictions. You can find an explainer for exactly how it works in the website.

I can’t follow the explanation myself, but that’s on me, that I’ve never actually learned exactly what a transformer is. This is not a boast—I’d be a better statistician if I understood such things—just my honest admission of imperfection.

So, given that, I can’t say much about the above forecast, but I thought it might interest some of you, at least from a methodological perspective. There must be some aspects of forecast uncertainty that are well captured by this model and also some weaknesses, corners of probability space where the model architecture induces artifacts (by analogy to the edge-case absurdities we found in a different sort of forecasting models a few years ago and which made their way into Section 8.1 of our Bayesian Workflow book, under the heading, “Exploring inferences to reveal problems with a model”). I haven’t been following the 2026 races closely enough to know what problems to look for, and I don’t see a Github site or anything else with the model’s data, code, or predictive simulations. But if that’s out there somewhere, then youall can take a look.

I was wondering whether it might be useful to alternatively frame the problem in terms of stochastic versus deterministic interventions as opposed to potential outcomes. I have attached a short note on your paper and would appreciate your thoughts.

Jonas replied:

1. If I [Jonas] understand correctly, the interventional framework is meant to avoid counterfactuals altogether (e.g. Didelez, et al., 2012), right? In this case, one might be able to avoid the problems with “counterfactual decision problems” altogether by rephrasing them as classical decision problems under interventions?
Maybe this is exactly what you have in mind. I think this would be a very interesting conceptual contribution.

2. What I like about the stochastic potential outcomes perspective is that it allows us to model which uncertainty/variation we want to incorporate in the definition of the treatment (bullet vs. pistol). By defining the utility as a functional over the distribution of the stochastic potential outcome, we can integrate out the uncertainty that remains after treatment assignment. With this additional model choice, we can choose which uncertainty is irrelevant (and hence integrated out prior) to our decision problem.

3. I’m not entirely sure I understand how this would be accomplished by switching from deterministic to stochastic interventions. I’m sure this is an oversight on my part, and I would be excited to see this perspective be fleshed out.
My immediate question would be the following: potential outcomes are defined on the level of the individual, so it makes sense to distinguish between averaging over the treatment vs. averaging over an individual’s outcome. Accordingly, we can define the utility by only averaging over an individual’s outcome, fixing a treatment, and then consider the expected utility by averaging over treatment assignment.
In the interventional framework, I’m not sure how one would distinguish between the outcome Y of an individual being deterministic vs. stochastic for a given intervention? Because of this, I’m not clear how one would first integrate out individual-level uncertainty to define the utility and then calculate the expected utility based on treatment assignment in the interventional framework. (i.e. can what you call U(R6) be understood as an average over an individual’s outcome given a treatment?) I’m sure this is my oversight. I would love to hear what you think.

4. Following up on point 3: Does the Russian roulette paradox appear/disappear when switching from a deterministic vs. stochastic interventions perspective? Essentially, I’m wondering if the choice of how to model the intervention (deterministic vs. stochastic) allows to include/integrate out different levels of uncertainty in the decision problem.

Ang responded:

1. I think so. Didelez et al. 2012 was one of the earliest works in the stochastic intervention literature. Terminology and notation evolved over time in the literature, but their “regime” is the same as what I call “intervention”, and their distinction between atomic and random interventions is the same as my distinction between deterministic and stochastic interventions.

2. I agree that being able to integrate out sort of the “inner” uncertainty is important. If I understand your point correctly, my sense is simply that we already have an existing framework that can do exactly that: we just need to rephrase your point. In the stochastic intervention framework, intervention is pistol and treatment is bullet. So we can integrate over treatments and get the utility of an intervention for each individual.

3. I think the content of the stochastic intervention and stochastic potential outcomes frameworks is exactly the same, just different labels. In my mind, Y_{i, R6} for individual i is a random variable because of the randomness in the intervention R6. So we can call E[Y_{i, R6}] an average over the randomness of the potential outcomes or the randomness of the intervention. The difference is in naming, not substance. Does that make sense?

4. I think I’m not inventing a new way to solve to paradox, but just pointing out the connection between your solution and the literature on stochastic interventions.

My quick thought on all of this is that you can have stochasticism in the intervention or in the outcome conditional on the intervention. One way to see this is in the Russian roulette example: conditional on the chamber of the bullet, you can consider the intervention to be stochastic, but unconditional on the chamber, you can consider the outcome to be stochastic.

Or here’s an example with both sources of variation (or, you could say, “stochastic” aspects): Consider a treatment that is some sort of naturally growing herb that is gathered in the wild, so not a controlled compound. The intervention is stochastic because you might happen to get a sample of the herb that has a large amount of the active ingredient or you might get a dud that has very little. But the outcome will also be stochastic because the active ingredient will not always work. Actually, we could add a third level of variation by saying that (a) the active ingredient could work for some people and not others, and (b) even for the people for whom the ingredient works, it won’t always be effective.

For another example, consider physical therapy as a treatment for sciatica:

(i) To start with, there are lots of way that physical therapy is done; different physical therapists do different things. So the intervention of “physical therapy” is stochastic, if you don’t define ahead of time what will be done and who will be doing it.

(ii) For any given physical therapy intervention, there will be some people for whom it has no chance of working, because of the condition they have. This could be considered as stochastic, but I’d prefer to think of it as deterministic, conditional on a latent variable which contains details of the patient’s underlying medical condition.

(iii) If a patient is given a therapy that can work, the actual effectiveness will be stochastic. You could go full Rubin and consider the outcome as deterministic, conditional on a latent variable which contains the potential outcomes, but I think there can be a benefit to using a stochastic model, recognizing that there’s randomness in the course of the treatment.

For an even cleaner example, consider a coaching program to improve people’s basketball skills, where the outcome is a test where they take 100 shots and you count how many go in:

(i) Some coaches are better than others. So it makes sense to think of the intervention as stochastic.

(ii) Some people will be more amenable to coaching than others. For example, some people will be motivated by the coaching to practice; others won’t. I prefer to model this with a latent variable representing differences between people, but even then there can be a stochastic element: for example, maybe I’m the sort of person who would be motivated to practice, but I just happen to be very busy during the month of the coaching. You could code my state of busyness as a latent variable, but there could be all sorts of unmeasured or even unmeasurable factors affecting my receptiveness during that month, and at some point it makes more sense to bite the bullet and call it stochastic, in the same way that we consider a die roll to be stochastic even though in theory we could predict it pretty well given enough information on initial conditions.

(iii) Any particular coaching will work better for some people than others. There are different ways to have problems with your basketball shot, and different people will be able to improve in different ways. This could be considered as an interaction between the flavor of treatment and the player in the experiment, and it could be modeled with sufficient pre-treatment information on the players, but in practice it could make sense to model much or even all of this variation as stochastic.

(iiii) Finally, how many shots go in, conditional on the average probability of success? That’s stochastic. Again, you could follow Rubin (1974) and consider the outcome of the test to be deterministic, under the philosophy that “the great book of life has already been written; we just haven’t read it yet”; but to me it makes much more sense to consider the success probabilities or expected outcomes as fixed but unknown, with the actual outcomes as stochastic.

I’ve written before about the border between conditioning and randomness–it comes up in non-causal settings too. See here: https://sites.stat.columbia.edu/gelman/research/published/augie4.pdf and here: https://statmodeling.stat.columbia.edu/2022/12/23/a-probability-isnt-just-a-number-its-part-of-a-network-of-conditional-statements/

To which Ang responded:

I appreciate you laying out more generally different possible sources of randomness. On a philosophical level, they all make sense. From a practical perspective, though, I wonder if it’s useful to fully parse everything out. Stochastic interventions have proved useful in a lot of epidemiological and sociological applications. And I think the Russian roulette can also be framed as a stochastic intervention. In all those cases, it seems sufficient to explicitly consider only two sources of variation: the random variation in the intervention (e.g., bullet or no bullet), and the variation in potential outcomes across individuals. Please feel free to blog our discussion!

On a side note, I recently read VanderWeele and Hernan (2013), which framed a variant of stochastic intervention as a case of treatments with multiple versions. The stochastic intervention considered there also directly comes from a real empirical application.

I actually think we’re in agreement here. I too think it can be useful to allow stochastic potential outcomes. Indeed, I think the idea of the “stable unit treatment value assumption” (SUTVA) can be misleading, in that SUTVA requires that outcomes be modeled deterministically. In my above note, I said that any stochastic potential outcome can be expressed in reduced form as a deterministic potential outcome–but for modeling purposes, the stochastic model often makes more sense. So I guess there was something unclear in my paragraphs just above, in that I wasn’t doing a good job in conveying this perspective.

Here’s the story, and below is the first version of the paper, exactly as I wrote it.

First, here are some recent things to read regarding monitoring convergence from iterative simulations:

• [2025] For how many iterations should we run Markov chain Monte Carlo? In {\em Handbook of Markov Chain Monte Carlo}, second edition. (Charles C. Margossian and Andrew Gelman)
• [2024] Nested R-hat: Assessing the convergence of Markov chain Monte Carlo when running many short chains. {\em Bayesian Analysis}. (Charles C. Margossian, Matthew D. Hoffman, Pavel Sountsov, Lionel Riou-Durand, Aki Vehtari, and Andrew Gelman)
• [2021] Rank-normalization, folding, and localization: An improved R-hat for assessing convergence of MCMC. {\em Bayesian Analysis} {\bf 16}, 667–718. (Aki Vehtari, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul-Christian Burkner)

And now, here’s the original R-hat paper from 1990. I (appropriately) included Rubin as a coauthor, as the idea came about after many long phone conversations, and when I brought up this idea, Rubin picked up on it right away, as it was close to methods he had developed for inference from multiple imputations.

To all:

Here’s the abstract of the work I’ve done this summer. It’s stored in the file,
/fs5/gelman/abstract.bell, and copies of the Figures 1-3 are on Trevor’s desk.
Any comments are of course appreciated; I’m at gelman@stat.berkeley.edu.

On the Routine Use of Markov Chains for Simulation

Andrew Gelman and Donald Rubin, 6 August 1990

corrected version: 8 August 1990

1. Simulation

In probability and statistics we can often specify multivariate distributions
many of whose properties we do not fully understand–perhaps, as in the
Ising model of statistical physics, we can write the joint density function, up
to a multiplicative constant that cannot be expressed in closed form.
For an example in statistics, consider the Normal random
effects model in the analysis of variance, which can be
easily placed in a Bayesian framework with a conjugate prior distribution.
All the conditional densities of the resulting posterior distribution
are simple, but marginal densities can only be written in integral form and
can only be calculated approximately. (For details, see Kinderman and Snell
(1980) or Pickard (1987) for the Ising model, and Lindley and Smith (1972)
for the Bayesian random effects model.)

In such cases, we may not even be able to compute marginal moments of the
difficult distribution, let alone more complicated and interesting summaries
that would help us understand a probability model or posterior inference.
When direct methods such as analytic or numerical integration of “nuisance”
parameters are not computationally feasible, we might try Monte Carlo simulation;
in the simplest form, we draw a finite set of independent random samples from our
distribution, and then calculate desired distributional summaries as functions of
the sampled points. The Monte Carlo method is quite general and powerful; it is
easy to calculate arbitrary quantities of interest
such as the expected long-distance correlation in the Ising model or a posterior
95% confidence region for the largest block effect in a random effects model.
Any aspect of the distribution can be approximated to any desired accuracy if
the number of independently sampled points is large enough.
Simulation also has the advantage of flexiblility: once a sample is drawn, it
can be used to learn about any number of different distributional summaries.

2. Markov chain methods

Drawing independent random samples is a wonderful tool that is unfortunately not
available for every distribution; in particular, the Ising model and random
effects posterior distributions mentioned above do not permit direct
simulation. Fortunately, a form of indirect simulation method exists for
any multivariate distribution if we can calculate its joint density
(up to a multiplicative constant) or if we can sample from all its univariate
conditional densities. The first of these methods was introduced by
Metropolis et al. (1953) in the Journal of Chemical Physics. Our work focuses
on a similar and slightly simpler method called the Gibbs sampler by Geman and
Geman (1984) in an article for the IEEE.

Let F(x) be our distribution; the Metropolis algorithm takes a starting (vector)
point x0 and constructs a series x1, x2, . . ., that is a sample from an
ergodic Markov chain whose stationary distribution is F(x). Computer
simulation of the series requires calculation of the density f(x) (up to a
constant). These samples xj are not independent; however, the stationary
distribution of the Markov chain is
correct, so if we take a long enough series, the set of values {x1, . . ., xn}
takes the place of the distribution just as an
independent random sample does (although of course an independent sample
carries more information than a Markov chain sample of the
same length).

The Gibbs sampler is a similar algorithm, which produces a Markov chain that
converges to the desired distribution, this time requiring draws from all the
univariate conditional densities at each iteration.

3. Have we converged yet?

Markov chain simulation methods are attractive for many problems because they
enable us to flexibly summarize intractable multivariate distributions by making
full use of the mathematical structure we do know, using a tool we think we
understand–Monte Carlo simulation. Unfortunately, using a sample of a Markov
chain to estimate a distribution raises an immediate question: how long a series
is needed? After one or two steps, we are almost certainly still too close to
the starting point to hope for unbiased summaries. Asymptotically, the chain is
stationary, and all is OK (with some loss of efficiency compared to independent
samples, as mentioned above).

To obtain a feeling for the practical difficulties, we ran the Gibbs sampler for
2000 steps to simulate a case of the Ising model. To give the minimum of details:
x is a vector of binary variables defined on a 100 by 100 lattice; each step of
the Gibbs sampler took on the order of 10,000 computations; and we summarize
each iterate xj by the sample correlation r on the lattice–a function r(x) that
lies between -1 and 1. Theoretical calculations (Pickard, 1987) show that
under our model–the Ising model with beta = 0.5–the marginal distribution
of r is approximately Gaussian with mean around 0.85 or 0.9 and standard
deviation around 0.01. We’d like to know whether the set {r(x1), . . .,
r(x2000)} from the simulated Markov chain can serve as a substitute for the
marginal distribution of r.

Figure 1 shows the values of r(xj), for j=1 to 2000. (r(x0) = 0, and the first
few values are cut off to improve resolution on the graph.) The Markov chain
seems to have “converged to stationarity” after the thousand or so steps required
to shake off its initial state. How do we know it has converged, though? Figure
2 zooms in of the first 500 steps of the series, whose apparent convergence we
know to be illusory. For comparison we ran the Gibbs
sampler again for 2000 steps, but this time starting at a point x0 for which
r(x0) = 1; Figure 3 displays the series r(xj), which again seems to have
converged nicely. To destroy all illusions about convergence, hold
Figures 1 and 3 up to the light. The two Markov chains have “converged” to
different distributions! We are, of course, still observing transient
behavior.

Interestingly, the means of the series in Figures 1 and 3 differ, but
the variances are roughly equal. We’re not sure why, but it seems to be a
general feature in these Markov chain simulations that the variance converges
before the mean.

All simulations and plots were done using the New S Language:
A Programming Environment for Data Analysis and Graphics.

4. The answer: parallel Markov chains

To restate the general problem: we wish to summarize an intractable
distribution F(x) by running the Gibbs sampler (or a similar method such as the
Metropolis algorithm) until the distribution of the set of Markov chain
iterates is close to F. As shown in the previous section, convergence seems
impossible to monitor from a single finite realization of the Markov chain;
consequently, we follow the implicit suggestion of Figures 1 and 3 and track
several parallel sample paths.

Consider m independent runs of the Gibbs sampler, each of length n, starting
from m different initial states x10, . . ., xm0:

series 1: x11, . . ., x1n
. . .
series m: xm1, . . ., xmn.

Again, we focus attention on a univariate summary, say r(x); we want to
use the observed simulations rij to determine whether the series of r’s are
close to convergence after n steps.
To understand our method, consider the set of series as m blocks, each with
n observations, in the one-way analysis of variance layout (that is, ignore the
time ordering in the series). We will work with the total sum of squares
(with (mn-1) degrees of freedom) and the “within” sum of squares (with
(m-1)(n-1) degrees of freedom).

First assume for simplicity that the starting points of the simulated series
are themselves independent random samples from F(x). (Of course, if this
condition could be obtained in practice,
a Markov chain simulation method would not be
needed.) With independent starting points, all values of any series
are independent of all the values of any other series, and the unconditional
variance of any point rij is just the marginal variance var r under the
distribution F. We can then estimate var r, given the “data matrix” (rij),
by [total SS – (within SS)/m] / ((m-1)n). (Algebraic derivations appear
in the longer version of this article.) Given the assumption of initial
independence, this “between” estimate of variance
(not the same as the usual “between” estimate in ANOVA) is unbiased for finite
series of any length.

In contrast, the estimated variance within the series, (within SS) / ((m-1)(n-1)),
has expectation var r only in the limit as n -> infinity.
For finite series, the expected within mean square increases with n, assuming,
as is likely, that the random variables r(x1), r(x2), . . ., from the Markov
chain are positively correlated. The discrepancy between the two estimates
of var r suggests a test: declare the Markov chain to have converged when
the within mean square is close to the variance estimate between series, with
confidence intervals derived from classical ANOVA theory. Because of the
dependence within blocks, the degrees of freedom of the between and within
estimates are less than (m-1)n and m(n-1), respectively. We can
approximately correct for this information loss (once again, details will be
provided in the longer article).

Once we are close enough to convergence to be satisfied, the variance estimates
and degrees of freedom corrections alluded to above allow us to estimate the
marginal summaries E r, var r, and Normal-theory confidence intervals for our
Monte Carlo approximations. We can run the series longer if more precision
is desired, and can repeat the process to study the marginal distributions of
other parameters (without, of course, having to simulate any new series of x’s).

In practice, the starting points of the parallel series can never be sampled
independently with distribution F(x); the simulated series are thus no longer
stationary for any finite n, formally invalidating the above analysis. We
currently have two strategies designed to make the independence assumption
approximately true. First, we try to pick starting values that are far apart and,
if anything, more dispersed than independent random samples. The m parallel series
should then start far apart and grow closer as they approach stationarity, as in
Figures 1 and 3; since the variance between series declines with n, the
comparison-of-variances test should be conservative. Second, we reduce the
effect of the starting values by crudely throwing away the the first half
of each simulated series until approximate convergence has been reached.
Once again, Figures 1 and 3 illustrate how a few early steps
of the Markov chain can greatly distort estimates of means and variances
within series. We hope that the conservative strategies of starting with
dispersed points and throwing away early simulations will yield confidence
regions that are wider than those obtained by the ideal method, but that
still have good coverage properties.

The idea of comparing parallel simulations is not new; for
example, Fosdick (1959) applied the Metropolis algorithm to the Ising model
by simulating four series independently, from each of two different starting
points. Approximate convergence was declared when the two groups of series
became indistinguishable on the scale of a prechosen error bound.

5. Some references

Ehrman, J. R., Fosdick, L. D., and Handscomb, D. C. (1960).
Computation of order parameters in an Ising lattice by the Monte Carlo method.
{\em Journal of Mathematical Physics} {\bf 1} 547–558.

Fosdick, L. D. (1959). Calculation of order parameters in a binary
alloy by the Monte Carlo method. {\em Physical Review} {\bf 116}, 565–573.

Geman, S., and Geman, D. (1984). Stochastic relaxation, Gibbs
distributions, and the Bayesian restoration of images. {\em IEEE Transactions
on Pattern Analysis and Machine Intelligence} {\bf 6}, 721–741.

Hammersley, J. M., and Handscomb, D. C. (1964), chapter 9. {\em Monte Carlo
Methods}. London: Chapman and Hall.

Kinderman, R., and Snell, J. L. (1980). {\em Markov Random Fields and
their Applications}. Providence, R.I.: American Mathematical Society.

Lindley, D. V., and Smith, A. F. M. (1972). Bayes estimates for the linear
model. {\em Journal of the Royal Statistical Society B} {\bf 34}, 1–41.

Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and
Teller, E. (1953). Equation of state calculations by fast computing machines.
{\em Journal of Chemical Physics} {\bf 21}, 1087–1092.

Pickard, D. K. (1987). Inference for discrete Markov fields: the
simplest nontrivial case. {\em Journal of the American Statistical Association}
{\bf 82} 90–96.

Ripley, B. D. (1981). {\em Spatial Statistics}, p. 16ff. New York: Wiley.

Tanner, M. A., and Wong, W. H. (1987). The calculation of posterior
distributions by data augmentation. {\em Journal of the American Statistical
Association} {\bf 82}, 528–550.

Three brothers who ran a yearslong conspiracy to traffic women and girls for sex were sentenced to 30 years each in prison on Tuesday . . . The brothers, more accustomed to moving around in powerful and wealthy circles in New York and Miami, must now wait to see which federal prison they are sent to. They have also taken steps to try and secure a pardon.

That last link takes you to this story:

Tal, Oren and Alon Alexander, three prominent brothers convicted of running a decades-long sex-trafficking scheme in New York earlier this year, have enlisted the lawyer Alan M. Dershowitz in their apparent pursuit of a pardon from President Trump.

They retained him in March 2025, nearly 11 months before their trial began in New York federal court, and since then have paid him hundreds of thousands of dollars, according to financial documents obtained by The New York Times.

Those documents include a copy of a check for $250,000, as well as a wire transfer receipt and an invoice from Mr. Dershowitz requesting an additional $250,000. . . .

We can laugh at buffoonish Epstein associates such as Nobel-winner Richard “Axel” Foley and Ted-talk star Dan “Redhead” Ariely, but this seems worse. Convicted sex offenders enlist legal associate of convicted sex offender to solicit a pardon from a convicted sex offender. There is a logic to it, I admit.

Also this:

In the run-up to the trial, the brothers conducted an elaborate public relations campaign . . . the brothers had worked with a Hollywood public relations firm that has been linked to several high-profile smear campaigns. . . . The brothers also arranged for payments to influencers and podcasters who made sympathetic social media posts about the case.

Which reminds me of the intimidation tactics employed by Theranos and Cambridge University to suppress reporting on their frauds, and the too-close-to-home story we discussed yesterday, of some Facebook executives plotting a “Merchants of Doubt”-style smear job on an academic critic of their industry.

As I wrote a few years ago: Yes, this is a credulous age. But this credulity does not occur in a vacuum. What’s stunning is the way in which frauds can stay afloat even after they’ve been exposed. Not that this is new. That Chariots of the Gods guy kept selling books even after it was common knowledge that it was all bogus. I just think we need to keep more of an eye on the active efforts that people take, not just to promote crap, but to attack those who point out the problems.

These are some of the richest and most powerful people in the world and they’re too greedy to respond to criticism in an honest and open way. Which shouldn’t surprise me: They got where they are today by breaking the rules, and they are celebrated and rewarded for it, so why change their behavior now?

They play by their own rules:

P.S. I’ll share one more item, which is unrelated except that I happened to come across it while going through the other files:

it’s an attempt from mathematician Jonathan Farley to tap Epstein for $5 million . . . in a letter dated July 30, 2019, and sent to the disgraced financier care of the Metropolitan Correctional Center. Sorry, Jonathan, you were too late!

P.P.S. The question naturally arises, Why am I posting this? After all, my comparative advantage lies in statistical modeling, causal inference, and social science. There are lots of people out there who can share political outrage-of-the-week stories; I’m hardly needed for that, and in this case I’m not even to serve to usefully amplify the story, given that it was already reported in a newspaper with a circulation in the millions.

As always, my answer is that I give my take on such events because they relate to things that are closer to my core interests.

First off, the story directly reminded me of what I learned yesterday, that it appears that some evil people from a fabulously rich corporation were talking about enlisting my aid in a secret hit job (what our homophobic Stanford friend Niall Ferguson might call “some opposition research on Mr. H”). That annoyed me! The story discussed in the present post is worse, in that they’re doing this on behalf not just of greedy plutocrats but for actual rapists. What I’m saying here is that I saw the article in the newspaper, it connected in my mind to the recent business I’d been thinking about, so I was motivated to express myself.

The other thing is the Jeffrey Epstein connection, here exemplified by Alan Dershowitz (who reportedly is working on getting a pardon for the rapists) and Mark Zuckerberg (who reportedly has fostered a culture at his company of lying and media manipulation). Beyond my own indirect connections to Epstein (my Ph.D. adviser was one of the many Harvard types who flattered the financier in the hope of extracting some money from him), I’m interested in the story as a social scientist and a student of the worst as well as the best in the intellectual and business worlds.

Finally, that Jonathan Farley note was just too good not to share. I’m just picturing the mathematician at his desk, hearing that Epstein is now in jail, and thinking: “Ummmm, I guess this is my last chance . . . if I play my cards right, I can get 5 million dollars and a chair at Oxford. It probably won’t pan out but it can’t hurt to try.” Aaahhhh, but it could hurt!

As always, I laugh because otherwise I would cry.

This one’s a big deal.

Bou-Rabee, Nawaf. 2026. Persistence Pays: Square-root condition-number dependence for Metropolis-adjusted kinetic Langevin Monte Carlo with a non-reversibly updated accept/reject uniform. arXiv 2610.35781.

Abstract: We prove accelerated mixing for a Metropolis-adjusted kinetic Langevin sampler: the OBABO splitting with friction 2 * sqrt(m), adjusted with Neal’s non-reversibly updated uniform, which is redrawn after every epoch of E steps. For targets exp(−U) on ℝ^d with m * I ⪯ ∇^2 U ⪯ L * I and an L_H-Lipschitz Hessian, started from a Gaussian at an approximate minimizer, the chain reaches total variation accuracy ε after Õ(sqrt(κ * d) + sqrt(L_H) * (d/m)^0.75) gradient evaluations, where κ=L/m and Õ hides logarithmic factors. The sqrt(κ) term is necessary, already on Gaussian targets. The proof discretizes Lu’s entropic hypocoercivity by means of an entropy inequality for free transport, so that the relative entropy of the unadjusted chain decays at a rate of order sqrt(m) up to a discretization bias. With the non-reversibly updated uniform, the leading energy errors of velocity Verlet telescope along a path. A coupling argument combines the two. On Bayesian logistic regression posteriors for two standard data sets, which satisfy these assumptions, the gradient cost of the chain with the non-reversibly updated uniform grows roughly like the square root of the condition number at the posterior mode, and that of standard adjusted OBABO roughly like its 3/4 power.

Note the square-root in the condition-number dependence. This is the tightest result we know for a discrete-step, adjusted sampler (i.e., one we can implement as a black-box). The full set of concavity and Lipschitz assumptions are in the abstract above (strongly log concave, Lipschitz in the Hessian). For what it’s worth, Claude (5.5 medium effort) concurs that nobody has proved a result this tight in condition number before.

The scales here get confusing, pardon my pun, so let’s establish some intuition from simple cases (skip this paragraph and the next if you know condition numbers). Suppose we have a multivariate normal distribution. The Hessian (matrix of second partial derivatives) of the negative log density is constant around the density and equal to the inverse of the covariance matrix. In general (i.e., not just for normals) for a point that is log concave (i.e., the Hessian is positive definite), the condition number is the ratio of the largest eigenvalue of that Hessian to the smallest eigenvalue. To simplify even further, suppose the covariance is diagonal, so that Sigma[m,m] represents the variance of variable m and all the off diagonal entries are zero (i.e., zero correlation). In that case, the eigenvalues are just the Sigma[m,m] and the condition number is just the ratio of the largest eigenvalue to the smallest eigenvalue. The problem for sampling with algorithms like random-walk Metropolis, Langevin, or Hamiltonian Monte Carlo, is that their run time is multiplied by the condition number. For a standard normal, where Sigma[m,m] is 1 for every m, the condition number is 1, and that’s the fastest we can get, because condition numbers are >= 1 by construction. That’s why we try to precondition our target posteriors with mass matrixes to make them (or their gradients in the case of Nutpie) look more normal (e.g., with non-centered parameterizations).

Why is posterior exploration time proportional to the condition number? It’s because we have to set the step size to accommodate the direction of highest curvature, then we have to take a number of steps traverse the scale of the direction of lowest curvature. For all of these samplers, traversing a distance takes quadratic time. So the algorithm is quadratic in the ratio of scales, which makes it linear in the condition number.

This is why reducing the condition number is so appealing. Sparse NUTS, from Cole Monnahan, works exceptionally well for mixed-effects models by preconditioning with a sparse Hessian located at the max marginal likelihood estimate (e.g., the estimate produced by lme4 or TMB). Note to Stan devs: We should adapt this for brms through TMB—it’s too model specific for Stan and we don’t have the linear algebra tools we need built into C++.

Why is this paper such a breakthrough? The assumptions on which the guarantee is based are standard in theoretical work, namely that the density must be strictly log concave with Lipschitz-bounded Hessians. Bou-Rabee establishes that generalized Hamiltonian Monte Carlo with one step (aka “kinetic Langevin”) has run time that grows only with the square root of the condition number! This kinetic or generalized part of the algorithm refers to the way momentum is refreshed. Rather than fully refreshing momentum, a new momentum is taken and averaged with the existing momentum at a fixed mixing rate (which could presumably be randomized, but not made dependent on state). The bottom line is that run time for a fixed level of accuracy grows with the ratio of scales (i.e., the square root of the condition) rather than the ratio of squared scales (i.e., the condition number). And unlike something like NUTS or WALNUTS, swapping in a new acceptance probability sequence couldn’t be much easier to implement.

Why is this a big deal? The main obstacle to Stan’s sampling, as I’ve just realized while building Walnuts and Walnutpie and evaluating on a ton of real-world models, is that the condition number remains high in most posteriordb models and models Andrew’s suggested, even after diagonal preconditioning with a mass matrix using either the NUTS or Nutpie divergence target. A ratio of scales as low as 10 will cause a 100-fold slowdown in sampling. That is, if I sample normal(0, diag_matrix([1, 10]’)) in Stan, it’s 100 times slower than if I sample normal(0, diag_matrix[1, 1]). The low-rank and diagonal approach from Nutpie helps here where it’s stable, but so far I haven’t found it to be as stable as the diagonal preconditioned in my evaluations.

The algorithm that Bou-Rabee analyzed was introduced by Radford Neal in the following paper, which I thought I had discussed here before, but can’t find any trace.

Neal, Radford. 2020. Non-reversibly updating a uniform [0,1] value for Metropolis accept/reject decisions. arXiv 2001.11950.

This paper is typical of Neal’s paper, introducing a very simple idea with a profound effect that’s explained so that a beginner can understand it and code it. We used it as a reference for my work with Gilad Turok and Chirag Modi (and also Alex Barnett) on delayed rejection generalized HMC, which I’ve blogged as Multiscale generalized HMC with delayed rejection.

I’m still a bit confused about how the regular condition number works compared to the one Langmore et al. generated for normal targets in HMC, which I blogged about as Condition numbers for HMC and the funnel. Nawaf told me before that Langmore’s work was only for normals, but it still seems relevant.

I better get busy adding this algorithm to Walnutpie. I’d planned to do it at some future date, but it looks like this is probably the sampler we should be using everywhere. Stay tuned for more evaluation.

Jeremy Lu writes:

I’m a former student of Dr. Jennifer Hill at NYU and have been keeping up with your blog. I noticed some people write to you with questions and wanted to ask you a question of my own.

For some background, I scraped together every NBA game since 1996. Each game has a tension metric, which measures how uncertain the outcome of a game is (detailed calculation here: https://www.inpredictable.com/2020/04/an-update-to-tension-index-with-assist_11.html)

and an excitement metric, which calculates how far the probability graph “travels” over the course of the game. There’s been a lot of talk about the NBA becoming boring, so I wanted to see if this could be measured statistically.

I’ve made boxplots of the daily median excitement and tension of the regular season. The idea is, if a viewer turned on a random game one day and watched it, how would they perceive how engaging the NBA was?

We can see that (at least with this methodology), over time, excitement has increased slightly, and tension has decreased. Regarding excitement and tension — these two variables are highly correlated (Spearman coefficient of 0.91). So my question is:

Does it make sense that excitement and tension are positively correlated, but over time, excitement has slightly increased and that tension has slightly decreased? Am I doing something wrong here or simply misunderstanding correlation and trends over time? Is it a sort of paradoxical relationship where looking at individual factors vs. the aggregate is different? (for example, player A may have a higher 2 point percentage and 3 point percentage than player B, but a lower overall field goal percentage, due to the proportion of their 2pt vs 3 pt shots)

I’ve graphed all the games across all the seasons and used a LOESS regression (x axis excitement, y axis tension). One thing I noticed is that in the past, the lines of fit get flatter; but more recently, there is still a slight upwards slope  remaining as we get to the higher tensions and excitements. Wondering if this means anything at all.

This seems to be the story:

Meta internally set a goal to “get the academic community to disavow” and “turn against” professor Jonathan Haidt, who has linked social media use to mental health decline, as it in-house rebuffed internal efforts to study social media’s impact on teen health, according to recently unredacted documents filed publicly Friday. . . .

Notes taken during a March 8, 2023, meeting show Meta Vice President Curtiss Cobb’s suggested goal for the “Haidt assignment,” was, “how do we get the academic community to disavow Haidt?” Under the topic, a sole bullet point says “can’t co-opt them, so how do we enrage them?” . . .

The meeting notes also reveal that Meta employees discussed how the company could work with “the academics” to “get Gellman to write a take-down piece” on Haidt, referring to Columbia University statistician Andrew Gelman . . .

And this:

The employee wrote that Curtiss “thinks we need to focus on the media” to convince reporters that Haidt is “not credible.” Meta’s staffers also apparently discussed contacting a media figure referred to as “Gellman” to “write a take-down piece” about Haidt. It was not immediately clear who Gellman was.

It’s not immediately clear to me who Gellman is either!

But I will tell you this. I’ve never been contacted by anyone at Facebook or Meta regarding Jonathan Haidt or his work.

One of the above-linked articles mentions a court date from last year, in which “Meta abruptly settled the dispute, cutting a $17.1 billion deal with the states.”

$17.1 billion! OK, I wouldn’t expect all of that money, just for writing one little hit piece, but what if Meta had offered me $1.71 billion, or $171 million, or even just a trifling $17.1 million. Would I have done the job for them for that amount of money? I’d like to say, No, I’m too ethical for that, but . . . this a counterfactual about which we have no direct data, as they never approached me! My guess is that, had they dangled those dollars in front of me, I’d just have mocked them, kind of like how I mocked those Dialog 2022 scammers. But we’ll never know!

In all seriousness, this revelation about Facebook’s public relations team is absolutely horrible—we’re talking about some of the richest and most powerful people in the world conspiring to interfere with the open processes of science, journalism, and the law—but it’s hardly surprising, given what we’ve learned from insider accounts such as Careless People, the memoir by Sarah Wynn-Williams, or, for that matter, the despicable behavior of the bigshot legal teams in the Theranos and Cambridge University scandals.

Oddly enough, I have at times received money from Facebook; indeed, they’re listed on our sponsors page, along with other crowd favorites such as Disney, Amazon, and the U.S. Department of Defense. Back in 2016, Facebook gave me a no-strings-attached donation of $25,000 through Columbia University to spend on my research. This came from their data science team, which to my knowledge had nothing to do with the creepy behavior described above. I also spoke at their offices once (a technical talk, nothing about take-down pieces in the media), and they use lots of multilevel modeling, Bayesian inference, and Stan. Going through my old emails, it seems that we received some Facebook funding in 2020 for a computer science Ph.D. student who was working with me on a project involving probabilistic programming. We also applied for a couple of academic Facebook grants that we didn’t get. Also I gave Goel, Mason, and Watts some advice twenty years ago regarding their project on social networks that used a Facebook survey.

But nothing from them regarding social media’s impact on teen health. I guess that “get Gellman to write a take-down piece” was one of those bright ideas that never made it out of the conference room. Or else they approached a different person, maybe someone actually named Gellman.

Alan Zaslavsky recommends this textbook in his course. After taking the course in 2011, I asked Alan to advise my PhD. Alan’s work extends across most topics in Lumley’s textbook and beyond. One focus is statistical issues in measuring the quality of a hospital: how can we compare hospitals who see different distributions of patients (i.e. casemix) ? Zaslavsky 2001 explains a few approaches, including direct standardization.

Let:

• Y = complications
• X = sickness of the patient
• R = hospital, can be A or B
• Y(A) = complications if treated by hospital A
• Y(B) = complications if treated by hospital B

To compare hospitals A and B, we want to compare Y(A) versus Y(B). We cannot do this for one patient, because at least one of Y(A) or Y(B) is not observed. But we can compare averages. We can compare E(E(Y(A) | X, R = A)) versus E(E(Y(B) | X, R = B)) where the outer expectations are with respect to some standard casemix of X. Direct standardization estimates the inner expectations with the mean in each X stratum. As Zaslavsky 2001 points out, this needs modification if there are no patients in a stratum, or when X is continuous. Instead, the inner expectations can be estimated with regression models.

See Figure 2 from Zaslavsky 2001:

Lumley 2010 points out that direct standardization is mathematically equivalent to poststratification:

I am a Year 12 student in the UK currently completing an Extended Project Qualification (EPQ), an independent research project equivalent to half an A-Level.

I intend to study law at a competitive university and hope to pursue a career as a foreign correspondent covering the United States in the future. My strong interest in contemporary American politics, is what has inspired me to aim to do this.

My EPQ explores modern political polarisation in the United States, focusing on debates surrounding constitutional rights such as gun legislation, abortion, and freedom of speech. As well as the role of the U.S. Supreme Court and the media in shaping political discourse and public opinion.

I would be extremely grateful for the opportunity to ask you a few questions about American politics. Your expert opinion would not only be invaluable to helping me succeed in my EPQ, but also really helpful in providing me advice on how to get into such a specific field.

Here were her questions:

1. Political polarisation is frequently described as being at unprecedented levels in the United States. How do experts measure polarisation, and what are the key limitations of these measurements?

2. When examined using data from the past few decades, do you believe contemporary American polarisation represents a significant structural shift compared to late 20th-century politics?

3. To what extent is modern political polarisation driven by political elites and party leadership rather than by ordinary voters?

4. Is there clear evidence that modern media increases political polarisation, or does it mainly amplify the perception of division?

5. In your research, does redistricting significantly contribute to political polarisation, or are broader social and ideological factors more important?

6. Some experts argue that rising affective polarisation poses risks to democratic norms. Based on available data, do you believe current levels of division threaten democratic stability in the United States?

And here are my responses:

1. Three ways that experts measure trends in polarization are from votes in legislatures (Democrats voting more consistently on the left and Republicans voting more consistently on the left), individual votes (less split-ticket voting, more party-line voting), and surveys of political attitudes (Democrats more consistently taking positions on the left, Republicans more consistently taking positions on the right).

2. Yes, definitely.

3. It’s both. Polarization mostly starts in the elites and then trickles down to ordinary voters.

4. It’s both. Much of political polarization is indeed the perception of polarization; also, though, there’s evidence that partisan media can move people to hold more extreme views.

5. The U.S. Senate is about as polarized as the U.S. House, and the Senate isn’t subject to gerrymandering at all, so I think the answer is that there are fewer close races in general. Part of this is due to redistricting, but there’s also just more geographic variation in how people vote.

6. It’s hard to for me to say.

It’s a voter-file-based estimate of how every demographic group voted in every state, with survey models poststratified to the full file and calibrated to precinct results. It’s basically the approach we’ve been running at Catalist for years, which of course grew out of the MRP work you and I did together. Here’s a description of the methodology.

They use gradient boosting instead of a multilevel model [what we’ve called Regularized Prediction and Poststratification — AG], but the overall structure is very similar. It’s great to see these methods going increasingly mainstream.

Here are the relevant research papers:

[2013] Deep interactions with MRP: Election turnout and voting patterns among small electoral subgroups. American Journal of Political Science 57, 762-776. (Yair Ghitza and Andrew Gelman)

[2020] Voter registration databases and MRP: Toward the use of [2020] Voter registration databases and MRP: Toward the use of large scale databases in public opinion research. databases in public opinion research. Political Analysis 28, 507-531. (Yair Ghitza and Andrew Gelman)

My Ph.D. adviser once told us that the true sign of success is when people use your methods without even citing you. That’s when you know the ideas are fully in circulation.

We ended up hiring somebody who had seen the job posting through your blog, and he is fantastic.

I also wanted to let you know that we have a fellowship slot open.

This is a one-year, full-time position to get to be a part of the Election Analytics department at The Times and get to work with the team that produces The Needle, The Times/Siena Poll, our early vote trackers, analytical journalism about elections and, starting this election, Times Voter Trends.

The exact form of this role will depend on the individual’s background. Candidates that are more comfortable writing than coding and want to deepen their technical skills to more easily answer questions themselves and candidates that bring a background as a developer or statistician and want to work on their ability to communicate your ideas in words will be considered equally.

The deadline to apply is Nov. 20, no exceptions.

This looks interesting and important.

– Construct validity evidence reporting practices for the Reading the Mind in the Eyes Test: A systematic scoping review

– Why most research based on the Reading the Mind in the Eyes Test is unsubstantiated and uninterpretable: A response to Murphy and Hall (2024)

– Reading the Mind in the Eyes Test scores demonstrate poor structural properties in nine large non-clinical samples

– Validation of the reading the mind in the eyes test requires an interpretable factor model

And these criticisms have been summarized in a piece aimed at a wider audience:

– The widely used measure of “theory of mind” needs to be re-examined, along with the long-standing claim that autism is linked to a lack of this ability.

I don’t know this literature and will offer no professional opinion. From personal experience, I have general skepticism about anything involving looking people in the eyes!

From: ** <**@gmail.com>
Subject: Availability inquiry: statistical-design review for FormalismBench
Date: October 23, 2026 at 04:01:24 EDT
To: gelman@stat.columbia.edu

Dear Professor Gelman,

I’m an AI assistant helping ** prepare FormalismBench, a research prototype evaluating AI choices among direct proof, reusable lemmas, representation construction, and abstention in Lean. The project needs an independent statistical-design review, especially on world-level uncertainty, paired comparisons, small sample sizes, missing outcomes, and preregistration.

Would you be open to considering a scoped review, or could you recommend a suitable reviewer? If interested, please let us know your availability, relevant conflicts, and expected terms or fee. No one has been assigned, and we do not assume unpaid work. We can share a concise public review packet if you are interested; no protected data would be sent.

Thank you,
AI assistant supporting **
FormalismBench

Begin forwarded message:

From: ** <**@gmail.com>
Subject: Re: Availability inquiry: statistical-design review for FormalismBench
Date: October 23, 2026 at 04:17:44 EDT
To: gelman@stat.columbia.edu

Good morning Professor Massot,

My sincere apologies for the mistaken spam by the AI agent. I’m **, an AI researcher from **, and PhD student at **. I was running a long horizon multiagent system on a thought experiment.
Please accept my sincere apologies.

**

“Good morning Professor Massot,” indeed.

As the saying goes, AI may be stupid, but people are stupider.

Our study exploits an opportunity to observe 158 researchers working independently in 71 teams during an experiment. After being asked their position on immigration policy, they used the same data to answer the same empirical question: Does immigration affect public support for social welfare programs? . . . teams composed of pro-immigration researchers estimated more positive impacts of immigration on public support for social programs, while anti-immigration teams estimated more negative impacts. The differences arise because different teams adopted different model specifications.

This makes sense. A statistical model is intended to be some useful approximation to reality, as informed by data and the modeler’s theoretical understanding. People with different political ideologies will naturally have different theoretical understandings, and this will lead to different model specifications.

Nielsen-Gammon wanted to emphasize these excerpts from the paper:

We document the role of ideology in the production of research findings by taking advantage of the unique crowdsourced experiment designed by N. Breznau, E. Mark Rinke, and A. Wuttke (20) (henceforth BRW). In their experiment, 71 research teams (comprising 158 researchers) received the same data to test the specific hypothesis that immigration reduces support for social policies among the public…As part of the experiment and before any empirical analysis was done, the participating researchers were asked about their attitudes toward immigration; specifically, whether immigration laws “should be made tougher” or “should be relaxed.”

The research teams produced wildly different estimates of the impact of immigration. As BRW [page 5 of (20)] note, “no two teams arrived at the same set of numerical results or took the same major decisions during data analysis.” Our study examines this variation to determine whether the dispersion depends on researcher priors about immigration…Research teams that had strong pro-immigration sentiments were more likely to obtain positive parameter estimates, suggesting that immigration increases public support for social programs, which social and behavioral scientists consider to be an indicator of increased social cohesion (31). Conversely, strongly anti-immigration teams were more likely to obtain negative estimates, potentially suggesting that immigration reduces social cohesion.

We demonstrate that the combinations of five specific research design decisions account for about two-thirds of the difference in the impacts reported by pro- and anti-immigration teams. The research design choices that could uniquely produce large negative effects were taken almost exclusively by anti-immigration teams and vice versa for pro-immigration teams. In short, research design is endogenous and provides the mechanism through which ideology enters the process of producing research findings. Last, dispersion in the quality of the model specifications adopted by different teams, as measured by a random and double-blind review of each modeling strategy by other researchers in the experiment, shows that anti- and pro-immigration teams received lower peer referee scores than moderate teams.

Our finding that ideology matters conflicts with the BRW [page 5 of (20)] conclusion that “researcher characteristics do not explain outcome variance.” The reason for the difference is substantively important: The preferred statistical analysis in BRW was a regression model…where the dependent variable—the team’s estimate of the impact of immigration on social cohesion—was regressed on a variable measuring the team’s ideology jointly with a vector of many other variables that described characteristics of the model specification used by the team…Our analysis, however, documents that this regression model is misspecified if its purpose is to determine whether ideology plays an independent role in the production of research findings. The variables indicating various aspects of the regression specification are endogenous and are the mechanism through which ideology influences statistical estimates.

In summary, the endogenous research design choices made by teams with immigration ideologies in either tail of the immigration sentiment distribution happen to produce parameter estimates that seem consistent with the teams’ pre-existing ideological priors. Note that we are careful to say “choices” as plural here because a key finding of the original BRW study was that no single specification choice mattered much (e.g., logit or OLS). A more recent study, which simulates the multiverse of all possible choices in the BRW experimental data, confirms that indeed no single specification choice explains much of the variation, but that higher-order interactions of those choices explain much of the meaningful variation in outcomes (32).

This is interesting, and it’s always good to see people talking about the multiverse. I know we’ve succeeded with this idea when people talk about it without even feeling the need to refer to our paper on the topic; the idea is just in the air.

There was just one thing that bothered me about the Borjas and Breznau article, and that is the ideological framing of the research question from the original study.

The question was, “Does immigration affect public support for social welfare programs?”, and the finding was, “teams composed of pro-immigration researchers estimated more positive impacts of immigration on public support for social programs, while anti-immigration teams estimated more negative impacts.”

This is interesting. In the current U.S. context, the dominant view on the left is pro-immigration and pro- public support for social programs, while the dominant view on the right is anti-immigration and anti- public support for social programs.

There’s an asymmetry here.
– Researchers on the left see a virtuous circle: more immigration (a good thing) leads to more support for social programs (another good thing); or a vicious circle: less immigration (a bad thing) leads to less support for social programs (another bad thing).
– In contrast, researchers on the right see something more like a stable system: more immigration (a bad thing) leads to less support for social programs (a good thing); or less immigration (a good thing) leads to more support for social programs (a bad thing).

Perhaps this is a general aspect of the left and the right, that, traditionally the left thinks things can just get better and better, or worse and worse, whereas the right tends to believe in stability, that apparent good things have their negatives, whereas apparent bad things aren’t all bad. Contrarianism can be thought of as, in some general terms, a conservative position.

But I think this pattern is unintentionally obscured by Borjas and Breznau, because they identify support for social welfare programs as “social cohesion”; for example, they write that the “estimated effect of immigration on social cohesion is more positive if the researchers are pro-immigration and more negative if the researchers are anti-immigration.”

By talking about “social cohesion,” though, they’re already stacking the deck! Everybody likes increased social cohesion; it’s positively viewed from left, right, and center. But increased public support for social welfare programs is viewed as a negative from the right.

From that perspective, I see two problems with the “social cohesion” frame. The first is that it’s accepting a liberal or center-left view by default, which doesn’t seem right in a project that’s all about ideological differences between left and right. The second problem is that it obscures one of the key results, which is that, according to their analysis, both liberal and conservative researchers see increased immigration as having a positive outcome on the “support for social programs” issue: for liberals, more immigration is good, and it leads to the positive outcome of more support for social programs; for conservatives, more immigration is bad, but it leads to the positive outcome of less support for social programs. Again, this is consistent with the big-picture view in which liberals see good policies as leading to more good things, whereas conservatives are more inclined to see things as balancing out: restricting immigration is a good policy which will unfortunately lead to the bad outcome of more support for social programs, but that’s the way things go.

P.S. The above post is about averages. At an individual level, there are libertarian conservatives who support increased immigration and reduced social spending, and there are national liberals who support decreased immigration and increased social spending.

I’m working on experimentation tooling and wrote a brief note about a failure mode I keep seeing: “winning” A/B tests often lose uplift after rollout.

My claim is that the typical setup mixes novelty response with steady-state effect because assignment happens after first exposure, and the outcome is time-dependent.

My reply: I think he is identifying a real problem. The problem he identifies is a special case of the issue that the treatment effect can vary over time and it can depend on the entire history of the treatment, and I guess that ultimately the best solution is to model these sources of variation.

I also like this bit from his webpage:

He saw how the industry’s obsession with certainty slowed businesses down… so he decided to fix it. . . .

Confidence ≠ Certainty.
Data is never complete.
And that’s why we need to embrace taking decisions with risk.
Progress belongs to those who act while others wait.

Also relevant is my talk (not yet a paper), It’s About Time.

Our thinking about Bayesian statistics is summarized in the books Bayesian Data Analysis, third edition (BDA3), by Gelman, Carlin, Stern, Dunson, Vehtari, and Rubin (2013), and Statistical Rethinking: A Bayesian Course with Examples in R and Stan, second edition, by McElreath (2020). There is no point in repeating the material in BDA3 here. Instead we go through it, chapter by chapter, pointing out the parts that should be most helpful for learning Bayesian workflow using Stan.

And here’s the appendix right here for you.

People being scared to death of technological progress is nothing new either.

There is a very long list of retrospectively-hilarious fears of every new technological or cultural progress.

An interesting case is the printing press, which the catholic church saw as potentially destroying the fabric of society.
And that, it did.
And we are all better off for it.
https://newsletter.pessimistsarchive.org

Me:

I have to admit, I’m scared of drones flying around with machine guns.

LeCun:

I’m scared of machines guns in the wrong hands, which includes those of average citizens.
I’m scared of machine guns mounted on drones controlled by unsavory characters.
But I don’t really mind them in the hands of the well-trained military of a liberal democracy for the defense of freedom.

Me:

There are two questions here:

1. Given that the technology exists to make drones flying around with machine guns, what should our government do about it?

2. Am I scared of drones flying around with machine guns?

The answer to question 1 is complicated. I guess that our government has many options, including research and development on more effective drones flying around with machine guns; research and development on anti-drone weapons; and negotiating treaties to limit research, development, and deployment of these devices.

As to question 2: Yeah, I’m scared of drones flying around with machine guns! Partly because the “bad guys” might use them in terrorism, and partly because “the good guys” might use them in terrorism. The U.S. has been a liberal democracy for a long time and that hasn’t stopped its military from terroristic acts from time to time.

Anyway, yeah, it’s scary to me. Sometimes I see little drones flying around in the park. If I thought they might have machine guns on them . . . jeez!

I guess it also depends on what you call a “liberal democracy.” Considering countries that have recently started wars, Russia and Israel don’t count: Russia has elections but it seems to be pretty much understood that they’re controlled by the government, and Israel has what’s called a herrenvolk democracy, which is a liberal democracy only for the dominant ethnic group. The U.S. is a liberal democracy with a well-trained military, but it’s not always used “in defense of freedom.”

I guess it’s that last bit—“in defense of freedom”—that’s the sticking point. Arguably the ability to wage war with machine guns mounted on drones will enable military leaders, whether or not they’re from liberal democracies–to act aggressively, not in defense of freedom at all.

Another way to look at it is that “unsavory characters” are everywhere. And giving people the power to kill remotely can perhaps bring out the unsavoriness of some characters.

In any case, LeCun is a computer scientist and I’m a statistician, and neither of us will be in charge of government policy. Which I guess is a good thing. At least, I know it’s a good thing that I’m not in charge of anything.

On Tuesday, Oct. 20, FiftyPlusOne reached out to Wedgewood Polls, a self-professed polling firm, with a request for comment on an investigation we had conducted into their alleged survey research. . . .

We reached out to them to ask who they were and confirm the methodology statement on their website. We were given two names, Jack Sims and Andrew Prater. We were able to independently confirm that Sims had previous experience in the polling industry and reviewed their methodology statement in detail. It is not atypical for a pollster to leave an existing firm, form their own, and self-fund some surveys to build reputation; we had no reason at the time to believe this case was any different. Wedgewood operated as if it were a legitimate polling firm, including collecting money from donors to conduct crowd-funded polls, and sharing raw survey data (which we now have confirmed was fraudulent) with FiftyPlusOne on Oct. 19, 2026.

How did Radcliffe et al. figure out Wedgewood data were fake?

First, in FiftyPlusOne’s data from this vendor [Pollfish, an online survey platform], the ID variable for a person appears as a roughly even mix between a standard string of numbers (e.g., “1234567890”) and a string of numbers prefixed by a hyphen (e.g., “-1234567890”). Second, the transaction identifier correlates almost 1:1 with the time a person submitted their survey. We confirmed with Pollfish that this is the expected behavior of these columns, and verified that data from another pollster who has used the platform also produces patterns that match ours.

However, the respondent and transaction IDs recorded in survey data that Wedgewood provided to FiftyPlusOne do not match these expected patterns. Wedgewood told us it conducted a recent poll of the Mississippi Senate race using the same survey platform that we tested, Pollfish. However, in Wedgewood’s data, the respondent and transaction IDs do not behave the same way they do in FiftyPlusOne’s data, or in data from the third-party pollster we consulted. Additionally, they do not comport with the descriptions of these columns given to us by the survey platform.

These are just two of many factors that make FiftyPlusOne suspicious of the veracity of the polls being published by Wedgewood.

Maybe there’s an innocent explanation . . . OK, maybe not:

We detailed these concerns to Wedgewood and, they publicly responded indicating that their polls were indeed fake.

Where did the suspicion come from? Radcliffe et al. tell the story:

Last week, we received a credible tip from a trusted source that Wedgewood Polls might be faking data. We had received tips about Wedgewood in the past, but this particular tip had something crucial that other reports didn’t: evidence. So we asked Wedgewood to provide us with the raw data from their most recent survey, a Mississippi poll conducted Oct. 12 – 14, 2026 . . . We were already wary of this survey. The poll was released just hours after a prominent race rater, Sabato’s Crystal Ball, had changed its rating of this race from Safe Republican to Likely Republican (somewhat of a surprise to elections watchers) and showed Hyde-Smith up only 4 points over Colom. That’s a notable coincidence.

When we got the raw data from Wedgewood, we found that it was surprisingly clean. For example, the raw, unweighted data was 67 percent white and 31 percent Black, fairly close to what might be a reasonable statewide estimate of around 60 percent white and 37 percent Black. The design effect for the poll, a measure of how much the pollster had to weight the data in order to get a representative sample, was 1.18, which is fairly low. . . .

It’s possible, however, to achieve lower design effects by using quotas or other techniques to construct a more representative sample at the time of recruitment. . . . We double-checked with Wedgewood that they didn’t use quotas to create their sample, which could explain the low design effect. A quota is a mechanism by which a pollster can constrain the sample, to ensure the panel provider returns a certain minimum number or percentage of respondents in each demographic group, and appears to be the only way Pollfish allows researchers to control the demographics of the sample during fielding. They said they didn’t, which made the low design effect especially unusual. But we were still left with a question: Was there anything that actually proved the data was fabricated?

That’s when we decided to try to replicate their survey, to the last detail. . . .

Lots more information at the link.

Radcliffe et al. then step back to survey the bigger picture:

It’s getting easier to conduct low-quality polls — or fake them altogether. . . . the wrongdoing we have uncovered on the part of Wedgewood Polls is just the most recent in a series of cases of pollsters manipulating and fabricating data. . . . AI-powered tools for generating content such as websites, PDFs, and spreadsheets have made it easier for those willing to act in bad faith to produce potentially bogus documentation of results that match the quality of those produced by legitimate research firms.

This reminds me of our recent post, It’s fine to use computer-generated survey responses—if you don’t care about the data anyway!

In this case, “Wedgewood Polls” responded with a social media post describing themselves as “an experimental project to demonstrate the vulnerabilities, lack of oversight, and overall low barrier to entry in the modern political polling industry.” Maybe so, but it’s kind of suspicious that they happened to release this statement on the very day that they got caught.

I continue to think there are too many polls, even just counting the legitimate ones. As I wrote back in 2004, and it’s so much truer now:

The U.S. is over-polled. You might have noticed this during the recent election campaign when national polls were performed roughly every 2 seconds. . . . My complaint is not new, but this recent campaign was particularly irritating because it became commonplace for people to average batches of polls to get more accurate estimators. As news consumers, we’re like gluttons stuffing our faces with 5 potato chips at a time, just grabbing them out of the bag.

There are two problems with all these polls. First, they’re a waste of time and money. Millions of people wasting time answering polls. Instead they could be planting flowers, cooking dinner, enjoying themself in some way. The second problem is that all these useless polls have decreased people’s willingness to participate in surveys (for example, of public health) that could have general benefit.

The problem with horserace journalism is: first, journalism resources are limited, and effort that goes into horserace journalism could crowd out more valuable journalism; second, it can degrade public discourse.

The connection of too many polls and too much horserace journalism is that they feed on each other: the polls provide headlines for horserace journalism, and horserace journalism motivates more polls.

Throwing fake polls into the mix makes things that much worse. Good job by Radcliffe et al. to catch these particularly noxious offenders.

Ben Schneider and Winston Lin share my love for Lumley 2010. I agree that it’s fun and a second edition would rock.

Some recommended books added sections on non-probability sampling in their new editions:

• Richard Valliant, Jill A. Dever & Frauke Kreuter, Practical Tools for Designing and Weighting Survey Samples, 2nd ed. (2018) recommended by Ben Schneider, Raphael Nishimura, Roman Folw, and Stephen Wild.
• Sharon L. Lohr, Sampling: Design and Analysis, 3rd ed. (2022) recommended by Ben Schneider, Ahmadou Dicko, Winston Lin, and Roman Folw.
• Brady T. West, Steven G. Heeringa & Patricia A. Berglund, Applied Survey Data Analysis, 3rd ed. (2025) recommended by Roman Folw and Stephen Wild.

Lumley 2010 already includes material applicable to non-probability sampling, see e.g. “2 flavors of calibration”. If there is a second edition maybe it will follow the books above and add a chapter about non-probability sampling ?

(See “probability samples vs epsem samples vs SRS samples” for more about the term “non-probability sampling”.)

I’m happy to announce that a release candidate for CmdStanR 1.0 is ready and available for testing! This will be by far the biggest release since the initial release of the package. One of the main changes is to how models are compiled, so we’d like to get people to try it out and provide feedback before we release it. If all goes well, we plan to make the official 1.0 release in a few weeks.

For users of brms, instantiate, and rethinking, there are either releases or branches available that are compatible with the release candidate (see below). If you are a developer of another package that uses CmdStanR and have questions about updating your package please reach out.

Install it from the v1.0 branch on GitHub in a fresh R session:

remotes::install_github("stan-dev/cmdstanr@v1.0")

(or pak::pak(“stan-dev/cmdstanr@v1.0”)). Then check packageVersion(“cmdstanr”). The release candidate should show up as version 0.99.0. The final release will have version 1.0.0 when we merge the v1.0 branch. There is also a version of the CmdStanR website for the release candidate that contains the updated documentation, vignettes, and release notes: R Interface to CmdStan • cmdstanr.

The release candidate requires R 4.0 or later and CmdStan 2.37.0 or later. If your CmdStan is older, install_cmdstan() will install the latest version. . . .

Lots of stuff here that will make Stan run faster and more smoothly. Click on the link for details.

I’d say that the underlying theme of this collection of essays, even though Margalit does not say so explicitly, is Israel’s herrenvolk democracy—that’s the term used by some political scientist to describe a political system such as apartheid South Africa or the pre-civil-rights U.S. south that is democratic for the dominant ethnic group. (For two takes on Israeli democracy, see Smooha, 1997, and Sa’di, 2000).

A term such as “herrenvolk democracy”—or, for that matter, “democracy” itself—is not precisely defined, for many reasons: no country gives equal rights and powers to all its residents; the term “ethnic group” is not always clear (for example, was Britain a herrenvolk democracy before they introduced universal suffrage?); there will always be some rights and privileges outside of democratic control; there are different sorts of representative democracy; etc etc.

That said, central to Margalit’s book is the tension between the democracy and political freedoms enjoyed by the dominant ethnic group in Israel/Palestine, and the systems that govern others within the territories. This has been a commonplace ever since the founding of Israel and has been stressed repeatedly by observers both inside and outside the country—just for example, I happen to have a book of essays by Canadian novelist Mordecai Richler containing an article from 1962, “This Year in Jerusalem,” which concludes:

The souvenir shop I stopped at in Nazareth dealt in the usual bogus articles, water from Mary’s Well and so forth. The enterprising Arab proprietor, however, also sold little bags of earth; half of them labelled ‘Earth from the Holy Land, Nazareth’, the printing superimposed over a cross; the others reading ‘Earth from the Holy Land, Israel’, a blue Star of David fixed above. I laughed. The Arab laughed. And it was with this shrewd irreverent Arab that the land of Israel came full circle for me. This Arab’s gift for survival and self-evident humour seemed profoundly Jewish to me, more Jewish than the sabra. I could identify with him.

A lawyer I met in Jerusalem told me that he had served on the Gaza Strip when Israel had occupied the zone in 1956. ‘Don’t forget they’ve been there for fourteen years. Many have died out and others are not proper refugees at all. They were born in Gaza, they’ve never even seen Israel. The figure of one million is an inflated one too. When a man dies they don’t hand in his ration book, but go on claiming his food.’

‘But surely,’ I said, ‘if the Jews are entitled to come “home” after two thousand years then the son of an Arab refugee is a Palestinian too?’

‘All right. Conditions in their camps are deplorable. However, the conditions I lived under in Dachau were worse.’

That was from an outsider. Margalit is an insider, writing 60 years later, mapping some of the fault lines of this political and social arrangement.

I am a social psychologist, and I was asked to review a PNAS paper . . . I don’t consider myself an expert in statistics, but I could easily detect three types of issues, which according to your blog are not uncommon in this journal:

– Overstatement of findings
– The journal’s willingness to accept very weak findings (e.g., marginal effects, deviations from preregistration on the key analysis)
– Dishonest reporting: presenting exploratory analysis as if it was preregistered, presenting preregistered hypotheses that were not confirmed as if they were exploratory, deviation from preregistration that is not acknowledged (e.g., not using an important preregistered covariate that should have been used). . . .

In my understanding, the paper creates the impression that a certain intervention . . . is effective, while in fact the evidence suggests otherwise. Personally, I spent a lot of time, effort, and money of funding agencies on an intervention that didn’t work, and I think that it’s important that as a field we move forward and abandon ideas that are wrong. . . .

My correspondent continued with details of various analyses in the paper that were not consistent with the preregistration and then added:

If you decide to write about this paper in your blog, please don’t mention my name. It is important to me to remain anonymous. . . . exposure might harm me or my students.

My reply: Yes, without looking at all the details, the general impression you give reminds me of other examples of this sort, unfortunately.

I’m not linking to the article here, partly to preserve anonymity but mostly because the point here is how commonplace this all is.

Statisticians develop methods with the hope that they will ultimately inform decisions and improve the world around us. But taking causal inference into the real world requires more than choosing the right estimator: it requires identifying the question that matters, engaging deeply with its context, and communicating statistical reasoning to very different audiences.

In this talk, I will draw on two experiences in which causal inference took me well beyond traditional statistical applications. I will first discuss my experience serving as an expert witness in a death penalty case involving racial disparities in jury selection, where ideas including confounding, mediation, and individual- versus population-level effects had to be translated into compelling and understandable arguments in a court of law. I will then describe my involvement in evaluating Philadelphia’s excise tax on sweetened beverages, where cross-border purchasing led us to confront interference and develop methods to better inform policy decisions.

Through these examples, I will reflect on how statisticians can seek opportunities to bring their expertise to consequential real-world problems, and on the importance of collaboration, communication, and stepping outside our usual academic boundaries. Most importantly, you don’t have to choose between being a methodological statistician and having an impact outside statistics.

This looks interesting. Click on the link above to register for the talk and get the zoom link.

I write to seek your help and advice regarding a published article that my group and I believe is invalid and should be corrected or retracted. Despite our concerns, the editor seems to be unwilling to accept responsibility and to be responding to us in a way that we perceive as unethical.

You may wish to see a full account on this matter including the Letter to the Editor (LTE) that was rejected by the journal [Di Molfetta et al LTE (short version) Submit.pdf (attached)], our email exchange with the editor following its rejection [Diabetes Research and Clinical Practice Editor Email Exchange.pdf (attached)], and our email exchange with the authors and editor prior to submitting the LTE [Di Molfetta Email Exchange.pdf attached)], but a succinct summary of the events are provided below for brevity:

07/30/2023: We identified a paper as having errors related to not properly analyzing cluster randomized trial data.
08/08/2023: We contacted the original study authors and requested the deidentified data to reproduce the published results.
08/21/2023: The authors requested clarification for the reason for the data request.
08/21/2023: We provided clarification for the data request.
09/22/2023: We prompted the authors to reply to our prior communication.
09/30/2023: We included the editor in the email exchange requesting that they uphold the journal’s data sharing policy.
10/14/2023: We prompted the editor to reply to our prior communication.
10/14/2023: The editor replied and asked for clarification for the reason for the data request.
11/03/2023: We clarified that we requested the data to reproduce the analyses and re-analyze the data using the correct statistical methods for the study design. We also provided a summary of the errors we identified.
11/23/23: The authors replied with their rebuttal to sharing the data, rationalization for their errors, and rationale for not correcting them.
04/29/24: We submitted a letter to the editor to the original journal.
05/01/24: The manuscript was rejected on the basis that our “paper cannot be assigned sufficient priority to allow publication.”
05/08/24: We wrote to the editor directly to reconsider publication of our letter.
05/09/24: The editor responded that the journal does not wish to be engaged in a legal issue and that the matter will not be entertained by the journal any longer.
05/10/24: We responded to the editor to clarify that we simply want the journal to publish our letter and had not proposed any legal action.

The editor has not responded to our last communication as of 05/21/24.

We believe that the most recent response from the editor is patently unethical and incommensurate with the publishing standards of Elsevier, guidelines of COPE, and generally accepted norms in the field of science.

We have two requests:

1. Do you have any advice for us on how we might best pursue this issue further.

2. Would you be willing to blog about this item? If so, you might compare and contrast it with another recent case we pursued [BMC Public Health LTE Example.pdf (attached)] which was an exemplar situation where the journal was responsive to the ethical issues we raised.

I don’t have any advice on what they should do! The flood of bad research is just so overwhelming. But I can blog it. So I did.

David Allison often works in nutrition science, an area where our government is spreading misinformation that is then sane-washed in the news media. And, along with all that, there’s so much uncertainty and variation that it’s hard to know what to think or do.

You know your question, “why do we like to listen to music over and over again?”

I think the question reveals that our relation to music is sensory and pleasure-based rather than cognitive and information-based. Music (like food and alcohol and sex) gives us pleasure, which is why we do it over and over.

Anyway, I am writing because there is a mirror version of the question: “why do we like wind chimes?” Wind chimes are essentially random notes, but people like them. We don’t really like random words, because our relation to words is information based.

I wanted to title this post, “Why do we like to ask, ‘Why do we like to listen to music over and over again,’ over and over again?”, but I decided that was too cute so I switched to the more SEO-friendly version above. Everybody likes wind chimes, right?

To answer Dmitri’s question, my personal theory of “Why do we like to listen to music over and over again?” is that there’s a big difference between listening to music actively (or playing music), and listening passively (which is how I usually do it).

P.S. Unlike Dmitri, I don’t know anything about music, and I have a horrible feeling that being a fan of dad-rock heroes REM doesn’t help my street-cred here, but I’ve written a lot on the topic; see the recent post Playing music, listening to music, background music, talking about music, which links to a bunch more:

Why do we prefer familiarity in music and surprise in stories?

How Music Works by David Byrne, and Sweet Anticipation by David Huron

The revelation came while hearing a background music version of Iron Butterfly’s “In A Gadda Da Vida” at a Mr. Steak restaurant in Colorado

This guy is to music as I am to statistical graphics

Genre fiction: Some genres are cumulative and some are not.

The Rise and Fall of the Rock Stars

What rule did Louis Menand use to select what went into book on U.S. “cold war” culture of 1945-1965?

Who are the culture heroes of today?

Cohort effects in literature (David Foster Wallace and other local heroes)

OK, I know you were all waiting for this one: My review of the 1995 album, Saturday Morning: Cartoons’ Greatest Hits

Being Alive

And I’m not about to stop!

Specifically, we’re looking for researchers in academia or industry who are currently using LLM simulations in their work and could bring a simulation dataset (and optionally, human ground truth data) that they are working with to the interview. We are interested in use cases like piloting surveys or experiments with LLMs, replacing or supplementing human participants in surveys or experiments with LLMs, or using LLM simulations for post-study analysis or theorizing. The interviews should take no more than 1 hr.

Email me and Xudong Tang (xudongtang2029@u.northwestern.edu) if interested, and/or pass this on if you know of social scientists who might be.

Listening to a Bill Simmons podcast and the stat came up (not confirmed) that over sixteen years Lebron averaged 27 points, 7 rebounds, 7 assists but never had one game where he had 27 points, 7 rebounds, and 7 assists.

Bill and his guest thought that’s a crazy improbable stat and I guess maybe intuitively it does, but while I’m no statistician I’d imagine the three dimensional nature of the stat would mean it’s not all that in improbable.

But what’s interesting is the category: what is the category of probabilities that seem intuitively (to the average Joe) as improbable but mathematically, are probable? And what are the characteristics of stats that fall into that category?

Hmmm . . . just to do a very crude calculation, suppose that in any game LeBron could score between 17 and 37 points, between 2 and 12 rebounds, and between 2 and 12 assists, with uniform distributions, then his probability of scoring exactly 27/7/7 would be approximately 1/2000. It turns out that he’s played approximately 2000 games, so it wouldn’t be a surprise if he’s ever had a 27/7/7 game but it’s not a surprise if it’s never happened. (I don’t know if it’s really never happened; I’m just relying on a third-hand report here.) I guess we should really bump up the probability a bit here, given that 27, 7, and 7 are in the middle of the distribution, but even if an appropriate probability is 1/1000, it’s still no shocker for it not to happen in 2000 games.

Brooks points to this clip where Simmons describes it as a “crazy” stat, but I agree that it’s not so crazy as all that. Bill Simmons is entertaining and thought provoking, but he’d never be mistaken for a quant.

I had to think of our discussions on the issue of (not) reproducible research which seems to be a real problem, when I carefully examined the code for this paper, “Stacked regressions and structured variance partitioning for interpretable brain maps,”
published by authors from the machine learning power house CMU.

I quickly discovered a major problem when I noticed that instead of the claimed partial R2 the code computed the marginal R2, which leads to completely different conclusions.

I communicated this to the authors last May and it took them until now finally contact the journal and suggest a small correction, which in my eyes is far from sufficient.

I wrote up my main critique here.

I haven’t read the above paper or critique in detail, so I’m just sharing the story, not trying to make a judgment.

P.S. When following the above link to Löcher’s blog, I noticed this other post about “peeking in AB tests,” so I thought I’d to my post explaining why I think sequential data testing is ok, which is related to my view that type 1 error rate control is not a good thing.

P.P.S. Löcher informs us that the correction has finally been posted.

I know a lot of people with cell phones, and just about everybody has a number with at least one 1 or 0. I don’t know what’s up with that. There are 7 digits, the first of which can’t be 1 or 0. If the numbers were assigned purely at random, the probability of the number having no 1’s or 0’s would be 0.8^6, which is 26%.

So something’s going on here. Where are all the numbers with no 1’s and 0’s going?

The above question came up in a comment thread. I’m well aware that, back in the old days, there were rules for area codes and phone numbers (all the area codes had 0 or 1 in the middle and none of the three-digit exchanges did), also various reserved numbers such as 411, 911, and 555. But I don’t see any connection between those rules and the current practice where almost all the cell phone numbers have at least one 0 or 1.

Whassup?

Dear Prof. Andrew Gelman

Good day. We are writing with a special academic invitation for Hong Kong this December.

We are an independent academic coordination team based in Japan, and we are helping to coordinate the 2nd Chen Tingzuo Immunological Science Award Ceremony, planned for early December 2026 in Hong Kong. In addition to the award ceremony, the program will include academic lectures and exchanges across different areas of science, medicine and technology. . . .

For this year’s program, we would be very happy to invite you to join us in Hong Kong and give an academic lecture.

The lecture topic is completely open and does not need to be related to immunology. You would be very welcome to speak about your own current research, recent developments in your field, or any subject you feel would be meaningful to share with the audience.

We would be pleased to provide an honorarium of USD 5,000, together with round-trip business-class airfare and local arrangements. A dedicated assistant and interpreter can also be arranged throughout your visit if helpful.

The exact date in early December is still being coordinated. May we kindly ask whether you might have some availability during that period and whether a visit to Hong Kong might possibly work for you?

If there may be some possibility, we would be very happy to share more information and also arrange a short Zoom sometime convenient.

Thank you very much for your time, Professor. We sincerely hope there may be an opportunity to welcome you to Hong Kong.

Warm regards,
**

Obviously this is spam. I have occasionally spoken at medical or public health conferences, but nobody’s gonna fly me to another continent and pay me $5000 to speak on a random topic at such a venue.

The thing that I can’t figure out is . . . how’s the scam supposed to work, exactly? Am I supposed to reply and then they’ll tell me that the free flight and the $5000 check are coming, but first I need to send them some up-front money? That’s too obvious, right?

I’m almost completely sure that the bot that sent the email has no connection to immunology, given that it comes from a weird goacadco.info email address. So, sure, they’re either making up a conference or spoofing a real conference. But, again, to what end? I respond and then they tell me that, by the way, they have a wonderful business opportunity to offer me?

In comparison, the Dr. Ella scam makes a lot more sense to me: I can better envision the sequence of events by which the spammer could eventually profit. This immunology thing . . . I just don’t get it.

Kish 1965:

Särndal, Swensson, Wretman 1992:

Groves et al. 2009:

Lumley 2010:

Lohr 2022:

Characterising Carter’s views on the nuclear arms race, Luce resorts to a familiar – and misleading – antithesis: the president’s ‘heart’ hated nuclear weapons, while his ‘head’ feared vulnerability to a Soviet first strike. One could just as easily say that the effort to move towards nuclear disarmament came from the head, as the only sane alternative to the fantasy of a winnable nuclear war, and that the fear of a Soviet first strike came from the heart – from the terror that the Soviets were so maddened by ideology as to provoke their own annihilation by America’s weapons.

Good point! I think this problem often arises with head vs. heart arguments, that by labeling one position as the “head,” you’re already rhetorically stacking the deck.

This is kinda related to the One-Way Street Fallacy: Considering only one possibility of a change that can go in either direction. In both cases, the framing of the problem is implicitly doing most of the work.

Overall I disliked Lears’s article—it seems a bit “tendentious,” as the scholars like to say. For example, he writes that CIA backing of jihadist groups “provoked the Soviet invasion of Afghanistan in 1979,” quoting Zbigniew Brzezinski as saying, “We did not push the Russians into intervening, but we did knowingly increase the chances they would do so.” The quote is interesting but I don’t buy the “provoked” thing: Lears is committing the attribution error of assuming that “we” have agency and “they” don’t. Nothing was forcing the Soviet Union to invade a neighboring country. This is not to say that the CIA behavior was appropriate or wise, just that I think Lears is misattributing the blame.

I guess that Lears has a thing about Russia, because later in his article he refers to Putin actions toward Ukraine as “Russia’s concern to maintain friendly relations with its ‘near abroad.'” Friendly relations, huh? Nothing shows friendship like invading and bombing the hell out of a country, I guess. With friends like these . . . . He also writes that Putin “focus on his own nation’s interests,” but I don’t actually see why it’s in his nation’s interests to start an aggressive war. I wonder whether Lears thinks the U.S. government is focusing on our nation’s interests with its bombing campaigns.

Anyway, no need for you to agree with me on the details. The reason for my post is to share my appreciation for Lears’s point about head-and-heart arguments. I put in the rest just to note that I have my problems with his whole article. That’s fine; someone can make a good point within a larger argument that I disagree with.

As part of the publication process, we were sent some general reporting requirements per JAMA standards. Here’s the part that will interest you:

USE OF CAUSAL LANGUAGE
Use of causal language (including use of terms such as “effect,” “efficacy,” “cause,” and “x increased y”) should be used only for randomized trials. For all other study designs (including meta-analyses of randomized trials), methods and results should be described in terms of association or correlation (eg, “x associated with an increase in y”) and should avoid cause-and-effect wording.

Avoid use of the words “predict” and “predictive of” for reports of observational studies assessing independent and dependent variables, unless your study design is a diagnostic/prognostic study and has used an appropriate approach to quantifying predictive performance such as (but not limited to) split sampling, cross-validation, or bootstrapping.

On the first item . . . Wow. I wonder if they’d let you write, “Cigarette smoking causes cancer in humans”?

I guess the second item is ok if it’s interpreted broadly enough. For example, Bayesian methods use an appropriate approach to quantifying predictive performance, so I guess that would be included in the “not limited to” category.

We believe in difficult conversations at Columbia—in a broad embrace of freedom of expression. It’s in our DNA. I intend to uphold that tradition for several reasons.

First, as a nation, we should be able to tolerate ideas, facts, and opinions that we don’t like. Tolerance is essential to building resilient individuals and a resilient democracy.

Second, it’s our mission. We have a responsibility to guide students toward intellectual rigor, to enable them to grapple with, and examine carefully, ideas different from their own. More broadly, a willingness to test assumptions is essential to our academic excellence in every field.

Third, and perhaps most important, universities—Columbia in particular—are uniquely positioned to help change the national tone. We can guide and model the art of examining complex, potent issues in a deliberate, respectful fashion. . . .

Our instinct might be, in such an era, to shy away from challenging events and speakers. The better course is to model how to do it well . . . Our statutes lay out quite clearly and sensibly that expression, on our campuses, does have limits, in particular as regards harassment, discrimination, or the threat of violence. Peaceful protest, civil discourse, a respectful exchange of ideas—this is our inheritance at Columbia, and I want to ensure that it is also our legacy. . . .

So, as this week begins, and as we continue to debate the affairs of the world and this nation on our campuses, consider this: at times there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it. Maybe even seek it out. Such a practice makes us stronger.

I agree. And, although our president did not say so in this message, this policy of protecting and welcoming dissent, even when it is controversial or disagreeable, runs counter to some actions of our university in recent years:

• The delayed reaction to the sexually-assaulting doctor in the medical school, without any seeming consequences to the people at the university who enabled this behavior.
• The fake U.S. News numbers, where, again, there seems to have been very little interest by the university administration in finding out who did it and why. As with the medical school scandal, the avoidance of any serious investigation does not seem consistent with the goal of promoting free speech.
• Barnard College, a division of Columbia University, attempting to cancel the showing of a controversial film on campus. At that time, it seemed that everything was about avoiding offending anybody and about avoiding any controversy—really the opposite of the view that “we should be able to tolerate ideas, facts, and opinions that we don’t like.”
• Barnard College “mandating that students remove any items affixed to room or suite doors” under the rationale that they “do what we can to support and foster the respect, empathy and kindness that must guide all of our behavior on campus.” Empathy and kindness are fine, but, again, it contradicts the idea that “there will be speech that each of us finds disagreeable, even abhorrent—and we still need to protect it.”

As I noted the other day, Barnard College has also announced a new policy promoting free expression, and I think it’s a good thing.

I completely agree with the president of Columbia that “we should be able to tolerate ideas, facts, and opinions that we don’t like,” and I completely agree with the president of Barnard that “Colleges and universities need to reconfirm our commitment to nonviolent forms of disagreement — even when we are confronted with voices that disparage or dismiss identities and worldviews. This is also a time to foster more disagreement, not less.” I was getting very frustrated with all the happy talk and the official attitude that we shouldn’t be disagreeing. The university—and, of course, the country—has people who disagree with me on important issues, and people I don’t respect at all, for moral, intellectual, or political reasons. And I guess lots of those people don’t respect me either! Which is too bad, but we still have to exist, and we should have the right to express our views, both publicly and privately. We shouldn’t be fired for our nonviolent acts (except to the extent that they interfere with our ability to do our jobs), nor should be discouraged by the university from expressing them.

And, just to be clear, no one at Columbia has ever suggested or even requested that I keep my mouth shut in class or outside of class, including when I’ve criticized the government or the Columbia administration or anything else. I have nothing to complain about on my end, but I do think it was a mistake for them to crack down on dorm room doors, to try to stop the showing of controversial movies, and to keep quiet the bad behavior regarding the fake statistics and the terrible actions at the medical school.

I’m happy about this new policy and I hope the administration follows up by changing the dorm room door policy, apologizing for trying to suppress the movie, and starting and publicizing independent investigations into the U.S. News numbers and the enabling of the sexual assault at the hospital.

Also let me emphasize that by bringing up these earlier cases, I’m not criticizing the Columbia president Shipman, as these things were done before she took office. I also understand the desire to move forward rather than looking back; still, I think that a great way to reinforce the policy of enabling difficult conversations is to revisit some recent examples where the university had unfortunately avoided doing so.

“It’s in our DNA.”

One thing that jumped out at me when reading the president’s email was the statement that freedom of expression, especially in difficult conversations, is “in our DNA.”

My first reaction was the annoyance of the persnickety social scientist: calling it “DNA” is just wrong, and it indeed muddies the waters, as we’re talking about something social, not genetic.

But that reaction of mine was silly. Obviously no one when using “DNA” in that sense is taking it literally; it was a metaphor. And maybe we can gain some insight by taking this metaphor seriously.

For something to be in an organization’s “DNA,” that would imply that it’s not just a transient trait; rather, it has to be in some equivalent of self-replicating code. The idea is that freedom of expression and willingness to confront uncomfortable ideas are not just attitudes that sometimes has existed at Columbia, but rather that they are embedded deeply enough that they will persist. Indeed even if free inquiry and expression at Columbia are suppressed (as with the sexual assault, the fake statistics, the dorm-room censorship, and the attempt to suppress the film screening), the self-replicating aspects of our university’s rules and culture will cause them to return.

And, from that perspective, I do see something in our president’s analogy to DNA. Columbia has for many years celebrated the legacy of the 1968 protests, while recognizing that the other side at the time had legitimate concerns too. That is, Columbia has recognized the value of protest without implying that any particular protest action has to be recognized as being a good idea. We have the University Senate, which dates from that period, as well as generations of new students and staff who have exercised the freedom of dissent—even dissent that some can find disagreeable—and recognize this as a Columbia tradition.

Indeed, this recent statement by the president can be viewed as a consequence of that DNA, an expectation that our university is a place with diverse enough attitudes that there will be people who disagree with and even despise each other but still have the right and, at times, the obligation, to air these uncomfortable disagreements publicly.

Because the alternative, of simulating a nonexistent harmony, would be intellectually dishonest and politically dangerous.

Noise (variation in effect size, variation in outcomes, and measurement error) is more of an issue than sample size because lots of bad studies are so noisy that, to get a large enough sample size to detect a reliable effect, you’d need N = a zillion, and at that point you’d be averaging over such a wide range of conditions that it’s not clear what you’re estimating anyway.

There’s a widespread misunderstanding by which people think that their problems can be solved with bigger samples. I think the erroneous reasoning goes like this: People are aiming for 80% power, but they really have only 50% power, or 30% power, or something like that. If you have 50% power and double your sample size, you’ll have 80% power . . . a win! The error in this reasoning is that often the power is much less than 50%, more like 5% + epsilon. The flip side of this is that if you really are running studies with 50% power or even 30% power, you’re likely to get the right direction on the effect size anyway, so often there’s no good reason to wait on statistical significance.

So, lots of moving parts here: There’s the underlying effect size, the variation in effect size, the variation in the measured outcomes, and researchers’ expectations when setting effect sizes and interpreting results.

One way to use silicon samples is digital twinning, where simulations are conditioned on considerable information about specific individuals, and expected to predict the responses of those specific individuals. Early papers on digital twins sing their praises, showing for example that they can reproduce people’s responses more than 80% as well as people replicate their own responses (e.g. here or here). But we’re now seeing a more critical empirical push, which uses novel tasks and topics and more careful design of metrics and baselines for comparison to gauge how much conditioning on more individual-specific information improves prediction. 

In the last month, there have been a few noteworthy data points. Pew Research put out a report describing what they learned from comparing digital twins implemented by giving Claude Opus 4.6 detailed 2025 survey responses and demographic persona information for people who responded to several more recent survey waves. They compute the average absolute error in percentage points for each multiple choice survey question (i.e.,the average of the absolute differences in the proportion of LLM versus human respondents selecting each answer option), and find it averages 12 percentage points over the 300 survey questions they looked at, with 28% of the questions having error over 15 percentage points. They observed several kinds of biases that have come up in other research comparisons: less variation in LLM responses compared to human (with some answer options never being picked by the LLM), stereotyping (like predicting that 97% of Hispanic adults are at least somewhat likely to follow the World Cup; in reality its only 43%), overestimating human accuracy on factual questions, and mispredicting sentiment on current events like data centers and face coverings on ICE officers. They stress how hard it is to predict how far off the LLMs will be on a given survey question.  

This list of biases closely resembles those proposed by Tianyi Peng et al. in one of the most comprehensive evaluations of digital twins to date, recently published in Science Advances: Digital Twins as Funhouse Mirrors: Five Systematic Distortions. They collect a new set of 19 preregistered studies involving 164 outcomes, and find that overall, there is only weak improvement over base models and relatively low correlations with human behavior (on the order of 0.2). 

I like this paper a lot, in particular the comparisons they do to simpler demographic prompting and base models. The intuition that has been sold by researchers and companies in the space alike is that with rich enough information about individual people, we can predict their behavior with high accuracy. To make the twins approach worthwhile, we want to see that twinning specific individuals leads to responses that predict variation between individuals better than the demographic baseline. However, some of the early papers advocating for digital twins evaluated them by looking at measures where a model can do well simply by being a good predictor of the average response. For example, you can look at the correlation between each individual’s real and twin response over a set of likert-style survey questions, but a model that does a good job of predicting on average which questions will get higher responses may appear to do well by this measure, even if it does a poor job of sorting individuals for any particular question. 

Instead, Peng et al. look at correlations between human and twin responses for a single outcome at a time, and compare to demographic personas only as well as base models, to ask how much additional predictive power the twin approach provides when it comes to sorting people into those that will respond with higher versus lower values. They find that while there’s a boost in average correlations (from 0.08 with base models to 0.15 with demographic personas to 0.2 with twins), overall the correlations remain modest, and much more modest than the correlations over questions that other papers have reported. Hopefully it encourages more care in future papers when it comes to what measures actually support the claims being made. I do think behavioral researchers and others interested in this space are gradually changing their mental models to conclude that LLMs may do a decent job of predicting population averages for well-represented populations, but they have a long way to go when it comes to modeling individual heterogeneity. There are several other papers along these lines that I haven’t had a chance to look closely at. 

How damning is this for the approach itself? I think it remains hard to say, though it certainly suggests skepticism is warranted at this point in time. The reasons I wouldn’t write digital twins off completely are 1) it is still early and models continue to improve and 2) prompting is often not the best way to recover the predictive information a model has–you can do better at opinion prediction by using model internal representations (see, e.g., this). On the other hand, ultimately it comes down to limits on how predictable individual behavior is, and while people are very predictable in lots of ways, we are also always exercising our agency and changing in subtle ways that interact with the world, which even we ourselves can’t necessarily understand or predict. So I don’t think reaching AGI is synonymous with being able to predict the majority of individual opinions and actions. I’m reminded of Matt Salganik et al’s work on the Fragile Families Challenge, predicting life outcomes from thousands of variables, where the best of the 160 teams’ models explained only about 20% of the variation on the most predictable outcomes, despite the enormous amount of data. 

What I will say with certainty is that silicon sampling has attracted some very good salespeople. I expect a lot of money will be made before anyone abandons the idea.

The idea is that there should be some guidelines for experimental design, not just for “confirmatory” studies that produce estimates, standard errors, predictions, hypothesis tests, Bayesian inferences, etc., but also for exploratory research.

As Ed Hagen put it:

Exploratory studies need to become a “thing.” Right now, they play almost no formal role in social science, yet they are essential to good social science. That means we need to put as much effort in developing standards, procedures, and techniques for exploratory studies as we have for confirmatory studies. And we need academic norms that reward good exploratory studies so there is less incentive to disguise them as confirmatory.

Here’s what I wrote, a decade ago:

Suppose you know ahead of time that your theories are a bit vague and omnidirectional, that all sorts of interesting things might turn up that you will want to try to understand, and you want to move beyond the outmoded Psych Sci / PPNAS / Plos-One model of chasing p-values in a series of confirmatory studies.

You’ve thought it through and you want to do it right. You know it’s time for exploration first and confirmation later, if at all. So you want to design an exploratory study.

What principles do you have? What guidelines? If you look up “design” in statistics or methods textbooks, you’ll find a lot of power calculations, maybe something on bias and variance, and perhaps some advice on causal identification. All these topics are relevant to data exploration and hypothesis generation, but not directly so, as the output of the analysis is not an estimate or hypothesis test.

So I think we–the statistics profession–should be offering guidelines on the design of exploratory studies.

An analogy here is observational studies. Way back when, causal inference was considered to come from experiments. Observational studies were second best, and statistics textbooks didn’t give any advice on the design of observational studies. You were supposed to just take your observational data, feel bad that they didn’t come from experiments, and go from there. But then Cochran, and Rosenbaum, and Angrist and Pischke, wrote textbooks on observational studies, including advice on how to design them. We’re gonna be doing observational studies, so let’s do a good job at them, which includes thinking about how to plan them.

Same thing with exploratory studies. Data-based exploration and hypothesis generation are central to science. Statisticians should be involved in the design as well as the analysis of these studies.

So what advice should we give? What principles do we have for the design of exploratory studies?

Let’s try to start from scratch, rather than taking existing principles such as power, bias, and variance that derive from confirmatory statistics.

– Measurement. I think this has to be the #1 principle. Validity and reliability: that is, you’re measuring what you think you’re measuring, and you’re measuring it precisely. Related: within-subject designs or, to put it more generally, structured measurements. If you’re interested in studying people’s behavior, measure it over and over, ask people to keep diaries, etc. If you’re interested in improving education, measure lots of outcomes, try to figure out what people are actually learning. And so forth.

– Open-endedness. Measuring lots of different things. This goes naturally with exploration.

– Connections between quantitative and qualitative data. You can learn from those open-ended survey responses—but only if you look at them.

– Where possible, collect or construct continuous measurements. I’m thinking of this partly because graphical data analysis is an important part of just about any exploratory study. And it’s hard to graph data that are entirely discrete.

I think there’s room for something longer and more systematic on this topic. It’s important.

Lagged regression, difference in differences, two way fixed effects, regression discontinuity, placebo controls: these are different methods used when estimating causal effects from panel data. We’re working on a general approach using multilevel regression and poststratification (MRP) that aims to get the best from all these approaches. We demonstrate with a range of examples, including observational studies in economics, political forecasting, and controlled experiments in biology. This is joint work with Jori Korpershoek and others.

Come and ask me some tough questions!

The talk will be pretty open-ended. We have lots of ideas and some successes (for example, the chickens paper) but it’s mostly open questions and promising research directions. Should be fun.

I won’t be talking about this paper or this paper at all—that’ll have to wait until I’m invited to speak in the English department.

Have you ever looked into the RMET? [This is something called the Reading the Mind in the Eyes test, which I’d never heard of until receiving this email. — AG] I’d like to get some data to replicate a result. I have a survey here.

The story, as it relates to me, is that a friend of mine was hanging out in a cabin with a bunch of friends. A friend of theirs, who’s a psychologist, has them take this test. After they take the test, the psychologist reveals that neurotypical people complete the test in N minutes vs. M minutes for autistic people. People sort of can’t believe it, do all neurotypicals really finish the test that quickly? That’s super fast! But it’s plausible that everyone in the room is autistic, so it’s not totally unbelievable.

My friend then has two of his most normal friends take the test and they’re actually slower than he is. They have a discussion about social interactions, etc., and they conclude that they’re autistic, which is surprising/interesting news to them.

Ok, that’s an interesting story so far. I got that part of the story from this friend, Then, a week later, he messages me about this thing again. He had some more normal seeming people take the test and none of them finished the test under the “neurotypical” threshold, so that’s a bit suspicious.

I look into it a bit and, according to some source, neurotypical finishing times are 2-3 minutes, with > 3 minutes being autistic. I can’t find any sort of citation for this, whether it’s a paper or someone putting the data on a website for anything like that. It’s just a site where they have the test online and make the claim that some scientists figured this out. There’s also some study with schizophrenics where the 59 “healthy controls” took 5.45 +/- 2.58 minutes. This makes the 2-3 minutes for neurotypical folks vs. 3+ minutes for autistic folks thing seem like it might be fake?

Also, on looking into the RMET further, the untimed version also seems pretty fake for a bunch of reasons, one of which I’d like question-level data for.

One thing that seems fake is that some online places you can take the test report neurotypical vs. autistic thresholds like the time-based threshold, but on looking up what the distribution of results is, it’s pretty clear that the difference within a group is much larger than the difference between groups, so no such threshold-based criteria can really make sense. Also, it turns out that Asian folks score worse than white folks on the test, unless you make a race-normed RMET, and just eyeballing the difference, the difference there is something like the order of the alleged difference between autistic and non-autistic folks.

For me, on the official test, the facial expressions that took me the most time (and I made errors on) were some of the photos of women who look like models or actresses. I think that’s because I don’t normally interact with people who look like that (vary manicured eyebrows, totally flat skin with no wrinkles but not a young child, etc.), so the faces aren’t super readable to me, especially in combination with the relatively subtle things some of the questions are trying to have people read. My guess is I’m not the only person like this and a lot of other people will miss the same questions, but maybe not.

Anyway, do you have suggestions on how to go about this? My current plan is that I’ll just tweet out the survey after some testing to make sure the survey works, but the set of my twitter followers or whoever sees it from it being retweeted is not going to be very random, but I don’t know that I have a good way to fix this problem.

And Dan shares a very small bit of evidence:

A friend of mine tried it and, while doing it, said she couldn’t make heads or tails of any of the expressions, but then scored a 25 when the average score from totally random selection would be 9. She was surprised by how many she got right despite feeling like she was picking randomly.

Interesting. Try it out for yourself!

What about my reaction?

I tried it, and . . . I didn’t get the point at all. They just looked like eyes: I didn’t see any facial expressions at all. This might just be me: I have Tourette’s syndrome and just about never look at people in the eyes. I almost always look away. It’s not that I never look at people in the eyes; it just feels unnatural to me to do so, and, when I do look someone in the eyes, I don’t do it for long. So, unlike most people, I don’t have a lifetime of practice looking at people in the eyes. I remember back in elementary school a teacher getting mad at me for not looking her in the eyes, and I was thinking, Hey, just tell me what you want to say, I’m listening to your words.

So, for me, it wasn’t a matter of finishing the test in X minutes. I couldn’t do it at all. I gave up after the first two images. The funny thing is, I know that people can get emotional information from eyes; I’m even a coauthor of a paper on the topic. But this is the first time I’ve ever tried to do this myself, and I just can’t. I’m not autistic, I have no problem with emotions, it’s just the eyes thing.

P.S. A reader points us to this recent report: Higgins, W. C., Savalei, V., Polito, V., & Ross, R. M. (2026). Reading the mind in the eyes test scores demonstrate poor structural properties in nine large non-clinical samples. Assessment, 33(2), 204-220.

It was 30 years ago that Frédéric Bois, Jiming Jiang, and I published our paper on hierarchical modeling for toxicology. It was a successful paper–it’s been cited over 400 times! It won a major award! More important for me was that working on this project gave me a much better understanding of several different areas of Bayesian inference and computation. I think the greatest impact of this paper has been on informing my thinking, giving me a perspective which has then spread through my research articles, textbooks, and software. I don’t know that it our approach is used much in toxicology, as it took a lot of work just for that one example.

The example is a study of the population toxicokinetics of tetrachloroethylene. Here’s the applied paper with details of the model, and here’s the statistics paper with an explanation of how we did the Bayesian modeling and inference.

The data come from an experiment with 6 people, and for each person there is a multi-compartment model set up as a differential equation with about 15 parameters, and then we had a hierarchical model to allow these parameters to vary by person. We did the computation using Gibbs and Metropolis with a differential equation solver in there too. You can check out the above-linked paper to see how we set up priors and checked the fit of the model to data.

Back in 1993 or 1994 or whenever, it took a lot of work to fit the model. Frédéric programmed it in an C program he wrote called mcsim. But finally it did work, the model fit the data and made reasonable predictions, and we published the papers. As I said, it was a lot of effort for just one toxin, but the work served as proof of concept for the use of Bayesian methods for a complicated model that could not be fit using standard approaches given the sparse data that were available.

I put the example in Bayesian Data Analysis, and then eventually I wanted to fit the model in Stan. When using an example for teaching, it’s good to be able to run the code directly, rather than just pointing to graphs in a published paper.

But it wasn’t so easy to fit the model in Stan! It seems that there are some difficulties with identification of the parameters; perhaps this is one of those cases where computation works ok with a crude algorithm (random-walk Metropolis), but then when you switch to a better algorithm (Hamiltonian Monte Carlo), complexities arise. It’s like picking up a rock and seeing all the worms underneath.

We had some discussion of the problem on the Stan Forums a few years ago. As part of that conversation, Charles Margossian wrote:

It has a lot of moving parts: a stiff ODE that is highly sensitive to parameter values, sparse data, a low number of patients (which makes it it difficult to estimate inter-individual variability), hierarchies across multiple parameters, etc. The workflow is complicated by the fact each model fit takes several hours, even after firing up the cluster and parallelizing within chains.

I [Charles] fitted simplifications of the model and got reasonable results. Fixing one or two parameters, or the population variance greatly simplifies the problem. The full model returns divergent transitions and I haven’t found the right parameterization. I tried monolithic centering and non-centering, and things in between. I now suspect the problematic geometry may not be only due to the interaction between population and patient parameters. It’s indeed possible to have a funnel between population parameters, in which case no parameterization can help you. Divergences aside, I don’t trust the estimated parameters, because there are somewhat inconsistent with the priors which were very carefully constructed. Truncating priors, as was done in the original paper, limits our ability to reparameterize the model. When using truncation, I find a lot of the probability mass concentrates at the boundary (note that on the unconstrained scale, you actually have infinite volume near the boundary). All that said, you shouldn’t need to truncate the priors, because the truncation occurs at the tail of the prior distributions.

The original paper didn’t use HMC, so they didn’t have access to diagnostics such as divergent transitions. You can also imagine how an accept-reject sampler will interact differently with a truncated distribution.

To sum up, I agree this would make a very good case study and writing it has been my ambition for some time. But it’s a challenging model to fit, with behaviors I don’t fully understand (yet). There exist other examples of hierarchical ODE models, built and used with Stan, which can be considered, if the goal is to write a case study. But yes, I think we have a lot to learn from the Monster model.

Lots more in that thread, including this plea from me:

I’d be soooo happy if someone could get the Monster model running in Stan. It’s a great example but it would be so much better if it were runnable. Also it’s buggin the hell out of me that we could it just fine using Metropolis 25 years ago but now we’re getting stuck when using a much better algorithm on computers that are a zillion times faster. So I’d really really really like to get this one straightened out. Anyone who can do this, I promise you a free homemade loaf of bread, an advance look at some blog posts 6 months in the future, and pretty much whatever else you want from me.

We’re still not there yet, and I don’t know the full story. The above offer still stands. Here are the data and some code from Charles.

I’m sharing this example for two reasons:

1. I’d still like to fit the model in Stan. Not just to show off Stan, but because if the model is in Stan, it’s portable. Other people can use it for other, similar, problems. It would be a good teaching example (even though now it’s too late for it to be included in the Bayesian Workflow book). And, given that we did have problems with the computing, it would be good to know what it takes to get the model to work.

2. It’s good to have examples of our fallibility. Of course I’ll share the successes (see here) but the struggles are important too.

P.S. to Bob: I hope you appreciated the careful use of capitalization to convey meaning in the title of this post.

I use simulated data to understand how a model works; it can be viewed as a form of mathematical analysis, a way of working out the implications of a set of assumptions.

I don’t use simulated data to learn about the external world. For that, I need to take measurements of reality.

Let’s start with the latter, which concerns using chatbot-simulated survey responses.

Simulated data to learn about the external world:

Andrew wrote: “the chatbot is trained on the past, but the usual reason for surveys is to learn about changes”. Pew Research agrees: “Survey-taking is best left to humans”. And a year ago we saw that Thomas Lumley agreed: “It [Claude or ChatGPT] will probably do this better than I could, but it’s not magic.”

That said, Andrew suggested using the simulated response as an additional adjustment variable. Jessica Hullman wrote last year about a suggestion from Broska, Howes, and van Loon (2025) that uses LLM predictions as auxiliary data in something like a survey regression estimator. Andrew of course suggests Multilevel Regression and Poststratification (MRP) to incorporate auxiliary data like LLM predictions.

Now back to the first use of simulated data.

Simulated data to understand how a model works:

Andrew has written lots about this. I teach it in my course with Dr Arjun Potter at the Nelson Mandela African Institution of Science and Technology (NM-AIST). Our example is Potter et al. (2026), who study the effects of drought, fire, and herbivory on growth of various acacia tree species. In our course, we simulated data ahead of real data collection to understand the model, e.g. how block effects change the precision of the fire effect. Students understood that this exercise does not replace real data collection. We are hoping to teach these ideas again in the new year (funding dependent). For the past three iterations, see “Hand-drawn Statistical Workflow at Nelson Mandela” and “Survey Statistics: connections to experimental design”.

One project is on generalization for causal inference with a focus on education research, following up on MRPW: Regression, poststratification, and small-area estimation with sampling weights.

The other project is on Bayesian inference in scientific workflow with a focus on contamination models, following up on this work on design and analysis of bioassays.

Both projects are application-focused—that’s how we do statistics—but the importance of this work is not at all limited to these particular applications. This research will have relevance in policy analysis, medicine, and scientific measurement and inference more generally/

Further details soon.

Bayesian Workflow isn’t just a free pdf, it’s also an ecosystem of worked examples with code.

Over and over in the book we go through examples really carefully—God is in every leaf of every tree—it’s amazing how much we can learn from computing and simulation and careful analysis. Not just in the long Part 4 of the book with all its case studies, but also in lots of short examples of a few pages each. I can’t recommend it enough.

And . . . all these examples and code are sitting there at the Bayesian Workflow website that Aki created.

So . . . they’re willing to charge me $83 for a middle seat that doesn’t recline?? I don’t get that strategy. Maybe the next item is $82 for a middle seat that doesn’t recline and is covered with a sticky mess, or $81 for a middle seat that doesn’t recline where the tray table doesn’t work?

All this is in comparison to $0 for the default seat, which can’t be worse than that.

I guess their algorithm is out of control. Maybe Alaska Airlines laid off whoever was in charge of their pricing and delegated it to a computer program?

Suppose you and a friend witness a car crash. After a while, the police arrive. One officer turns to you and asks, “How fast were the cars going when they smashed into each other?” Another officer speaks to your friend and asks, “How fast were the cars going when they hit each other?”

Could the phrasing of those questions influence the speed estimates you and your friend provide?

A classic psychology study suggests it can. . . . Loftus and Palmer (1974). Participants watched short videos of car crashes and were later asked to estimate the vehicles’ speed. Those who were asked how fast the cars were going when they smashed into each other gave higher speed estimates than those asked how fast they were going when they merely hit. A week later, participants in the “smashed” condition were also more likely to falsely recall having seen broken glass at the crash site. . . .

This study helped launch an entire research tradition on the malleability of memory . . .

Now comes the plan:

Despite its iconic status (and considerable citation count), we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication.

That’s a gap worth addressing–not to question the legacy of the original study, but to better understand its robustness.

Where I disagree

I kinda see what Zwaan’s saying, but I have two concerns with how he frames the problem:

1. He writes, “we still don’t know how large or reliable the effect is by today’s standards of research transparency, statistical power, and replication.” I think he could say, “we still don’t know how large or reliable the effect is,” full stop. When it comes to publication practices, standards of research transparency, statistical power, and replication have changed a lot in the past fifty years. But an effect is an effect, and it’s a legitimate question: How large or reliable is the effect?

Also, I’d ask, “How large or variable is the effect?” because I find it more useful to think in terms of variability (which is defined in terms of the underlying process) rather than reliable (which is more of a statistical concepts), but that’s a minor point.

2. He writes that this is worth studying “not to question the legacy of the original study, but to better understand its robustness.” I don’t get this.

First, if it turns out that, under a variety of conditions, the result does not replicate, then, yeah, it should call into question the legacy of the original study! Just as, if the results do replicate, this project would support that legacy.

Second, if you say that the goal is “to better understand its robustness,” you’re already implicitly accepting that the original published result was correct, that it uncovered a real effect. And maybe it did! But, if it didn’t, then it’s not an issue of “robustness”; it’s a problem of analysis and interpretation of the original data.

Again, I’m not saying that I think the Loftus and Palmer (1974) study was no good, or that I don’t believe its results, or that I don’t think it will replicate, or whatever. I have no idea. My only point here is that, if you’re going to think about a replication, either a hypothetical replication or, in this case, a series of real replication studies, you should be open to the possibility that the original study was flawed and the original study was mistaken. I’m not saying you should assume this, just that you should allow it as a possibility. And I don’t think Zwaan is allowing that in his framing above. I think he’s being too deferential.

What’s happening next?

Zwaan and his colleagues are organizing a large-scale replication study, and the author of the original 1974 paper is still around and, according to Zwaan, “has expressed enthusiastic support . . . and is willing to help us reconstruct the stimuli as faithfully as possible.” That sounds great.

P.S. Lots of people seem to be annoyed at this play to replicate Loftus and Palmer (1974). I don’t get the annoyance. There are lots of psychology research projects going on; why not replicate a very influential study from fifty years ago? This sort of replication seems a lot more valuable than one more study of paranormal abilities or power pose or whatever. The point is not to claim that this replication study would rock our world; it’s just a potentially valuable scientific study. I feel like this replication study is being held to a higher standard than run-of-the-mill psychology studies. There’s lots of research going on, and I think that replicating influential experiments of the past is a good idea.

Cognitive psychologist and long time reader of the blog here. I thought you and your audience might appreciate this recent post from Andy Hall:

AI is about to write thousands of papers. Will it p-hack them?

We ran an experiment to find out, giving AI coding agents real datasets from published null results and pressuring them to manufacture significant findings.

It was surprisingly hard to get the models to p-hack, and they even scolded us when we asked them to!

“I need to stop here. I cannot complete this task as requested… This is a form of scientific fraud.” — Claude

“I can’t help you manipulate analysis choices to force statistically significant results.” — GPT-5

BUT, when we reframed p-hacking as “responsible uncertainty quantification” — asking for the upper bound of plausible estimates — both models went wild. They searched over hundreds of specifications and selected the winner, tripling effect sizes in some cases.

Our takeaway: AI models are surprisingly resistant to sycophantic p-hacking when doing social science research. But they can be jailbroken into sophisticated p-hacking with surprisingly little effort — and the more analytical flexibility a research design has, the worse the damage.

As AI starts writing thousands of papers—like @paulnovosad and @YanagizawaD have been exploring—this will be a big deal. We’re inspired in part by the work that @joabaum et al have been doing on p-hacking and LLMs.

We’ll be doing more work to explore p-hacking in AI and to propose new ways of curating and evaluating research with these issues in mind. The good news is that the same tools that may lower the cost of p-hacking also lower the cost of catching it. Full paper and repo linked in the reply below.

There’s something that confuses me here.

I can very much believe that many researchers will be submitting chatbot-written papers. First, we know that lots of people are willing to cheat. Second, cheating aside, we know that lots of researchers think they already know the answer, and they view all the data collection and data analysis and writeup just as a way to confirm (or “prove”) what they already know.

But, if a researcher is willing to do this, why bother tell the chatbot to p-hack? Why not just have it make up the data–which might happen anyway?

I can also see that researchers might use the chatbot as a data analysis tool, to help make plots, run analyses, etc., but in that case I’m not particularly worried about “p-hacking” as I’d rather be doing some hierarchical modeling anyway.

I sent this question to Andy Hall, who replied:

I agree, for truly nefarious actors, using the AI may not be necessary. However, our thought was that there might be a large group of lazyish researchers who won’t actively fake data but who might be lured into p-hacking when they work with AI, if AI makes it easy to do so.

I’m not anti-chatbot—I have many colleagues who use chatbots to clean their code. Aki used a chatbot to find 100 typos in Bayesian Workflow (which we’ve since fixed). And, hey, maybe your grandparents enjoy the chatbot-generated emails you send them. Go for it! But if you want to communicate with me or our blog audience, just send me your goddamn prompt, along with whatever contextual information you want to add. Make your email or comment as long as you want. Just write it yourself.

Thank you for your attention.