Unlike the Moon, which hangs in the sky like a lonely grandparent waiting for someone to visit, Mars leads a rich orbital life of its own and is not always around to entertain the itinerant astronaut. There is just one brief window every 26 months when travel between our two planets is feasible, and this constraint of orbital mechanics is so fundamental that we’ve known since Lindbergh crossed the Atlantic what a mission to Mars must look like. . . .

We shouldn’t send human beings to Mars, at least not anytime soon. Landing on Mars with existing technology would be a destructive, wasteful stunt whose only legacy would be to ruin the greatest natural history experiment in the Solar System. It would no more open a new era of spaceflight than a Phoenician sailor crossing the Atlantic in 500 B.C. would have opened up the New World. And it wouldn’t even be that much fun. . . .

It wasn’t always like this. There was a time when going to Mars made sense, back when astronauts were a cheap and lightweight alternative to costly machinery, and the main concern about finding life on Mars was whether all the trophy pelts could fit in the spacecraft. No one had been in space long enough to discover the degenerative effects of freefall, and it was widely accepted that not just exploration missions, but complicated instruments like space telescopes and weather satellites, were going to need a permanent crew.

But fifty years of progress in miniaturization and software changed the balance between robots and humans in space. Between 1960 and 2020, space probes improved by something like six orders of magnitude, while the technologies of long-duration spaceflight did not. Boiling the water out of urine still looks the same in 2023 as it did in 1960, or for that matter 1060. . . .

Mars is also not the planet we took it for. . . . The surface might be dry, but in most places there was water ice just underneath. Dynamic surface features hinted that water (or at least brine) was flowing to the surface from deep underground. . . . The news from the ground also got better. Arriving at Gale Crater in 2012, the Curiosity rover found itself looking at an ordinary lake bed, complete with organic sediment and odd stick-like structures that would be called fossils if we found them on Earth. The crater had been habitable for millions of years in the past, and something in it was still emitting methane at night. Over in its own crater, the Perseverance rover found complex organic molecules of indeterminate origin.

But the really exciting news for Mars was the discovery of unexpected life on Earth. . . . not just dozens of unsuspected microbial phyla, but two entire new branches of life . . . These new techniques confirmed that earth’s crust is inhabited to a depth of kilometers by a ‘deep biosphere’ of slow-living microbes nourished by geochemical processes and radioactive decay. . . . This underground ecology, which we have barely started to explore, might account for a third of the biomass on earth.

The fact that we failed to notice 99.999% of life on Earth until a few years ago is unsettling and has implications for Mars. The existence of a deep biosphere in particular narrows the habitability gap between our planets to the point where it probably doesn’t exist—there is likely at least one corner of Mars that an Earth organism could call home. . . . if our distant relatives are still alive in some deep Martian cave, then just about the worst way to go looking for them would be to land in a septic spacecraft.

But the fact that a Mars landing stopped making sense has not had the slightest impact on NASA’s plan to go there in a rocket-propelled terrarium.

And more:

The chief technical obstacle to a Mars landing is not propulsion, but a lack of reliable closed-loop life support. . . . The technology program required to close this gap would be remarkably circular, with no benefits outside the field of applied zero gravity zookeeping. The web of Rube Goldberg devices that recycles floating animal waste on the space station has already cost twice its weight in gold and there is little appetite for it here on Earth, where plants do a better job for free. I would compare keeping primates alive in spacecraft to trying to build a jet engine out of raisins. Both are colossal engineering problems, possibly the hardest ever attempted, but it does not follow that they are problems worth solving. In both cases, the difficulty flows from a very specific design constraint, and it’s worth revisiting that constraint one or ten times before starting to perform miracles of engineering. . . . The only way to explore Mars in our lifetime is to ditch the requirement that people accompany the machinery. . . .

In recent years, there’s been a remarkable division in space exploration. On one side of the divide are missions like Curiosity, James Webb, Gaia, or Euclid that are making new discoveries by the day. These projects have clearly defined goals and a formidable record of discovery.

On the other side, there is the International Space Station and the now twenty-year old effort to return Americans to the moon. These projects have no purpose other than perpetuating a human presence in space, and they eat through half the country’s space budget with nothing to show for it. Forget even Mars—we are further from landing on the Moon today than we were in 1965.

In going to Mars, we have a choice about which side of this ledger to be on.

This all makes sense. I’ve never thought much about this Mars mission thing because it’s always seemed like a bit of a joke. But if powerful people are really gonna try to use this as pretext to take a big chunk out of our national resources, then, yeah, it’s good to have people like Cegłowski pushing back.

To adjust for lots of variables, Si et al. 2020 turned to MRP (Multilevel Regression and Poststratification) and equivalent weights based on these models (see “struggles with equivalent weights”, continued struggles, and “equivalent models, equivalent weights (locally)”).

In a simulation study they compare:

• Ind-P: MRP with commonly-used Independent Normal priors
• Str-P: MRP with a structured prior, see below.
• Ind-W: equivalent weights version of Ind-P
• Str-W: equivalent weights version of Str-P
• Rake-W: classical raking weights, a type of calibrated weights
• PS-W: classical poststratification weights, another type of calibrated weights
• IP-W: inverse probability of selection weights

They cover the “3 flavors of survey weights”: equivalent weights, calibrated weights, IP-W.

I won’t bury the lead, they found MRP performed best, then equivalent weights, then classical weights (calibrated or IP-W). See their Figure 4.1 for the simulation scenario without terribly many empty poststratification cells:

With many empty poststratification cells, Str-P outperforms Ind-P. (They don’t redo Figure 4.1 for this scenario, which confused me a bit.) So what is this structure that helps ?

In “improving with structure” we saw that Gao et al. 2021 found it helpful to use the ordinal structure of variables like age. Si et al. 2020 use the interaction structure:

We induce structured prior distributions to be able to handle deep interactions and account for their hierarchy structure, where the high-order interaction terms will be excluded if one of the corresponding main effects is not selected.

I asked about sparse priors for MRP back in “Sparsified MRP”. I didn’t remember that Si et al. 2020 had worked on this ! Ok so they write their structure more generally but I find it easier to read with a specific example. Consider just 2 variables from their motivating NYC Longitudinal Study of Wellbeing: age (5 categories) and race (5 categories). Here’s how their Ind-P prior differs from the Str-P:

(I had Claude type up my hand-drawn notes, though I still share Brendan Leonard’s preference for hand-drawn materials.)

Si et al. 2020 say these are similar to the Horseshoe prior. It differs in two ways, I think ? First, Si et al. 2020 have the selection at the batch level (e.g. age or race). The usual Horseshoe would have local scale lambdas for each age and race category. Second, the usual Horseshoe would use a half-Cauchy rather than half-Normal prior on these local scales.

Ok let’s get back to the original concern: adjusting for lots of variables can lead to very large weights. Si et al. 2020 show in Figure 5.1 that the equivalent weights based on this structured prior model look much less variable than calibration weights (I don’t see the IP-W weights in the figure itself):

Walnutpie is an MCMC sampler for continuously differentiable densities coded in Python, accepting models coded in Stan, PyMC, NumPyro, JAX, and plain old Python.

Walnutpie is not an official Stan project

I thought this was worth saying up front. It may eventually migrate to Stan, but for now, we followed the Nutpie approach of building a standalone sampling package that worked with a range of packages for defining models.

R version

We plan to develop an R interface after we release version 1.0.0 of the Python interface. So hopefully in 2026.

pip installable

Walnutpie is on PyPI, so it’s pip installable. The documentation includes information on getting started, running models, and posterior analysis. We have not yet included case studies for modeling tools other than Stan and Python.

• Walnutpie documentation

Stan through C++

Walnutpie runs Stan models through C++ using BridgeStan, so there is no Python dispatch overhead for Stan sampling. The basic architecture of the API is based on Adrian Seyboldt’s sampler Nutpie. We are working on doing that for other packages like NumPy and JAX to the extent that we can.

GitHub source

Development discussions and source code are managed through GitHub.

• Walnutpie on GitHub

Features of Walnutpie

We are almost ready to release the paper on arXiv with all of the gory pseudocode details of all of the algorithms used for Walnutpie. It will explain the following points in detail.

1. Walnuts: The underlying Hamiltonian Monte Carlo sampler is Walnuts. Walnuts uses Nuts for choosing the number of steps per iteration. It further allows step sizes within the Hamiltonian dynamics simulation to be lowered when necessary to preserve simulation accuracy. This helps with robustness and with accuracy in multi-scale distributions (i.e., ones where the curvature as represented by the Hessian varies around the posterior). With a high tolerance threshold, Walnuts reverts to Nuts’s behavior.

2. Mass-matrix warmup: The mass-matrix warmup strategy is an online form of Nutpie (links to: the paper and software). Nutpie minimizes Fisher divergence by estimating the inverse mass matrix as the midpoint (in the appropriate manifold) between an estimate based on the variance of the draws and the covariance of the scores (gradients of the log density). The target is better than Nuts’s variance of draws in both convergence speed and sampling efficiency. Walnutpie only supports diagonal mass matrices (Nuts supports dense matrices and Nutpie supports low-rank plus diagonal and even more general normalizing flow approaches). The approach is online in the sense that it is not blocked like warmup in Nuts or Nutpie—it updates every iteration by exponentially discounting the past to mimic Stan’s exponentially increasing history sizes. We also borrow Nutpie’s mass matrix initialization based on a regularized outer product of gradients at the initial point.

3. Step-size warmup: The step size adaptation strategy has not changed, but the underlying stochastic gradient descent algorithm is Adam rather than dual averaging. We found Adam to be faster to converge and much more stable. Matt Hoffman included a hack in the original Nuts approach to stabilize dual averaging, but even with that it is not as stable as Adam.

4. Concurrency and automatic stopping: The underlying sampler is multi-threaded (using C++11 threads) with shared data. On top of the multi-threading, we have layered a convergence monitor in a separate thread that communicates with the chains through lock-free, latest-only, single-producer/single-consumer (SPSC) buffers (specifically, a triple buffer). The monitor automatically stops warmup when the mass matrices and step sizes have converged within tolerance to their cross-chain averages. The monitor automatically stops sampling when a target (traditional, non-split, non-ranked) R-hat; threshold is satisfied for the unnormalized log density, which typically converges more slowly than any of the individual parameters. It can also be configured to run for a fixed number of warmup and/or sampling iterations. The link between the original R-hat and effective sample size makes this essentially an unscaled ESS target.

5. Ragged chain summaries: Asynchronous concurrent execution of chains with automatic stopping produces chains of different lengths. Because ArviZ does not accept ragged chain input of this kind, we have included posterior analysis tools for means, variances/standard deviations, quantiles, traditional R-hat, effective sample size, and Monte Carlo standard error that work with ragged chains.

6. C++20: Walnutpie is implemented in C++20. As a programming language type fanatic (I’ve written two books with “type” in the title!), I don’t know how I survived without C++ concepts before C++20.

7. ctypes FFI: The foreign function interface in Python uses ctypes rather than a higher-level interface, which sidesteps the requirement of ABI compatibility of C++ binaries.

Developers

We would also like to welcome new developers who may want to get involved. There are already a stack of improvements we’d like to make, which we have enumerated on the GitHub issues.

Sources of algorithms

The Walnuts algorithm was a joint effort among Nawaf Bou-Rabee, Sifan Liu, Tore Kleppe, and Milo Marsden. Nutpie was developed by Adrian Seyboldt. Nuts, in the form used currently in Stan, was originally developed by Matt Hoffman and Andrew Gelman, then improved with multinomial sampling and mass matrix adaptation by Michael Betancourt. We haven’t yet added cross-chain adaptation as developed by Ben Bales, but the pieces are all in place to do so.

Brian Ward and I wrote all of the version 0.0.1 code with design advice and code review from Steve Bronder. Claude (the LLM) helped with code review and testing, but we didn’t use it to write the actual code (not out of principle, but because Brian and I both prefer the control of doing things manually).

Feedback

We would very much appreciate any feedback people have, including feedback on the documentation and ease of use, the source code, and performance.

We are happy to get feedback through issues on GitHub, through replies to this post, or through mail to one of the developers.

Regarding your blog of today, believe it or not, I knew Marshall Brickman pretty well before he became famous. He was an undergrad at the University of Wisconsin in Madison when I was a grad student there. We belonged to the same very left-wing eating co-op, “The Green Lantern.” In that era, undergrad males at Madison and elsewhere, had to take ROTC classes, and he kept intentionally failing them because he was very leftwing politically.

Inasmuch as I am a helpful sort, I offered to nominate him for “Military Ball King,” but he declined. Marshall was a really gifted musician as well as a writer. In addition, he looked strikingly like the actor Richard Carlson who played Herb Philbrick in the 1950s television series I Led 3 Lives.

The gimmick is that you’d buy the book if you wanted to enter the comedy field. Which doesn’t seem very likely, but, then again, who reads books either? So maybe I’m one of the few people reading this one for entertainment value alone.

One of the interviews is with Woody Allen collaborator Marshall Brickman, coauthor of the 1970s classics Sleeper, Annie Hall, and Manhattan. Lots of interesting stuff in this interview, including this:

Marshall Brickman: [Woody] found a whole new area of insight: relationships of a certain kind, psychoanalysis, and the creation of the so-called loser—mostly with women. To some extent, the lower-with-women character was someone Bob Hope would play, but in a much more general and mainstream way. Woody’s character was more ethnically and culturally specific.

Mike Sacks: It takes true genius to develop a comic character like that.

Brickman: It does, but it also requires a collaboration with the audience. It’s the only way you can do it. You have to get out there and do a variety of material. Over times, certain things, statistically, will continue to work, and other things will drop away, and the audience will tell you what seems correct for you—for what you project onstage as a personality.

Sacks: But even with that said, you can work for twenty years and never connect with the audience half as much as Woody Allen.

Brickman: That’s right. That’s the genius. Creating something that somehow resonates with an audience that strongly.

If only Allen had died in 1986 (after releasing Hannah and Her Sisters) or maybe 1992 (after Husbands and Wives), just think how high his reputation would be. Even setting aside his personal life, releasing a series of flawed projects takes a toll. Not everyone’s a Scorsese or Spielberg who can keep it up into their old age. If Woody had gone in 1986, we’d still be speculating about the amazing work he didn’t have the opportunity to do.

But the thing that really interested me in the above-quoted interview is what Brickman said about collaboration with the audience.

This is true of teaching too! Teaching a class, even writing a book, is a collaboration with the audience. A book doesn’t read itself. With books, the challenge is that the audience doesn’t see it until it comes out. To continue the analogy, a live class is like stand-up; a book is like a movie.

When writing books, we always have the audience in my mind. The trouble is that the book might never reach the audience. That’s what I think has happened with Active Statistics. I’d like to rearrange it and rerelease it as a book called Statistics Stories. Or maybe, Important Concepts of Statistics in Story Form. Also I wish we’d given Regression and Other Stories the more serious and descriptive title, Applied Regression and Causal Inference. Maybe we can rerelease it under that title.

Anyway, yeah, “collaboration with the audience”: definitely.

He’s using the chickens model from our Slamming the Sham paper in the more general setting of placebo control tests.

In econometrics, a “placebo control test” does not need to literally involve a placebo treatment; it more generally is used to describe a procedure in which the same statistical analysis that was used to estimate a causal effect is applied to a different dataset, or a different part of the existing dataset, in which the treatment did not occur.

For example, if you want to measure the effect of an intervention that occurred in 2021, you could repeat the analysis but using data from a different year. Or if you want to measure the effect on a particular outcome, you could repeat the analysis but looking at a different outcome that should be unaffected by the treatment.

The idea is that, if your estimation method has an artifact or systematic bias, this should show up in the placebo analysis as well, indicating a problem. Conversely, if the placebo analysis does not show an effect, this is taken as evidence that there is no artifact.

In his post, Gazvoda argues that, rather than using the result of the placebo check to make a go/no-go decision, it should be possible to partially adjust the treatment effect to account for the information in that supplementary analysis.

This makes sense to me, and of course I’m happy that he’s using our chickens model.

There’s a tricky thing going on here with the placebo check, which, interestingly, arose in the chicken example too, and that is that we usually don’t have any good theory for where the effect is coming from in the placebo control. After all, if our causal identification is working as designed, we shouldn’t even need the placebo comparison, as we’re already getting an unbiased estimate of the treatment effect. The placebo control is typically there to address unspecified concerns of bias. And, indeed, in practice, researchers don’t always do placebo controls. And when, as hoped, the placebo control shows no statistically significant effect, the inclination is to take that as a reassurance and move on, in the same way that is done with other robustness checks.

The lesson Gasvoda takes from the chickens example is that if replications are available, you can assess the evidence for the placebo adjustments being relevant: you can fit a multilevel model to estimate how much adjustment needs to be done.

From a sociology-of-science point of view, it’s interesting to me that the conventions in biomedical statistics and econometrics go in opposite directions:

– In biomedical statistics, the default recommended behavior is to compute the difference in differences, taking the estimated effect from the main experiment and subtracting the estimate from the placebo experiment. As we explain in the chickens paper, this correction has the disadvantage of doubling the variance of the estimate, a true statistical crime if, as is often the case, the effect of the placebo treatment is indistinguishable from zero.

– In econometrics, the default procedure, if you’re calling it “difference-in-differences estimation,” is the same as above. But if you’re calling it “placebo control,” and the placebo estimate is not statistically significant from zero, the default is to ignore the placebo results entirely, not to adjust for them.

In general we recommend a partial adjustment, with the amount of adjustment depending on the problem at hand. If there is internal replication, as in the chickens example, the appropriate adjustment factor can be estimated from the data. If it’s a one-shot experiment, you’ll need to use prior information. I don’t have any good examples demonstrating how to do that; it’s something we should do.

Statistical methods can be characterized in terms of how they can go wrong. Some common modes of “how can things go wrong” include:
– biased measurements,
– selection bias into the observed dataset,
– differences between treatment and control groups,
– nonstationarity or, in general, differences between sample and population of interest,
– biased or noisy statistical estimates,
– missing data,
– problems with functional forms (for example, using a linear model for a nonlinear relation, or not including important interactions),
– problems with error terms (dependence, choice of distribution, etc.),
– unmodeled spillovers, hierarchical structure, or other violations of causal model assumptions,
– researcher degrees of freedom and forking paths,
and lots more!

Regression discontinuity is used for observational studies, and the #1 thing that can go wrong in an observational study is #3 on the above list: differences between treatment and control groups.

The problem is that when researchers perform regression discontinuity analysis, they focus on the problem with the functional form of the expected outcome given the one predictor, often ignoring other potential predictors even if they are screaming to be included (as with age in this example). There’s all this obsessing over the functional form (and I guess Guido and I contributed to this) that’s kind of missing the point (as we discuss briefly here).

So, the problem is not that these users of regression discontinuity don’t know what to do with other predictors; it’s that they don’t know that they don’t know this.

As Tolstoy might have said had he been a data analyst, Models that fit are all alike; every poorly-fitting model is poorly fitting in its own way.

P.S. Here’s some background from the Quote Investigator on various version of the saying that I used as the title of this post.

Your process limits information search, computation, and time so it seems fast and frugal to me. Happy you still associate me with that term. It was something I coined as a grad student.

In your proposal, only hit papers get audited. It reminds me a bit of Mel Brooks’ The Producers in which the protagonists use the logic “who would audit a flop?” and stay under the radar by intentionally producing a bad show. Fraudsters have likely attempted to make their work seem worthy of publication while ensuring it doesn’t attract too much attention. Just like in The Producers, though, this sometimes backfires.

I don’t know about that! My impression with fraudsters is that they think that fraud is normal science, perhaps out of some mixture of bad education in research methods, a view that “everybody does it,” and a general lack of understanding of how non-cheaters (like you and me!) think. Think of people like Wansink who gave general advice to to the world on how to p-hack, or Gino and Ariely, who published papers on dishonesty, or Mary Rosh, who surely believes that whatever shady statistical manipulations she does are nothing compared to the dastardly deeds done by the Democrats.

I’m sure there’s tons of below-the-radar cheating and bad science that we don’t hear about, but a fair number of prominent science fraudsters seem to enjoy the limelight. One reason for this seemingly self-sabotaging behavior, I think, is that cheating enabled these people to attain great professional success for years. They had no reason to think the juice would stop flowing.

To return to my proposal with Andy King: I think it’s ok that only the hit papers get audited. Bad papers that get no intention aren’t doing much damage, right?

Goldstein adds:

By the way, I was just having a conversation about your sensing that something was amiss with the LaCour study. For years now I have used this quote of yours in a talk I give about putting numbers into perspective. I argue that it’s really important that people learn how to put numbers into perspective because if they don’t, they won’t notice that something is unusual and worthy of a deeper audit. You somehow sensed something was up with the Lacour result. You didn’t think it was fraud yet but you knew it was strange because you know how to put such differences into perspective:

A difference of 0.8 on a five-point scale . . . wow! You rarely see this sort of thing. Just do the math. On a 1-5 scale, the maximum theoretically possible change would be 4. But, considering that lots of people are already at “4” or “5” on the scale, it’s hard to imagine an average change of more than 2. And that would be massive. So we’re talking about a causal effect that’s a full 40% of what is pretty much the maximum change imaginable. Wow, indeed. And, judging by the small standard errors (again, see the graphs above), these effects are real, not obtained by capitalizing on chance or the statistical significance filter or anything like that.

My colleagues and I recently wrote a paper on this general topic of average effect sizes. It’s our contention that people generally are way too optimistic about possible effect sizes, in large part because they don’t think about variation. If you ask someone to hypothesize an effect size, you’ll typically get a guess of the largest effect that might occur.

But what if you don’t really care about effect size–you just want to know about the effect?

For example, maybe you don’t believe that women during certain times of the month are three times more likely to wear red or pink shirts, but you are interested in some sort of evolutionary psychology theory of sexual display. In that case, why should the effect size matter? Why care that a study reported an estimate that was ridiculously implausible?

I have two to this questions, and thus two reasons why effect size is important even for problems where you don’t directly care about effect sizes:

1. Effect sizes vary. An treatment that has an effect (that is, a true effect, not just an estimated effect) of 0.1 for one group of people in one setting could have an effect of -0.2 in some other scenario. A treatment effect in an experiment is the sum of all sorts of things, positive and negative, and there’s no logical reason to think the sign of the effect will be preserved. Effect size matters. The issue is not just that a smaller and more realistic effect size is less important; it’s also that smaller effects can be more easily produced by other factors, and this reduces the generality of any claims, even if the experiment at hand was done well.

2. Experiments produce standard errors as well as estimates. If the standard error from a study is large compared to any realistic effect size, then the study contains very little information. Effect size is important in understanding the informativeness of an experiment, and to do this right you need to have some sense of what the true effect size could be. You can’t just use a point estimate from the study itself, as this estimate will inherently be too noisy to use to judge the information in the study. As I wrote in this note for the Annals of Surgery, Post-hoc power using observed estimate of effect size is too noisy to be useful.

I was reading ‘The Bestseller Code.’ The book reports results from some regressions of the form:

bestseller or not ~ features of content

This got me thinking about what you can recover from such an exercise.

Say that there are two types of novels: type x and y. Let’s say that people prefer reading novels of type x. They are k% more likely to read a novel of type x than y.

At time t, the total number of novels is T, with novels of type x being 50%. And we can recover k% by regressing the number of people who read a novel on the type of novel.

Reader preference for type x novels leads writers to produce more novels of type x. At time t+1, the percentage of type of novels x being produced is 70%. But say that the total number of novels of type x that people can read is fixed at n_t. When we regress the number of people who read a novel on the type of novel, we can get that people are less likely to read novels of type x than y because we have more novels of type x at t+1.

I wrote a brief post based on the point here.

I agree, and this sort of thing has always puzzled me. I’m sure economists have looked into the matter, but I don’t know the literature so I’m just guessing here. Whenever people talk about the relative profitability of different genres, I wonder about what happens when the more profitable genre gets flooded with content. That doesn’t mean that bestseller regressions are useless, just that they represent at best some equilibrium of a process that I don’t understand very well.

The idea is that the distribution of the test statistic depends on the true parameter vector–there is no simple pivotal quantity as would exist under a linear model with no constraints–but you can work out the distribution conditional on the true parameter vector, and then you can average this distribution over the posterior for the parameters.

You can think of this as Bayesian–the marginal posterior distribution of the test statistic–but at the time I was thinking of it more as a generalization of the existing “plug-in” approach that would use a point estimate of the parameter vector. From that perspective, posterior averaging is a technique for getting a distribution with better frequency properties than you’d get from the maximum likelihood estimate, and that’s because in an image reconstruction problem with positivity constraints and lots and lots of pixels, the maximum likelihood estimate will almost certainly be on the boundary of parameter space. So just about any sort of averaging should get you closer to the true parameter value.

As noted above, I started working in this area in 1988. Around 1991 I got my thoughts organized enough to write a paper and give a talk on the topic, and . . . it got a generally hostile reception! The Bayesians didn’t like it because they didn’t like chi-squared tests or frequentist hypothesis testing more generally. They wanted me to do Bayes factors, which even then I realized were generally a bad idea (for more on the topic, see chapters 6 and 7 of BDA3 (it was all in chapter 6 of the first two editions) or this article from 1995). The non-Bayesians didn’t like it because it was Bayesian, and because I just presented the method, without trying to justify it based on minimax properties or whatever.

Xiao-Li, Hal, and I finally published a version of the article a few years later, and I resigned myself to the fact that posterior predictive checking was going to remain in the Bayesian world. Predictive checking was a hard sell to the Bayesians, but, after a few decades of exposure to BDA and lots of applied examples, they started to get used to the idea. I gave up on trying to promote the idea outside the Bayesian community.

But . . . it turns out that posterior predictive checking has been taken up for non-Bayesian uses! Aki pointed me to this R package called “performance” that “provides posterior predictive check methods for a variety of frequentist models.” Here’s their vignette on checking model assumption – linear models,” and here’s the article, performance: An R Package for Assessment, Comparison and Testing of Statistical Models, by Daniel Lüdecke, Mattan Ben-Shachar, Indrajeet Patil, Philip Waggoner, and Dominique Makowski. It came out in 2021 and has already been cited over 2500 times.

So it looks like people really are using posterior predictive checks for non-Bayesian models. And it took less than 40 years for it to happen!

We publish interval forecasts under a pre-registration scheme: each forecast is serialised, hashed and timestamped into a Bitcoin block before publication, so the stated interval cannot be adjusted after the outcome is known. Across 56 resolved forecasts our stated 50% intervals contained 47 outcomes; at the 24-hour horizon, 36 of 44. Under honest 50% intervals, P(>=36 of 44) is about 1.3e-05.

The diagnosis, and the part I would most like criticised: walk-forward across 20,058 observations showed the miscalibration was conditional rather than uniform – 55.6% coverage across all days, 67.1% restricted to the calmest fifth. Our first hypothesis, that the sample carried too much old high-volatility history, was tested and rejected. The surviving explanation is that interval width ignored the current volatility regime. Conditioning the historical sample on regime measured at each window’s open (not its close, which leaks the outcome) brings coverage to 50.8%.

Two things I am unsure about. The instruments are correlated, so effective sample size is well below 56 and I have not done that properly. And at a 30-day horizon the conditional interval comes out wider than the unconditional one, which I have kept but cannot fully account for.

I asked him what was the application, and he replied:

Crypto prices. 24-hour and 7-day intervals on eight pairs (BTC, ETH, SOL, BNB, XRP, DOGE, ADA, LINK), scored against Binance closes.

I chose it as the substrate rather than the subject. Outcomes resolve within a day, the reference price is unambiguous, and there is no data-vintage or revision problem, so a coverage check accumulates evidence quickly and cheaply. The same construction is what we use for sports and prediction-market questions, but those resolve slowly and the sample is thin.

I realise crypto invites a certain reaction, and the reaction is mostly deserved. The calibration question doesn’t depend on it – the same test applies to any published range.

He adds:

If it helps frame it, the two things I’m least confident about are the ones I’d want readers to attack:

1. The eight instruments are correlated, so the effective sample size is well below 56 and I have not handled that properly. The binomial p-value I quoted assumes independence it doesn’t have.

2. At a 30-day horizon the regime-conditional interval comes out wider than the unconditional one. I kept the result because it’s inconvenient, but I can’t fully account for it beyond “calm periods have historically preceded larger monthly moves”, which feels like a description rather than an explanation.

Data and outcomes are CC-BY, one row per sealed forecast with its hash:
https://huggingface.co/datasets/neuportal/neuportal-sealed-crypto-forecasts

My only quick thought here is that it’s indeed a good idea to look at conditional calibration, but you have to be careful only to condition on things in the forecast, not on the outcome. So when he writes, “67.1% restricted to the calmest fifth,” this is relevant as long as “calmest fifth” is something that can be determined before the outcome occurs.

More generally, yeah, it’s hard to evaluate the calibration of forecasts for correlated outcomes–this comes up in election prediction too. My only answer is to try to model intermediate outcomes as much as possible to avoid getting stuck in a purely empirical mode of evaluations.

Maybe you in comments have other ideas?

Revolutionary Road was bleak, too, so that aspect of Yates’s stories did not surprise me, but there was one big difference. In Revolutionary Road, you get the sense that the characters are tragic figures, going through their lives and taking wrong step after wrong step in a predetermined, necessary way, on the way to their sorry fates. The central irony of Revolutionary Road is not the characters’ failure to achieve their hopes, but rather the small stakes: not a Greek tragedy, it’s the alcohol-soaked middle-class suburbs. The dramatic but ultimately deflating title–the characters are not in fact on the road to any revolution–reinforces this point.

In contrast, the protagonists of the eleven stories in Yates’s collection have choices. They have “agency” (a topic I’ve discussed a couple times before; see here and here) and they do make decisions; the tragedy is that they end up in despair anyway. This is not a the sort of tragedy where the attempt to avert the prophecy leads to the prophecy being true; it’s more that the decisions made by the characters are never enough for them to avoid their fate.

Overall, Yates’s stories remind me of those of F. Scott Fitzgerald. They’re social documents in the manner of John O’Hara but with characters and plots that I find more convincing. In their directness, they prefigure later short-story writers such as Lorrie Moore and Raymond Carver, but, again, Yates’s stories seem more real to me. The only thing I find dated and jarring is his use of dialect (“terlet” for “toilet,” etc.), which, I know it’s part of the stories but it just seems distracting to me, in the same way that I find it distracting if writers make too frequent use of typographical tricks such as the use of italics for emphasis. That’s a minor thing, though.

Going meta

Much of what’s been written about Richard Yates is on the meta-topic of why his writing is not more popular despite his critical acclaim. An obvious reason is that Yates’s stories, humor-filled as they are, are pretty much unrelenting downers. In my earlier post, I compare to John Updike and Philip Roth, noting that a big difference is that those other writers scored salacious late-sixties bestsellers, which kept these authors in the conversation and led readers to return to their earlier work. Had Yates produced a bestseller later in his career, I think his early books could’ve become popular too. Also, Yates didn’t write for the New Yorker and, as I also wrote in my earlier post, by the time Revolutionary Road came out, the time had passed for his criticism of the emptiness of the criticism of the emptiness of suburbia.

Finally, Yates was writing in a time when there still were literary superstars: the aforementioned Updike and Roth, also James Jones, Norman Mailer, Saul Bellow, and others. After that came the literary-journalist superstars (Tom Wolfe, Joan Didion, Hunter S. Thompson, etc.), and now there’s not much of anybody at all. Literary stars like Sally Rooney don’t occupy anything like the cultural bandwith of the Gore Vidals of the earlier era–I’m not saying Rooney has less breadth than Vidal, just that her slot in the culture is much less. Richard Yates gets as much respect as modern masters such as Lorrie Moore or Garry Shteyngart or whatever, it’s just that expectations are lower. Nobody asks why Moore and Shteyngart don’t sell a zillion copies, because there’s no longer a mass audience for non-genre literature; in contrast, Yates could see the frustrating comparison to contemporaries such as Roth, Updike, etc., not to mention precursors such as John O’Hara and John P. Marquand.

Speaking of Roth: his first book, “Goodbye, Columbus and Other Stories,” is fine, and it’s a cultural milestone–he got a jump on the generation gap–but, as fiction, I think the stories in Eleven Kinds of Loneliness are much better; they just seem much more real and with more respect for the autonomy of the characters. And people keep going on about Roth. So, yeah, I see the frustration.

P.S. I looked up Yates on wikipedia and it pointed to this article by Stewart O’Nan from 1999, “The Lost World of Richard Yates.” It’s good. I’d never heard of Stewart O’Nan so I looked him up too . . . it seems that he’s published 20 novels and has co-authored with Stephen King. Wow–there are a lot of busy writers out there! Which I guess is part of the story.

From Little 2004:

Andrew’s 2007 “Struggles” paper and the 2026 MrPlew paper go the other way: start with an outcome model and back out the weights. These folks all worked pretty hard. I wondered if I could just use Thomas Lumley’s survey package to get the weights. See my posts “struggles with equivalent weights” and my continued struggles. But in the simulation I used a linear outcome model. It would have been more interesting with a logistic outcome model !

The linear case was handled by Andrew’s 2007 paper. The 2026 MrPlew paper extends this by noticing that the linear equivalent weights are derivatives, and using this to define locally equivalent weights, green highlighting by me:

Then the 2026 MrPlew folks use these equivalent weights to compare the target population to the weighted sample, as an MrP diagnostic. (Unlike the Huling & Mak paper on energy balancing weights, the MrPlew folks look at one covariate function at a time, not the entire covariate distribution at once.) One of their examples is from Lax & Phillips 2009 Gay Rights in the States, which I blogged about for pride. In this example, the implied covariate (im)balance for MrP looks not great ! Here’s their Figure 5b:

As Andrew blogged: “It makes sense that implied covariate balance can sometimes be worse for MRP than for raking. MRP is a smoothed version of raking, and unsmoothed raking can overfit.” Yes, but here it’s worse than the raw uncorrected balance ! The MrPlew authors caution here that the locally linear approximation may be extrapolating poorly, and that we shouldn’t over-rely on this model check. They also say that “logistic regression generally balances the variance-weighted covariates, but not the covariates themselves”. I am glad they included this illustrative example and I want to think more about it.

Thoughts ?

The logic of social science can work in two directions: generative modeling predicts behavior given assumed preferences, and inferential reasoning deduces preferences given observed behavior. Both these modes of reasoning can be valuable, but in choosing which mode to use, social scientists have the freedom to come to essentially opposite conclusions for any problem that comes in.

And here on the two modes of pop-microeconomics:

1. People are rational and respond to incentives. Behavior that looks irrational is actually completely rational once you think like an economist.

2. People are irrational and they need economists, with their open minds, to show them how to be rational and efficient.

Argument 1 is associated with “why do they do that?” sorts of puzzles. Why do they charge so much for candy at the movie theater, why are airline ticket prices such a mess, why are people drug addicts, etc. The usual answer is that there’s some rational reason for what seems like silly or self-destructive behavior.

Argument 2 is associated with “we can do better” claims such as why we should fire 80% of public-school teachers or Moneyball-style stories about how some clever entrepreneur has made a zillion dollars by exploiting some inefficiency in the market.

The trick is knowing whether you’re gonna get 1 or 2 above. They’re complete opposites!

More generally, almost any social science argument can be turned around 180 degrees, as I discussed here in the context of risk aversion.

Alex Tabarrok provides a great explanation of the challenges of two-way thinking, in this case regarding selection effects:

Should I be worried or reassured that my taxi driver isn’t wearing a seat belt? An econ puzzle.

I should be worried . . . and it reveals something of importance. First note that there is an incentive and a selection effect. All else equal, a driver without a seat belt should drive more carefully–that’s the rational response to increased personal risk. But drivers who forgo seat belts are probably more risk-loving or less safety-conscious across many dimensions. I think . . . the second effect, the selection effect, dominates: be worried. . . .

What makes this an economics puzzle is that it reveals a failure of the standard adverse selection story. . . . The taxi driver puzzle is a clean real-world case where the selection effect runs opposite to what adverse selection theory predicts. Adverse selection theory is correct that information asymmetries can challenge markets but it’s often not obvious which way the asymmetry runs . . . Moreover, preferences and norms can make the selection run the opposite way . . .

It’s a good example of how to avoid the one-way street fallacy.

I like Alex’s framing that there are real effects going in both directions, so that you can get a net positive or a net negative depending on which effect is bigger. I think this makes much more sense than supposing a single effect that goes in one direction or another.

This two-way thinking is also helpful when thinking about generalization. If someone does a study finding a positive effect of some treatment, and then you want to think about what will happen in a new setting, just think of it like this: There’s a positive effect A and a negative effect B, and the study finds that A – B > 0 in a certain setting. What will happen in a new setting? It depends on both A and B. There’s no reason to assume that the sign of A – B is some kind of universal invariant.

This comes up a lot in statistics too, that you have to be careful to avoid setting up your model in a way that would restrict the direction of the effects.

You’ll appreciate this I think. I ran Claude code on 96 specs for a popular difference-in-difference estimator with the identical specifications, ranging covariates only, for three languages (R, Python and Stata) and 2 packages for each. Different estimates — sometimes five times off. Appears to be coming from how each one handles failures on logits estimation with perfect separation, with some just refusing and others refusing and passing on garbage and others dropping down to run a logit on a constant despite being told to use Xs. And it has to do with how each package handles “big numbers” and this machine epsilon business that I barely know anything about. The fact that economists have pointed on selection on observables means they also haven’t invested in basic diagnostics around estimating propensity scores. But even turning Xs into z-scores still didn’t fix it. It’s the most extreme version of the garden of forking paths I think because people just assume languages and packages will all agree.

Ahhhh, perfect separation in logistic regression . . . one of my favorite topics! For an updated take, see section 14.6 of Regression and Other Stories. It makes sense that different packages will handle this differently.

Here’s Scott’s summary from his above-linked post:

The data come from a staggered rollout of mental health centers across Brazilian municipalities, 2002-2016. I have 29 covariates covering demographics, economics, health infrastructure, and population trends. Sixteen specifications sample the space from zero covariates up through all 29. All variation is coming from covariates, both within a package (i.e., which covariates) but more importantly across packages. And that’s the headline — even for identical specifications, you can get variation in estimates and sometimes as much as anywhere from an ATT of zero to an ATT of 2.38. Even if you drop the zero, some specifications yielded estimates ranging from 0.45 additional homicides to 2.38 homicides (measured as a municipality population homicide rate).

A five fold difference in estimated effects of a mental health hospital changing in the homicide rate is not trivial.

I guess one message is that there’s only so much you can learn from this sort of observational dataset. Ultimately it’s one data point in a larger explicit or implicit meta-analysis. More specifically, I like that Scott starts with the basic comparison and moves from there. I always think it’s a mistake to try to jump to the end.

Also I’d like to see a scatterplot with one dot for each municipality-year, plotting the outcome vs. some pre-treatment level, with different colors corresponding to treatment or control. I know that’s not the point of Scott’s post, but statistical packages can do graphs too! And more graphs, I’m not sure exactly what, but some way of getting a sense of the data. Ultimately this is being framed as a treatment vs. control comparison, so I want to see what the treatment and control data look like, before and after.

The multiverse!

Relatedly, Jessica points to two recent papers proposing the use of automated data analysis programs to perform multiverse analysis:

Many AI Analysts, One Dataset: Navigating the Agentic Data Science Multiverse, by Martin Bertran, Riccardo Fogliato, and Zhiwei Steven Wu

Editorial Screening when Science is Cheap, by Nic Fishman and Gabriel Sekeres

Jessica has discussed some related ideas at the end of this post and also here. And see here for lots more.

I agree with Scott and Jessica that these ideas all fit together somehow.

At the end of the second world war, Jews were a bit over 3% of the U.S. population, voted at a high rate, and were concentrated in the swing state of New York. Jews had two big issues–Israel and political liberalism, and the Jewish vote was a thing. Not the biggest thing in politics, but a powerful voting bloc in a swing state, that’s something.

Nowadays, Jews are about 2% of the population, and New York is no longer a swing state. When it comes to national politics, the Jewish vote doesn’t really matter. As I wrote nearly twenty years ago:

The underlying question, though, is why should we care about a voting bloc that represents only 2% of the population (and even if Jews turn out at a 50% higher rate than others, that would still be only 3% of the voters), most of whom are in non-battleground states such as New York, California, and New Jersey? Even in Florida, Jews are less than 4% of the population. I think a lot of this has to be about campaign contributions and news media influence. But, if so, the relevant questions have to do with intensity of opinions among elite Jews rather than aggregates.

The reason this comes up is that sometimes Jewish-related issues come up in politics and people will point to some poll or another of Jewish opinion. But Jewish public opinion doesn’t matter. When it comes to Jews in politics, what matters are the opinions and actions of campaign contributors and media executives.

Here’s the thing. Twenty years ago, it’s my impression that rich Jewish political donors and elite Jews in the news media were mostly politically aligned with average American Jews. Not exactly–I’m guessing the rich donors were not as far to the left on economic issues as random Jews in the country–but pretty much politically liberal and pro-Israel. Since then, things have changed: American Jews are split on Israel and remain strongly Democratic, but now there are many prominent Jewish donors and media executives on the right, both with regard to Israeli and American politics.

So, when a Jewish lobbying group taking some conservative position, and people point out that this is not the majority opinion of American Jews, my take on it is that, from a political perspective, the majority opinion of American Jews doesn’t really matter; to the extent that national politicians would think about Jewish issues, it’s the big donors and media executives that are the most relevant.

Similar issues arise with any ethnic group. The average voter will disagree on key issues with donors and influencers. It’s just particularly clear with Jewish Americans because the direct relevance of the voters has declined so much over the years, but people are still often in the habit of framing ethnic politics in terms of voters.

Mormons as the conservative counterpart to Jews

Indeed, you could say something similar about Mormons, who, like Jews, are a minority religion in the United States, with a strong ethnic identity, adherents concentrated in a few states, many rich people, strong political views, and lots of political involvement.

A few years ago we looked at political attitudes among rich and poor among different groups:

Jews mostly were voting for Democrats, Mormons for Republicans. No surprise.

More interesting was the difference between rich and poor. Rich Jews and poor Jews had similar voting patters–ok, actually, we’re comparing the upper third of income to the lower third here, so we’re not really talking about rich people here, we’re just using data from the higher- or lower-income people in national surveys. Rich Mormons and poor Mormons voted much more differently, with rich members of that religion being much more likely to vote for Republicans.

Since 2004, things have changed, and now there might be a divergence between rich Jews, who seem to be very pro-Israel and moving toward the Republicans (at least from what we hear about big campaign contributors), and the general mass of Jews in the country who are more divided on Israel and mostly remain supportive of the Democrats.

Similarly with Mormons: I guess the rich Mormons remain staunch Republicans, but lower-income Mormons may well have moved along with other lower-income white people toward the Republicans too. So the rich-poor voting gap among Mormons might not be so much larger than among Jews anymore.

My corresponded noted that my post pointed out all sorts of statistical analysis problems in the paper, red flags all of the place, and he asked why I would not have right away suspected fraud.

So let me explain.

My remark, “I don’t know that I would’ve noticed any problems with the paper had it been sent to me cold,” reflects that, when I’m sent a paper to review, I don’t review it forensically. Once I see the problems, I can’t un-see them, and, as I wrote in my post, the problems in that paper are clear, but in a quick review I might not have looked into all those details. I don’t think it is a requirement of a reviewer for a journal to investigate a submission in detail. Ultimately the correctness of an article is the responsibility of the author.

To put it another way, I don’t see journal review as a gatekeeping process that will keep erroneous articles from being published; rather, I see it as providing some information to the author and editor.

I’m on record as saying that the problem with peer review is the peers. That doesn’t make peer review useless. It is what it is. Peers can be very helpful in pointing out connections to the literature. I just don’t think it makes sense to see reviewers as gatekeepers. Again, the correctness of the work is ultimately the responsibility of the author. And, in any case, there will be post-publication review of papers that are of interest to later readers.

Generic language, that is, language that refers to a category as an abstract whole (e.g., ‘Girls like pink’) rather than specific individuals (e.g., ‘This girl likes pink’), is a common means by which children learn about social kinds. Here, we propose that children interpret generics as signaling that their referenced categories are natural, objective, and have distinctive features, and, thus, in the social domain, that such language affects children’s beliefs about the social world in ways that extend far beyond the content they explicitly communicate. On this account, even generics expressing uncontentious content (e.g., ‘Girls are great at math’) can lead children to think of categories as defining fundamentally distinct kinds of people and contribute to the development of stereotypes and other problematic social phenomena.

Here’s the full article.

In reaction to my recent post, If Cuomo had been able to run against Mamdani head-to-head, would he have won?, sociologist Kieran Healy posted a pair of maps showing precinct-level results from the recent New York mayoral election. One of them is above, and below is a detail from the other.

The top map uses a bidirectional color scheme of the sort that I like, except that usually I’d see the darker colors at the extremes, fading to white in the middle, so it was surprising for me to see this go the other way.

In his post, Healy shares a lot of detail, not just on his choices for what the maps should look like, but also on the data processing and all the steps along the way.

Following up, Healy writes:

The overall effect of a dot-density plot is sensitive to the choice of colors, particularly when there is more than one kind of dot being displayed. This in turn is heavily related to relative brightness, which in practice itself depends not only on the values encoded on the plot but on the sort of monitor or screen it’s being displayed or projected on, how much ambient light there is, etc, etc.

Again–also as noted in the post–while dot-density plots do better than choropleths in overcoming the “Land Doesn’t Vote” problem (in this case, “Precincts aren’t real”), at the end of the day any spatial representation of something like individual voting data is going to be caught out by this issue one way or another. So overall you just have to show multiple representations of the data, many or most of which will be better off not being maps at all. (Cartograms, whether based on grids or some sort of sphere-packing methods, are another solution, and of course create their own problems.) There’s no one beats-all-comers method, and I don’t present the dot-density map as one. I use stuff like this in my own classes precisely because you end up with a lot of choices to make on how to view the data, and it’s good to encourage students to work through the choices and their consequences.

It’s also good for getting across to students how those choices, and the tradeoffs associated with them, can’t really be effectively communicated in the graph or map itself, because they’ve already been made. And hopefully the students end up recognizing (as with any sort of method or tool) the importance of some working community of researchers providing the context in which these things get made, interpreted, and trusted. Images, graphs, and maps are a pointed case of the general issue, just because it’s so easy for them to escape that context when they circulate. (I have a recent general-audience talk about this.)

I agree with Healy’s points. Phil and I once wrote a paper, All Maps of Parameter Estimates are Misleading.

The meta-point here is that people have a (false) intuition that any complicated piece of information can be conveyed in a single plot. One reason I hate the famous Napoleon-in-Russia graph is that it has encouraged this sort of thinking. When trying to make or read a graph, it can be helpful to start by stepping back and acknowledging hat, in general, one single plot (or even two plots) won’t do it all.

The ill effects of ingested lead and other heavy metals had been known since the 1920s, when employees at TEL [tetraethyl lead]-refining plants began hallucinating butterflies and going into convulsions of violent insanity (at least ten died). ‘Smelter nose’, a finger-sized hole in the septum, was an occupational hazard at plants. Horses near the Bunker Hill stack dropped dead; children were hospitalised with kidney damage, forced to undergo excruciating chelation therapy. By the 1970s scientists were beginning to link lead emissions with surging delinquency and crime rates.

The industry’s response was to deny everything or, at best, occasionally raise the height of its smokestacks. Company quacks put out statements asserting that high levels of lead in human bodies were not only harmless but ‘natural’. Thomas Midgley Jr, a General Motors engineer with the diabolic distinction of having invented both leaded gasoline and chlorofluorocarbons, washed his hands in a can of TEL at a press conference, claiming he was ‘not taking any chance whatever’. He knew this to be a lie, having already succumbed to a bout of lead poisoning. (Years later, paralysed with what was said to be polio, he strangled himself in the ropes of a contraption designed to hoist him out of bed.)

In the 1970s, my dad worked for the EPA in their mobile source enforcement division: their job was to stop people from illegally selling leaded gasoline and to adjudicate petitions from mom-and-pop refineries that, for various reasons, wanted exemptions from the new rules on unleaded gasoline.

But that story about Thomas Midgley, Jr.: Wow. What an evil guy. The linked article (a review by James Lasdun of a book by Caroline Fraser) is just full of horrible stories.

I guess that the world is full of evil people and always will be. The challenge is to avoid putting them in positions where they can do a lot of harm.

Sometimes we have to estimate the population distributions. We’ve seen a few examples:

1. “2 flavors of calibration”: Say we have p(X), but we also need p(Z | X), the population distribution of another variable Z. We can estimate p(Z | X, R = 1) using survey data, but nonresponse could make this unreliable. Say we have population data on aggregates p(Z) (e.g. from census tables), then we can logit-shift to anchor to the population aggregate. See Kuriwaki et al. 2024.
2. “MRPW”: Say we have p(X), but we also need p(W | X), the population distribution of the survey weights W. We can estimate p(W | X, R = 1) using survey data. Then because we assume survey weights are proportional to inverse probability of response 1/p(R = 1 | W, X), we can get what we need by Bayes Rule.
3. “weights and MRP for voters”: Say we have p(X), but here we need p(X | V = 1), the distribution of X among the population of voters. By Bayes Rule we can get this via p(X) and p(V = 1 | X). The latter can be estimated from population turnout history and vote intent among survey takers.

In all cases, we have some anchor to the population, e.g. via aggregate totals, survey weights, or turnout history. This brings me to Andrew’s post asking 2016 pollsters to poststratify on party ID. We don’t have population aggregates to logit-shift to. But Andrew comments:

party ID is changing much more slowly than the distribution of vote preference, which itself is changing much more slowly than differential nonresponse.

So in our sample party ID (Z) is changing over time quickly, but mostly due to differential nonresponse:

p(Z | t, R = 1) = p(R = 1| Z, t)/p(R =1 | t) * p(Z | t) = differential nonresponse at t * party ID at t

Andrew cites his coauthored paper Reilly et al. 2001, which fits a model to smooth the poststratifying variable Z over time. This won’t help with the component of differential nonresponse that is constant or slowly changing over time, but it prevents the polls from jumping around with every swing in differential nonresponse.

Andrew also cites his coauthored paper The Mythical Swing Voter, which uses 2008 exit poll data on Z to adjust polls from 2012. This also relies on the assumption that Z changes slowly over time, our anchor in the absence of population data.

We’ve been talking about adjusting for party ID. Another variable we might want to adjust for is interest in politics. In “adjusting for interest in politics” we cite Andrew and Gustavo‘s Challenges in Adjusting a Survey That Overrepresents People Interested in Politics. They see an increase over time in interest in politics, which they say could be nonresponse bias (people more interested in politics taking surveys) and/or a change in the population due to increased political polarization.

I know you’ve posted on the topic more generally but don’t recall if you’ve discussed this in particular. Given the timing in relation to cuts in food assistance, It seems a particularly egregious example of the politicization of data.

The news article, published in late 2025 by an organization called Food Tank (“The Think Tank for Food”) is titled, USDA Ends Key Food Security Report, Leaving Advocates in the Dark, and it begins:

The U.S. Department of Agriculture (USDA) recently announced it will terminate its long-running Household Food Security annual report. The resource is one of the country’s most comprehensive tools for measuring hunger and food insecurity.

The USDA justified the decision as a cost-saving measure, claiming in a statement that the survey is “redundant, costly, politicized, and extraneous.” . . .

Produced for the past three decades by the USDA’s Economic Research Service (ERS), the report offers insights used by researchers, policymakers, and advocates working to reduce food insecurity in the U.S. Anti-hunger advocates argue the move will make it far more difficult to track the impacts of policy changes, including recent cuts to the Supplemental Nutrition Assistance Program (SNAP). . . .

Although advocates are looking for options to fill the research gap, Karen Perry Stillerman, Deputy Director of the Union of Concerned Scientists argues that there are no options that match the scope. “How are the data redundant?” she asks. “The USDA survey serves as the official data source of national food insecurity statistics.”

I followed the link [here’s a version from the Internet Archive] and here’s the USDA’s official statement:

Ummmm, what is this? The Ministry of Propaganda?? Imagine what’s it’s like if you’re a normal person working for the USDA, you just want to do your job, but this is the kind of crap you have to deal with.

But I have a serious question here. The USDA claims that the survey is “redundant, costly, politicized, and extraneous.” Just to go through these:
– I guess the study could be “costly”; to assess whether it’s too costly to be worth it, I guess you’d have to talk about its benefits.
– The study could be “politicized,” but nowadays just about everything is politicized, so that seems kind of irrelevant.
– I can’t see why they are saying the study is “extraneous”; it seems very relevant to USDA-related issues.

But the thing I wanted to focus on here is the claim of redundancy. The USDA says the study is redundant, while the advocates say, no, the data aren’t redundant.

One way for the USDA to address this is to give users help on how to get the equivalent of these data from other sources. I did a quick google and found this page on the USDA’s website:

This seems more serious. I assume the people who are in charge of this webpage have no connection to whoever is the hack who wrote that earlier press release.

So here’s my question. Do the data described on that linked page provide the equivalent information to the now-canceled Household Food Security annual report? Actually, both the Food Tank news article and the USDA press release leave me confused, as they refer both to a “Food Insecurity Survey” and to “Household Food Security Reports.” Does that mean they’re canceling a survey and also canceling a report?

If it’s the survey that’s expensive and redundant, then they could still do the report, no?

The obvious conclusion to be drawn from the ridiculous press release is that the government is canceling the survey and the reports for purely political reasons, some combination of wanting to avoid bad news coming out and ideological opposition to aid to the poor. But it would be good to know if they’re correct in saying this survey is irrelevant.

Maybe I’ll contact Karen Perry Stillerman of the Union of Concerned Scientists and ask what is the basis of her claim that the data are not redundant. Also I can contact the USDA economists on this page. The USDA economists have direct emails; Stillerman doesn’t, but there’s an email given for her media contact, who I guess can connect me to whoever is data-knowledgeable at that organization.

I’ll report back to you!

You want to set your movie in a futuristic New York where every building has a flying car port on top and there are highways through the air and half the people are genetically modified to be 7 feet tall and the sky is red and everyone’s left hand is a phone and all the police are robots? Cool—that will take you all of about ten seconds to get across onscreen.

You want to do all that stuff on the page? You’ve got tougher options. Some writers think they’re going to get away with devoting about ten seconds worth of space to establishing the context, or not mention it at all until it’s way too late. Because it’s never mentioned in the movies, it just is! Other writers try to take the time necessary to let us see, in detail, everything we’d see on the screen in ten seconds, and that turns out to be around thirty pages—because on the page, everything needs to be explained. Readers will want to know exactly what those air highways look like and how they work and maybe even where they came from.

The trouble is, as Makkai says, “what might have been colorful texture onscreen just takes too much of a reader’s energy.”

She continues:

One thing they love to do in Hollywood is save money when possible. You need two people to have a conversation, and a coffee shop set that’s already built and on the lot is a lovely choice. It’s got nothing to do with the story, and it’s meant to be kind of invisible, and that’s great.

I do think that setting is one of our most underutilized tools on the page, and there’s absolutely no reason to limit yourself to the budget or visual language of film. You have, right now, no matter who you are, an unlimited budget. And you might want to do something more interesting with setting, like picking one that puts pressure on characters, or one that presents opportunities for dynamic interaction.

All of this is interesting, and it spurs two thoughts:

1. I’ve read a few of Makkai’s novels and liked them a lot. She’s a very visual writer and a very structural writer. Her plots have structure–perhaps too much structure sometimes, in a way that can seem more constructed than real–and they also have a strong, almost Tolkien-like sense of place. So much so that I’ve wondered whether she prepares architectural drawings or three-dimensional models of her settings. The different rooms and levels of the house in the Hundred-Year House, the Chicago neighborhood in The Great Believers, the buildings and hilly environs of the school in I Have Some Questions for You . . . all these places seemed real, they seemed physical in a way that I don’t usually feel in novels that I read. And I think this sense of physicality really helps the stories, it gives a grounding in the same way that a story can be grounded in character or science or connections to current events.

One way to interpret this is that Makkai has followed her own advice and recognized that, when writing a novel, the physical setting comes at no cost (except for the labor of the writer), and so she makes excellent use of this opportunity. Or we could look at it from the other direction and say that Makkai happens to have a very strong visual and spatial sense, giving her the ability to create vivid, three-dimensional settings in a way that would be much more difficult for most other writers to do.

2. I think it’s much harder to make a living by writing than it used to be. Even setting aside competition from chatbots, reading is no longer a mass entertainment medium. Sure, there are a few bestselling authors of fiction (in today’s newspaper, these are Callie Hart, Nora Roberts, James Patterson, John Grisham, and a few others), but I don’t know that even they sell so many copies, and it’s nothing like past decades, when mass producers like Erle Stanley Gardner and John D. MacDonald sold millions of paperbacks and serious writers from John Updike to Anne Tyler could make a good living too. I get the impression that nowadays even a successful writer needs a full-time job. For example Gary Shteyngart is a Columbia professor, and Makkai herself seems to have multiple teaching positions.

So, if you want to be a serious writer, you need that other job, or else you have to write for TV or the movies, and I guess that would probably be TV. Indeed, the other day I finished Halle Butler’s novel, Banal Nightmare–I absolutely loved it, and my immediate thought was: Uh oh, she’s too good. She’ll get hired to write for TV–fair enough, that will give her a steady paycheck–and be reduced to writing novels as a hobby. Which I think is too bad: 80 years ago or even 40 years ago she could’ve looked forward to a successful career as a novelist, probably not enough to get rich from it (for that you need to get lucky and be in the right place at the right time) but enough to support a comfortable lifestyle as a full-time writer, someone like Peter De Vries. Nowadays, not so much. From a greatest-good-for-the-greatest-number perspective, that’s probably fine–more people watch TV than were ever going to read novels–; but from my perspective as a fan of written literature, I’m sad.

Anyway, I’m reading this long and interesting post by Makkai on the differences between the writing of fiction in books and for screens, and I’m thinking that the main economic function of novels is as low-cost prototypes for movies and TV shows. Literature is a sort of New Haven of the arts. So maybe Makkai’s advice would be good for writers who want to write novels, but then the successful ones will have to change gears once they move into TV writing. I’m not sure, it’s just a thought.

3. To return to Makkai’s original point: I think it’s interesting, the idea that, in a movie, the sets and backgrounds can be expensive, but they can be very quickly conveyed on screen. In contrast, when writing a book, you can construct an elaborate three-dimensional world at a cost of $0. But it can be tricky to explain it to the reader. In her novels, Makkai does a great job at both construction and explanation. I get the sense of there being a detailed three-dimensional world where the characters are moving, and Makkai conveys this through the development of the story in a way that works for the reader.

Lord knows there’s lots of competition from the likes of Soon-Yi “Woody” Allen, Larry “Lawrence” Summers, Nathan “Clippy” Myhrvold, and Columbia’s own Richard “Axel” Foley, but I think this one takes the cake.

In honor of another Epstein associate (see here), I’ll frame it as a “Linda problem”:

Vilayanur is 74 years old, outspoken, and very bright. She majored in neuroscience. As an adult, he was deeply concerned with issues of motor control in stroke victims.

Which is more probable?

1. Vilayanur is a psychology professor who believes in extra sensory perception (ESP)
2. Vilayanur is a psychology professor who believes in ESP and tried to get 3 million bucks from a sex trafficker to fund his lab.

Here’s the story and here’s the evidence:

OK, a dog-bites-man story: over-the-hill professor on the Ted talk cycle seeks return to former glory through the funds of a shady operator.

The interesting part to me is when Ramachandran writes:

In the interest honesty given the focus on such fringe phenomena – theres a 50% chance the whole effort could go up in smoke, but the 50% chance that it COULD lead to something – makes it worthwhile.

I get it that some old guy could think that autistic people have ESP: this kind of mystical thinking was big back in the 1970s when Vilayanur was young, and I think a lot of Boomers have a soft spot for the paranormal. So, sure, why not study the topic–it does no harm.

But to think there’s a “50% chance” that it could lead to something . . . jeez, what an idiot. There’s a big difference between the sane view that doing speculative research on the high-risk, high-reward principle that even if there’s only a 1% chance it comes to something, it could still be worth a shot; and the absolutely insane view that your supernatural study has a 50% chance of being real.

Look, he had tenure, and in any case people have the right to be idiots and not get fired, as long as they do their job well. Ramachandran might well have been an excellent teacher. Or, hey, maybe he was insincere, lying to Epstein in an attempt to scam $3 million from the sex-trafficking financier. But until I hear otherwise, I’ll take Ramachandran’s words at face value and just conclude that he couldn’t think straight about science, I guess in a comparably innumerate way as that physicist who thought that scientific citations were worth $100,000 each. There are a lot of innumerate people out there, and some of them up with tenured academic positions.

It happens. We accept that there are corrupt cops; we also have to accept that there are stupid professors. Some corrupt cops reach heights of political power; similarly, some stupid professors become Ted-talk stars. That’s how it goes.

There’s also the issue, which we’ve discussed before, of which sorts of pseudoscientific beliefs are socially acceptable and which are not. We can laugh or scream at Dr. Oz for promoting astrology (“People that fall under the Aries sign are known for being resourceful, assertive, and headstrong. An Aries can tend to ‘ram or dive in to things head first.’ When an Aries feels blocked, this pent-up energy may appear in the form of migraines, sinus issues, or even jaw tension.”) or out-and-out magic (“You may think magic is make believe but this little bean has scientists saying they’ve found the magic weight loss cure for every body type–it’s green coffee extract.”), but if they believe in the burning bush or the virgin birth or whatever, that’s kind of a different category. Maybe the supernatural stylings of Ramachandran and Oz could be put in the “religion” column and then it would all be cool. I guess the problem is when they try to bring science into the mix rather than just existing on pure belief.

Dr. Oz is currently running the Centers for Medicare & Medicaid Services. That’s a U.S. government agency! I hope he’s not funneling our tax dollars to hucksters promoting “the No. 1 miracle in a bottle to burn your fat,” etc.

The problem here is not the inclusion of no-longer-candidate Platner, as that’s noted in a footnote. Rather, as Ferguson says,

The Times shows Platner outraising Collins; whereas she is millions and millions of dollars ahead, as our charts show.

Here’s the chart that Ferguson shared with us the other day:

If Collins raised $39 million, why did the Times say that Collins only raised $8 million? (They also understated Platner’s total, but by a lot less, counting $13 million instead of $16 million.)

I asked Ferguson how this happened–how did the Times screw up so badly? He replied:

Probably they just used the campaign fund of the candidates. The Super Pacs report elsewhere. You have to look them up. This is normal; we’re clear about that when we did Platner. Republican candidates like Collins are really operating with a pack of funds. That’s the famous coordination discussion, BTW. Now rendered even legally moot by the Supreme Court decision.

Collins has many different vehicles supporting her. Easy to find and not new.
The Times reporters are just lazy; they know about the Super Pacs, but can’t bring themselves to do work. That’s the kind interpretation.

Dayum.

Ultimately, the problem here is no so much with the New York Times–large as they are, they’re just one news organization, and they’re trying to do their best–but with the hollowing-out of the news media more generally.

To put it another way, the problem is not New York Times is not the problem. The problem is that the New York Times is one of the few large independent news organizations out there. If there were lots of other orgs reporting these things, we wouldn’t have to rely on the Times not screwing up.

Ferguson continues:

Contrast the endless articles about Democrats talking in Maine deliberations. The billionaire-tasked Times guy should do some work on Collins.

Cf. our discussion of sources in the first post:

We have used data from the Federal Election Commission to construct similar figures for the much-discussed Maine Senate race. Incumbent Senator Susan Collins is running on the Republican ticket, while Graham Platner is her Democratic challenger. Our totals reckon in contributions from Super Pacs and other outside organizations spending on behalf of either candidates or against one (which we count as spending for the candidate’s opponent).

The note’s a killer, so I’ll copy it here:

Federal Election Commission bulk data downloads are not updated at lightning speed. There is a time lag before individual electronic filings are incorporated into those files. In this case, the bulk data downloads are missing the 12-day Pre Primary Report (12P) (filed before the June 9 primary) and contain contributions to the principal campaign committee up to and including May 20, 2026. The bulk downloads are also missing the independent expenditures spent through election day. We obtained the electronic filings of the candidates’ principal campaign committee and the independent expenditures to fill the gap in the bulk data downloads. We downloaded these electronic filings June 12-14. Collins uses multiple committees to raise and spend money, and these committees have different filing deadlines. The Pine Tree Results PAC filed a 12P and reports contributions up to and including May 20. The Lead Maine Committee has contributions until April 28. The Stronger Maine Super PAC has contributions until March 31, The Collins Victory Committee is March 31, and the Susan Collins for Maine JFC is March 31. Collins also raises money for her principal campaign and leadership committees via joint fundraising committees (JFCs). These are shared accounts that allow several candidates or party committees to raise money together. A single donor writes a “parent” check to the JFC, which then is divided among the participating committees. When Collins is the clear beneficiary of such arrangements, such as with the Collins Victory Committee, we count the full parent check as part of her donor distribution. When Collins is merely one of several candidates involved in the JFC, such as with One Team Senate Majority, we count only the subdivided portion given directly to Collins as part of her donor distribution and not the full parent check. Counting the parent check for committees she controls but only the subdivided check for committees she merely joins lets us credit each donor’s true contribution to Collins exactly once without double-counting the same dollars or absorbing money raised on behalf of other candidates.

So, yeah, they had to do some work.

The above-linked New York Times article sucks for two reasons:

1. It got things way wrong, completely missing the story of the Republican candidate’s massive fundraising edge.

2. It was written in an overconfident style with no indication to the reader that the news story was actually missing more than half of the campaign cash out there.

I’ll forward this to my colleagues at the Times. Maybe they’ll run a correction?

• Dan Ariely (professor at MIT and Duke, Ted talk star, and teller of a story about a possibly nonexistent paper shredder)
• Donald Rubin (professor at Harvard and one of the most influential statisticians of the twentieth century)
• Stephen Hawking (late physicist and culture hero)
• Henry Rosovsky (professor and dean at Harvard; ok, he’s just an economist, but some would count this in the “scientist” cattgory)
• Gerald Edelman (Nobel prizewinning biologist)
• Stuart Pivar (not an academic but a very successful industrial chemist, so, yes, he counts as a scientist for sure)
• Jessica Banks (“an inventor, designer, entrepreneur, and roboticist with degrees in Engineering from MIT and Physics from the University of Michigan”)
• David Gelertner (computer science professor at Yale, most famous as an unfortunate victim of the Unabomber)
• Roger Schank (cognitive psychologist and all-around asshole who, according to Wikipedia, worked at Stanford University, Yale University, Carnegie Mellon University, and Trump University)

The name that was most interesting to me on Azma’s list was V. S. Ramachandran, listed there as “neuroscientist studying music and the brain.” Many years ago I read a book by Ramachandran, “Phantoms in the Brain,” about his research curing the phantom limb syndrome and related topics. It was really inspirational, but I do remember telling a friend about it at the time and he cautioned me that you can’t always believe what you read in a book, that maybe Ramachandran was exaggerating his successes.

Anyway, Azma links to a news article in the school newspaper of the University of California, San Diego, which reports:

Emails released by the Department of Justice indicate that Jeffrey Epstein provided funding for a UC San Diego lab led by Vilayanur Subramanian Ramachandran, director of UCSD’s department of psychology’s Center for Brain and Cognition and emeritus distinguished professor.

The DOJ released more than 3 million additional pages of the Epstein files on Jan. 30, in which Ramachandran is named by Deepak Chopra, a lifestyle guru with ties to UCSD.

Chopra, a former UCSD family medicine and public health clinical professor, first connected Ramachandran’s lab to Epstein. Chopra told CBS News that he helped Epstein with his struggles with insomnia, including directing him to Ramachandran to learn about ongoing brain research.

OK, fine, nothing wrong so far. A colleague pointed Epstein to this guy’s lab.

But then . . . oh! check this out:

On Sept. 25, 2017, Ramachandran replied to Chopra in an email regarding a study the lab was conducting on an “autistic savant who displays telepathy.” Ramachandran wrote that he does not “have problem with [his] lab being funded by Epstein.”

Ramanchandran further wrote that if Chopra’s “pal [Epstein] is serious about setting in motion a lab for the study of extraordinary brain potential … something like 500,000 to 3 million would get the administrators excited.”

A subsequent email from Epstein to his accountant, Richard Kahn, instructed Kahn to send $25,000 from Epstein’s private foundation, Gratitude America Ltd., to the University of California Board of Regents to fund Ramachandran’s research on savant syndrome. He asked it to be mailed to UCSD’s psychology department’s chief administrative officer, Peter Hinkley, who is still in this position.

Chopra and Epstein’s conversation continues through Oct. 5, 2017, when Chopra updated Epstein on spending the day with Ramachandran to discuss the “pilot study of autistic savants,” confirming their relationship.

This combines several Epstein science themes:
– Junk science (“telepathy”)
– Exploitation of vulnerable people (that “autistic savant” who Ramachandran is using as funding bait)
– Greed (“something like 500,000 to 3 million”)
– Epstein being cagey (he only actually gives $25,000)
– The science-media industrial complex (Deepak Chopra)

The only satisfying thing in all of this is seeing these academic bigshots degrade themselves for so little. According to this website, Ramachandran’s salary was a mere $288,557.00 in the year 2020. I’m actually surprised it’s so low, but, looking it up, I see the source of my confusion. He has medical training and I’d imagined he was in the medical school, but he’s actually just in the psychology department, which doesn’t have the budget to pay med-school-level salaries. But even with his meager under-$300K salary, I’m pretty sure Ramachandran could’ve funded the $25,000 out of his own pocket. But, ohhhhh, that greed . . . he wanted millions!

I recently learned from a blog comment that Herman Chernoff passed away last week at the age of 103. He was born the same year as my dad.

I first met Chernoff–it’s not like he was a particularly formal guy, but I can’t imagine calling him “Herman”–when I was a student at MIT. I’d taken a statistics course and really liked it, and I wanted to know what class to take next. The instructor, Stephan Morgenthaler, recommended I ask Chernoff, who in turn told me that MIT didn’t have much to offer in that area so I should take a course at Harvard. Which I did. Then a year later I enrolled in Harvard’s statistics program, and Chernoff had moved there too. That year I signed up for his theoretical statistics course. It pretty much covered what I’d already seen in earlier classes, and it was offered at some early morning hour (8:30, perhaps) so I don’t think I actually attended any lectures after the first week of classes. Chernoff was very mellow about this–he didn’t give me a hard time, and he told me that if I could do the final exam I’d pass the class, so it was no problem. Only in retrospect did I realize it was stupid of me to miss the classes. Chernoff had a penetrating mind, and even discussions of familiar topics–maybe, especially with familiar topics–would have been chances for interesting, open-ended explorations. So, my bad. I made good use of many of my intellectual opportunities at Harvard, but this one I wasted. I was following typical student reasoning, thinking about course requirements and syllabuses rather than of opportunities for deep exploration.

What else can I tell you about Chernoff? I think his most important contribution was his 1954 paper on the distribution of the likelihood ratio, from which the above images are drawn. I thought a lot about these pictures when working with a positivity-constrained model in my Ph.D. thesis:

Related ideas motivated my work on posterior predictive checking, and this remains an area with challenging open questions.

What else? Somebody heard that, when he was a professor at Stanford, Chernoff had been known as “the Ax” because he was so harsh. But by the time he got to Harvard, he was in his sixties and had mellowed. Indeed, he was a nice guy, also a good person to talk to about statistical ideas. He would come to the statistics seminar every week–we would all attend all of them, we had a cohesive intellectual community in that small department. He’d sit in the front row, often he’d fall asleep in the middle, but then he’d invariably wake up at the end and ask a good question. It was cool to have someone around who could offer a thoughtful understanding of just about anything.

A couple decades after that, when I was considering a job at Harvard, Chernoff suggested we buy his house in Brookline–I guess that he and Judy were ready to move to some sort of assisted-care place. That would’ve been kinda cool to have that lineage. When Shaw-Hwa was at Columbia, he lived in what used to be Diana Trilling’s apartment. I guess that means it was Lionel Trilling’s apartment too, but to me Diana is the more interesting writer. Lionel’s always seemed like a sort of Reinhold Niebuhr figure: someone who was written about with a lot of respect in his time but whose writings now seem empty. Like Diana but not Lionel or Reinhold (in my opinion), Chernoff’s writings from the 1950s remain readable and interesting today. Also a banger is 1972 monograph of sequential analysis and optimal design. You wouldn’t go for it to find up-to-date methods, but read it from beginning to end and you’ll get a lot of clean insights.

P.S. For more on Chernoff, see this obituary posted by the Stanford statistics department.

With Bayesian Data Analysis, the big steps forward were:

• Going beyond Bayesian inference to also consider Bayesian model building (as a researcher, you construct the model, it isn’t just given to you as in a textbook), model checking (breaking through the absolutely horrible attitude, common to Bayesians in the early 1990s, that the model was “subjective” and thus should not be checked), and model improvement (continuous model expansion, not the misguided idea of assigning posterior probabilities).
• Going beyond simple conjugate models. BDA had lots of hierarchical models, also lots of computational tools so that you could fit the models you want by putting them together from understandable components. And I like how we had a clear separation between modeling and computing. The model comes first, then you figure out how to compute it. Or you set up a model that works within your computational constraints.
• A Bayesian approach to sampling and causal inference. This was Rubin’s framework in which unobserved units in the population and unobserved causal outcomes are treated as missing data and are part of a joint probability model. We worked this out in chapter 7 of BDA (which became chapter 8 in the third edition of the book).
• Lots of live examples. Not just “real-data examples,” but problems we’d directly worked on. This motivated us and I think it gave our readers a sense of how Bayesian methods worked not just in theory but in applied problems.
• A pragmatic view of probability as a measurable quantity. That’s right there in chapter 1. Bayesian methods are not the product of a philosophical stance; they’re a way to connect models and data using probability.

I could go on and on, but for that I can refer you to the Bayesian Data Analysis book.

And these are the key innovations of Bayesian Workflow:

• Going beyond Bayesian data analysis (model building, inference, model checking, and model expansion) to consider the larger process of statistical modeling, including comparisons of multiple models fit to a single dataset.
• A fuller use of informative priors. This is a big deal. In BDA we still had a bit of the Bayesian cringe going on. One reason we’ve moved toward stronger priors is that the replication crisis has taught us that the amount of prior information available in any given problem is often approximately the same as the information coming from an experiment (see here, for example). Informative priors also fit our increased focus on generative modeling, and we’re doing a lot more prior predictive checking to understand the implications of our models.
• More integration between modeling, data analysis, and computing. One way to see this is that the Bayesian Workflow webpage has the code to run all our examples. We also have lots of code snippets in the text as a way of demonstrating the way in which coding is central to our statistical workflow.
• Lots more live examples. It’s been 30 years since BDA first came out. One reason that Bayesian Workflow has 11 authors is that different collaborators worked on different examples (but the three principal authors read through the entire book, so the general approach should remain coherent).
• Simulation-based experimentation. This is something my colleagues have been doing more and more over the years. At its most basic, simulation-based experimentation provides a best-case baseline for statistical methods: if you can’t recover your quantities of interest with sufficient accuracy under ideal conditions (when your data are simulated from the model you’re fitting), then you know you’re in trouble. And often this is the case! Beyond that, we can simulate from one model and fit another, and see what happens. Simulation experiments aren’t always so easy to construct, as they involve specifying the entire data-generation process. But we think this is effort worth expending, as it involves thinking about the problem you’re working on.

I could go on and on, but for that I can refer you to the Bayesian Workflow book.

And now for the reviews

But you don’t have to trust me on this! Just listen to some of the eminent statisticians and educators who’ve reviewed our book:

Bin Yu (University of California):

An outstanding, protocol-driven guide for Bayesian data analysis, Bayesian Workflow by Gelman, Vehtari, McElreath and co-authors delivers a practical and comprehensive framework for iterative modeling, emphasizing simulation, diagnostic checks, and rigorous empirical validation, and with a long and impressive list of case studies. By treating data analysis as a structured, verifiable workflow, it provides an indispensable toolkit for diagnosing model failures, refining priors, and building reliable data analysis systems for reproducible conclusions, useful for beginning and veteran data analysts alike.

David Spiegelhalter (Cambridge University):

This is not a typical methods textbook, but instead it guides the reader through the whole process of fitting, critiquing and adapting statistical models to real-world problems. It is full of the accumulated wisdom of skilled practitioners, teaching through demonstration rather than theory, with both basic and highly sophisticated examples. I strongly recommend this book to statisticians who really want to understand what they can learn from their data.

Brad Efron (Stanford University):

A bravura performance…Gelman, Vehtari, McElreath and friends develop in detail a practical Bayesian data analysis workflow, from acquisition to final report, including full computational guidance.

Christian Robert (Université Paris Dauphine):

This original, thought-provoking, and transformative book is much much more than an implementation manual for Bayesian Data Analysis, even though it shares almost the same perspective. (The first sentence of the book states that the authors’ “conceptions of statistical practice, and of Bayesian statistics, have changed over the years”.) By providing a modus vivendi for undertaking Bayesian modelling from scratch in realistic settings where models are not magicked out of the blue, the authors explicit and rationalise the many steps required by such a bottom-up modelling protocol (“not a checklist, not a cookbook”, and not a flowchart!) in real situations. The contents read very well and very smoothly, with a seamless conjunction of intuition, modelling advices, computational details, and comparison tools. While unsurprisingly Bayesian, the perspective adopted therein remains both open and inclusive, with a welcome humility about the limitations and challenges of Bayesian workflows. This book should thus appeal to and profit a wide variety of readers, as providing guidance through an extensive collection of highly detailed examples, with shared code and exercises.

Rohan Alexander (University of Toronto):

Some statistics books show you how to beat an egg, others are recipe books: if this, then that style. This book teaches you how to cook. Written by authors who established so much of how we do Bayesian statistics, this new book is an indispensable guide for analyzing data in a trustworthy way. It walks you through the actual steps involved in building models to explore and understand datasets. Part 4 is particularly excellent – the authors provide many end-to-end case studies that will be useful for both practitioners and students. It highlights the value of their workflow-based approach. Filled with chatty asides, the book introduces the Bayesian workflow to a broad audience. It embraces the frustrations and complexities of actually doing Bayesian statistics and provides specific guidance throughout. Each chapter contains exercises and it could be the basis of an upper-year undergraduate course, or a first-year grad course, in applied statistics. It will be used for many years to come.

Mine Doğucu (Harvard University):

What makes Bayesian Workflow so exceptional is how it seamlessly pairs profound ideas about modeling with the adoption of modern computational practice. By centering the messy, iterative process of modeling through real-world case studies, the authors reject rigid cookbooks and checklists in favor of building deep situational awareness. Because the ideas are so clearly articulated and deeply applied, this book serves as an invaluable pedagogical resource. With its practical exercises, individual chapters or the text as a whole can easily be integrated into upper-level undergraduate or graduate courses, while also remaining accessible for self-guided readers. It is an indispensable read for anyone with foundational knowledge in Bayesian methods, regardless of whether they are applied practitioners, software developers, or methodologists.

You might also be interested in the journal issue on statistical workflow that we recently edited for the Philosophical Transactions of the Royal Society.

Again, here’s Bayesian Workflow on Amazon, here’s the publisher’s website, and here’s our website with data, code, and lots more.

Enjoy.

The implosion of Graham Platner’s Senate campaign in Maine has upended a marquee Senate race, leaving the state Democratic party just a few weeks to choose a substitute nominee. A planned nominating convention on July 25^th has drawn considerable candidate interest. But the mathematical properties of ranked choice voting add a strange wrinkle to these deliberations.

The Maine Democratic Gubernatorial Primary

Three of the top contenders to replace Platner are former gubernatorial candidates: Nirav Shah, former director of the Maine Center for Disease Control and Prevention; Troy Jackson, former Maine State Senate president; and Shenna Bellows, Maine’s secretary of state. All three ran for the Democratic nomination for Governor, losing the primary to Hannah Pingree, former speaker of the Maine State House.

June’s primary results are given below. (Data from Wikipedia.) Maine uses ranked choice voting (RCV) in primaries and federal elections, so voters could rank up to six choices for Governor. Using the instant runoff algorithm, candidates were sequentially dropped based on who had the fewest first-choice votes, and ballots were reallocated to each voter’s next-ranked choice. Jackson, Shah, Bellows, and Pingree were highly competitive, each receiving between 20% and 27% of first-choice votes. Of the four, Bellows was eliminated first, then Jackson. Shah fell to Pingree in the final tabulation round.

Candidate	Round 1	Round 2	Round 3	Round 4
Pingree	50,552 (23%)	55,360 (26%)	75,671 (36%)	111,750 (56%)
Shah	58,606 (27%)	62,860 (30%)	72,681 (35%)	86,950 (44%)
Jackson	45,959 (21%)	47,597 (22%)	60,010 (29%)	Eliminated
Bellows	44,770 (21%)	47,049 (22%)	Eliminated	—
King III	17,860 (8%)	Eliminated	—	—
Exhausted ballots	—	4,881 (2%)	9,385 (4%)	19,047 (9%)
Continuing ballots	217,747	212,866	208,362	198,700

These results have taken on extra significance as the state party seeks democratic buy-in for the selection of a substitute Senate nominee. Media outlets, for example, have routinely referred to Shah as the “runner-up” in the Governor primary. But analyses of the individual ballots cast in the primary reveal a surprising mathematical fact: though she was eliminated before them, Bellows would have defeated either Shah  or Jackson in one-on-one elections.

Mathematical Details

This unintuitive fact is a generalization of a well-known feature of ranked choice voting elections: it does not satisfy the Condorcet winner criterion.

First, some definitions. Suppose we have an election with a set of candidates C:

• A “Condorcet winner” is a candidate in C who would defeat all the other candidates in a head-to-head election. A Condorcet winner need not exist for any given C; think of rock-paper-scissors, where each option wins against one alternative and loses against the other. But Condorcet winners exist in many standard election settings.
• The Condorcet criterion is a feature of electoral methods: a method satisfies the criterion if it always selects a Condorcet winner when one exists.

Standard plurality elections – in which voters make one selection, and whomever gets the most votes wins – do not obey the Condorcet criterion. This is well-understood due to the “spoiler effect.” For example, a Libertarian candidate may attract voters who would otherwise prefer a Republican to a Democrat, siphoning enough voters such that a Democrat obtains the most votes.

Because voters express richer preferences in RCV elections, the method is considered better at identifying Condorcet winners. But it can easily be shown that RCV also does not satisfy the Condorcet criterion. This is not purely hypothetical. In a 2022 U.S. House special election in Alaska, Democrat Mary Peltola was elected against two Republican opponents: Sarah Palin and Nick Begich III. An analysis of the underlying ballot data revealed that Begich was a Condorcet winner. But he was eliminated in the first round because he received slightly fewer first-choice votes than Palin, allowing Palin to advance and lose to Peltola.

As RCV does not obey the Condorcet criterion, it stands to reason that the order of elimination need not correspond to who would win head-to-head elections. This is indeed true. A candidate eliminated in an earlier round may well have defeated one eliminated in a later round in a head-to-head election.

Results in Maine

We can understand the electorate’s preferences in Maine because the state releases its cast vote record: the anonymized set of rankings for every ballot cast. These data are available online and have also been analyzed extensively by the election advocacy group FairVote.

To assess how two candidates A and B would fare in a head-to-head election, we look at the set of ballots that rank at least one of them. Any ballot in which A appears before B, or A is ranked and B is not, represents a voter who prefers A to B; any ballot in which B appears before A, or B is ranked and A is not, represents a voter who prefers B to A.

In the table below, we summarize all the head-to-head matchups among the top four candidates. Note that if the final column is positive, then A defeats B; if it is negative, B defeats A.

Candidate A	Candidate B	% of Ballots Listing Neither	% Who Prefer A	% Who Prefer B	A vs. B Margin
Bellows	Pingree	14%	41%	44%	–3%
Bellows	Jackson	19%	48%	33%	15%
Bellows	Shah	12%	45%	43%	3%
Shah	Pingree	10%	39%	50%	–11%
Shah	Jackson	13%	50%	37%	12%
Jackson	Pingree	14%	33%	52%	–19%

Pingree wins all three of her matchups, indicating she was indeed the Condorcet winner. But notably, Bellows wins every matchup except the one against Pingree. She was preferred to Jackson on 48% of ballots while he was preferred on 33%, with the remaining ballots listing neither candidate. Bellows had a narrower margin against Shah, but she was preferred on 45% of ballots to his 43%.

These results reflect the strengths and pitfalls of RCV. Because voters’ ranked choices are recorded, we can better assess the electorate’s head-to-head preferences among many candidates. But elimination orders under instant runoff needn’t reflect these preferences. In closely contested elections like the Maine Democratic gubernatorial primary, this can yield unintuitive results – with big implications for the next big question: whom to choose as a substitute Senate nominee.

Following up on Rosenman’s analysis, I have a few points to raise:

1. Why should I care who would win in a head-to-head race? I’m not trying to ask this in an aggressive way; it’s just not clear to me why this should be the question to ask, or why we should care about a Condorcet winner. Another way to say this is that intensity of preference could matter too.
2. A related issue is that there are lots of people who could potentially be qualified to be the senator from Maine–after all, a senator doesn’t really have to do much, their staff does all the work, right? Just ask Senator Grassley from Iowa! My point here is not to trivialize the election–people live or die based on who is elected to Congress–just that the steps of choosing a candidate involve a winnowing from many many possible choices. The Condorcet winner criterion and other similar rules apply only after drastically limiting the number of options.  From that perspective, I’d be more inclined to rate candidates based on a summing of pluses and minuses for various attributes, rather than head-to-head comparisons. I get that the general election is a head-to-head race so you need to think about such things, but from a political theory perspective, or from a which candidate-to-choose perspective, I see this Condorcet thing as a blind alley.
3. Who you’d want to run for governor isn’t necessarily the same as who you’d want to run for senator.  I say this for two reasons.  First, they’re different jobs:  what it takes to run the executive branch of a state is different than what it takes to be a member of the national legislature.  Second, the main goal of a political party is to win the election, and it could take different things to win in the two races in Maine this year.  I don’t know how important this is, as I have no sense of politics in that state. I’m just raising the issue.

Recall the multinomial logit model that Train (2009) Chapter 7 calls the exploded logit:

P[ranking Other then Left then Right] = exp(f_Other) / sum_c’ exp(f_c’)   *   exp(f_Left) / (exp(f_Left) + exp(f_Right))

Without covariates, it has only 3 parameters: f_Other, f_Left, f_Right. It makes the independence from irrelevant alternatives (IIA) assumption to go from these 3 parameters to rank probabilities.

In contrast, the multinomial model in Rosenman & Liang 2026 does not make the IIA assumption and has 14 parameters, one for each of 15 possible rankings minus one so they sum to 1:

P[ranking Other then Left then Right] = pi_{Other, Left, Right}

Rosenman & Liang 2026 note that in RCV the election outcome is not expressable as one parameter. Instead, the winner is determined by instant runoff:

1. If a candidate wins >50% of first choice votes, they win.
2. Otherwise, the candidate with the least first choice votes is eliminated, and each ballot counts for its top remaining choice. Return to step 1.

Say you use polling data to estimate rank probabilities pi_j for each ranking j. These estimates differ from the true probabilities due to many sources of error (see our favorite Figure 2.5 from Groves et al. shown in quantity vs quality and is a mismeasured X better than none at all ?). Rosenman & Liang 2026 focus on sampling error.

How can we propagate uncertainty about the rank probabilities pi_j to uncertainty about the RCV winner ? If you have draws from the posterior of pi_j, you can do instant runoff on each to get a winner for that draw. This gives win probabilities according to your model and data.

To see the importance of uncertainty in RCV, let’s look at their 2022 Alaska House special election example. With 3 candidates, RCV is determined by 5 margins (see their Lemma 1). Most of these margins are well-identified by the data, but 2 were quite close: Palin vs Begich first choice margin and Peltola vs Palin pairwise margin. They plot these 2 margins in the right panel of Figure 1. The true outcome is the black dot, with sampling uncertainty shown as ellipses around it. For small sample sizes (the biggest ellipse), we see that a plurality of the mass falls into green, where point estimates would declare that Begich wins. Uncertainty quantification would help put this in context, giving all candidates win probabilities around 20-40%, showing the race is difficult to call with such small data.

For details, see Rosenman & Liang 2026.

Back in the day, Bayes was seen alternately as Jesus or the Devil, and there were hopes of a grand synthesis of subjective probability and local information in markets, a connection between formal statistical inference and individual decision making.

In retrospect, the cognitive science of the 1950s wasn’t all there, and hierarchical modeling hadn’t been integrated into Bayesian inference. Also some key pieces such as posterior predictive checking and general-purpose Bayesian computing weren’t there. So any attempts at unification were premature. Not that it was a bad idea to try! Much is learned from incomplete efforts. It’s just clear in retrospect that any unified theories of the time were bound to fail.

From the perspective of seventy years later, we can see Bayesian inference as a useful part of the statistical toolkit, a way to place regularization (a central part of all modern machine learning) in the context of scientific modeling. Many problems that can be solved with an entirely Bayesian approach, and others can be viewed as approximate Bayes–or, to put it another way, Bayesian ideas can help with all sorts of statistical modeling problems, even when other inferential methods are used.

What I’m saying is, it’s a good idea for everyone doing statistics or machine learning to understand the basics of Bayesian inference and computation, prior and predictive checking, and Bayesian model expansion, for their own sake and also as a way to make sense of statistical learning.

“Making Statistics Work: Information Theory and Bayesian Inference” is one of the most unusual statistics books I’ve ever read. I don’t agree with much of it–for example, right on the second page they start talking about “prior beliefs,” which isn’t how I think of things at all (see here and here)–and it’s written in a mathematical style which seems old-fashioned to me but has a kind of charm. You could almost say that it’s the statistics book that William Feller would’ve written had he been converted to Bayesianism.

What I really like is that the book is what it is–an forthright attempt at a modern expression of the aimed 1950s synthesis of mathematical statistics, physics, and economics. It clocks in at a crisp 300 pages. It has zero overlap with Bayesian Data Analysis and Bayesian Workflow, and that’s just fine. As I said, I don’t really buy their synthesis myself, but I respect their attempt. You can judge it as you will.

I’m reaching out to introduce the Voter Impact Index, a new data tool from PowerMoves that assigns every U.S. zip code a voter impact score based on the recent competitiveness of six federal and state elections tied to that location.

The Index may be useful in your teaching or research in a few concrete ways:

— Classroom discussions on political geography, voter mobilization, and the relationship between where people live and how much their votes matter
— Research applications exploring electoral competitiveness, voter sorting, and the civic behavior of movers (we estimate 15 million registered voters relocate annually)
— Student projects analyzing zip-code-level electoral data across districts

The underlying data, code, and methodology are fully open and accessible via GitHub through our website at PowerMoves.Vote — making it straightforward to build on or replicate.

PowerMoves is a nonpartisan project. The Index draws from trusted nonpartisan sources and assigns scores regardless of party affiliation.

I was curious so I looked up my own zip code, and here’s what came up:

A “medium” voter impact of 44/100. Are you kidding? Yes, you can get lower impact scores (just try typing in 02139), but something close to the midpoint on a 0-100 scale doesn’t sound right to me. We almost never have close elections. New York is not a swing state, and even our local elections are never close.

OK, the 2022 governor’s election in NY was pretty close, I’ll grant them that, and the 1994 race was even closer, as were 1982 and 1978 . . . but that’s going back pretty far, and they’re only weighting the governor races at 15% (go here and scroll to the bottom), so I was puzzled as to how voters in our district can be judged to an impact of 44 on a 0-100 scale. Even if you count the governor’s election as close (and it wasn’t that close), that would still only you to 18.

If you read through that document carefully, you can figure out what’s going on:

OK, there’s this weird bit about dividing by 2 or 3, but that’s not the key issue. The big problem, I think, is linearity. For example, in the 2024 presidential race, Kamala Harris won the two-party in New York by a 13-point margin. Not close at all! Really not close, considering that, had the state election been close, there’s no way that New York’s electoral votes would’ve been decisive. My voter impact for this election was approximately zero (see some calculations here, albeit from an earlier year). If you want to get technical about it, the probability my vote is decisive is something like 1/100 of the probability that a swing state’s voter will be decisive.

So if the “presidential election” contribution to this index is 100 for Wisconsin, Michigan, and Pennsylvania, and something like 50 in a state like North Carolina or Georgia, then it should be approximately 1 in New York. Or maybe 0.1. Or maybe 2. In any case, some tiny number. Even the governor’s race, which Hochul won by 6 percentage points . . . ok, that’s close, but, again, there are closer races for governor. I went online and looked it up, and there were a couple races decided by less than 1 percentage point of the vote. If those tossups count as a voter impact as 100, then maybe the New York race would be a 50? or maybe something less than that?

So if you add all up all these voter impact score and weight them, you might get something like a 10 for my district, if you’re being generous. Not 44.

It’s an interesting example. At first, doing this linear scaling could seem to make sense. But not if your goal is to measure voter impact.

To put it another way, their measure is underestimating the value of voting in a swing state or a swing district. The linear mapping smooths out the signal.

P.S. I replied to the above email to share my concern with the creators of this index. We had a cordial email exchange but ultimately they didn’t seem convinced by my argument and so they left the index as it is. Too bad. But, hey, they’re doing the work, it’s their call: if they want to categorize my zip code as having “median” voter impact . . . well, it’s a free country!

Some archaeologists have measured house sizes for Gini-coefficient-style studies aside from studying human remains to measure nutrition and rates of illness. I think that was what Michael E. Smith meant when he talked about hypothetical data: “archaeology can’t give social scientists population or GDP, but here are some things we can measure that might be useful for social science.”

I asked Manning where the quote came from, and he replied:

I think I got the idea from this response by Smith to a published paper:

This model of inequality in the Aztec Empire is not based on empirical data. While there is nothing wrong with hypothetical models per se, the paper is phrased as if it presents empirical findings. … There are simply not enough data available to do the kind of analysis presented in this paper. The tweaking of data and methods do not produce results that satisfy me as being reasonable estimates of the level of inequality in the Aztec Empire. Perhaps this is just an epistemological difference between our approaches to science and knowledge. Economists might look at this paper as a fine analysis, whereas archaeologists and historians will probably look at it as a study based on hypothetical data, and therefore divorced from the Aztec reality that we study.

Smith has a book that talks about the archaeology of inequality in Aztec Mexico: Timothy A. Kohler and Michael E. Smith, editors, Ten Thousand Years of Inequality: The Archaeology of Wealth Differences (University of Arizona Press, 2019).

Often in social science there is tension between what we can measure and what we would like to know.

Alley and colleagues draw on mediation analyses of longitudinal data from Millennium Cohort Study to argue that their findings “highlight the critically important role that childhood maternal warmth plays in shaping mental and physical health into late adolescence” (p. 716), and “suggest public health interventions aimed at increasing maternal warmth “may be particularly effective in positively impacting adolescent health” (p. 714).

Although the article is dense with tabularized information about key study variables, readers will search in vain for evidence to justify the authors’ conclusions and recommendations related to maternal warmth. What they will find instead are minuscule direct and mediated path coefficients (betas) linking maternal warmth to adolescent outcomes that amount to uninterpretable and unactionable statistical dust. The authors tell us as much in their seductively (if unintentionally) misleading statement that “Social safety at 14 years of age mediated 20% to 100% of the effect of early maternal warmth on physical health, psychological distress, and psychiatric problems at 17 years of age (b = 0.01-0.15; P < .001 for all)” (p. 709). A more straightforward, precise, and informative description of this finding is that social safety at age 14 mediated 20% of maternal warmth’s .01 effect on physical health, 60% of its .01 effect on psychiatric problems, and 100% of its .15 effect on psychological distress at age 17, for a total indirect effect of maternal warmth on subsequent outcome measures of less than 1%. The interpretability of these findings is further compromised by the extreme distributional skew of the 3-item social safety schema latent variable, with the vast majority of adolescents reporting that they had family and friends who helped them feel safe, secure, and happy (86%), someone they could turn to with problems (79%), and someone to whom they felt close (89%). The maternal warmth and harsh parenting measures are also marked by extreme distributional skew, with trained observers reporting that 86% of the mothers exhibited all 5 maternal warmth behaviors and 91% exhibited no harsh parenting behavior during naturalistic play with their children.

The authors do themselves and the readership of JAMA Psychiatry a great disservice by slipping through the normative scientific membrane and conflating statistical with theoretical and practical significance. This is an especially troubling breach within the context of contemporary concerns and public skepticism about the reliability and credibility of social and behavioral sciences research.

References

1. Alley J, Tsomokos DI, Mengelkoch S, Slavich GM. Childhood maternal warmth, social safety schemas, and adolescent mental and physical health. JAMA Psychiatry.
2025;82(7):709-717. doi:10.1001/jamapsychiatry.2025.0815

2. Bogdan PC. One decade into the replication crisis, how have psychological results changed? Adv Methods Pract Psychol Sci. 2025;8(2):25152459251323480.
doi:10.1177/25152459251323480

3. Murray EJ, Swanson SA. Causal inference in observational psychiatry: What do we need to know? JAMA Psychiatry. 2023;80(6):539-540.
doi:10.1001/jamapsychiatry.2023.0343

4. Richters JE. Incredible utility: The lost causes and causal debris of psychological science. Basic Appl Soc Psychol. 2021;43(6):366-405.
doi:10.1080/01973533.2021.1994229

Ahhhh, JAMA!

P.S. JAMA’s not all bad. My colleagues and I recently published a short paper there! Just about all journals are a mix of good and bad.

I’ll put the full message, with its 18 signatories, below the fold.

My reaction to all this is that it would be great if the journal could move to an open and free system such as is done by the Journal of Machine Learning Research–a journal that I believe was founded by people who had resigned from the editorial board of a commercial journal.

Even commercial journals that begin with good intentions can develop fatal problems. For example, check out the sad story of the Berkeley Electronic Press, a set of commercial journals that was founded by a friend of mine. My friend’s an economist, and I guess he might say that it was the iron logic of capitalism that reduced a once-noble endeavor to a rent-seeking enterprise.

So, yeah, I’d recommend that Statistics and Computing follow the path of JMLR, really try to imitate its structure as closely as possible. Bayesian Analysis is another free journal that appears to run with minimal overhead.

It kind of bugs me, though. Profit-making companies have done great things in publishing and communication. A quick glance at our shelves reveals lots of wonderful books, almost all of which were published privately. As were Bayesian Data Analysis, Regression and Other Stories, Active Statistics, and the rest of my books. Lots of great movies are made for money too.

On the other hand, Arxiv is nonprofit, as is lots of the web, on which I post all my published and unpublished papers. Wikipedia is nonprofit, and this blog is written using WordPress, which appears to be another nonprofit organization. I teach at Columbia University, which is private but nonprofit, not run perfectly by any means but still going strong.

Scholarly publishing is a funny industry because it’s my vague impression that it started out as a low-budget noncommercial enterprise, and then some private rent-seekers moved in. I guess these companies were doing something special or they wouldn’t have been so successful at taking over, but now it seems to have gone too far. More sites like Arxiv, JMLR, and Bayesian Analysis would be a good thing. Right now we’re always having to figure out where to publish our papers; it’s just an absolute mess. These journal submission websites make the Department of Motor Vehicles office look like a lean machine by comparison.

P.S. Retraction Watch ran a story on this, where they quoted Robin Ryder as saying, “If the editors regroup elsewhere to form a new journal, they hope to publish with a society, Ryder told us, citing journals like Journal of the Royal Statistical Society and Annals of Statistics, ‘none of which force authors to pay APCs.'”

I think it would be a mistake for them to follow the Journal of the Royal Statistical Society and Annals of Statistics. Both these journals have arduous paperwork-laden submission processes and both charge for access.

If you’re going to start over, why not follow the model of JMLR and Bayesian Analysis and make it all free? Cut out the middleman entirely!

P.P.S. Here’s the full text of the letter from the 14 members of the editorial board:

We are writing to resign as Associate Editors of Statistics and Computing effective 31 December 2026, due to Springer Nature’s recent decision to impose Article Processing Charges (APCs) upon all authors.

We have the greatest admiration for your hard work for this journal and for the scientific community: under your stewardship, and that of previous Editors, the journal has published many articles of extremely high quality and it is a leading journal in our field. It has been an honour to play a small part in this during our time as Associate Editors, and we resign with great regret.

We understand that the decision to introduce APCs was not your own but was imposed by corporate management, and we are sorry that our resignation will put you in a difficult position. Nonetheless, the APCs that Springer Nature will introduce on 1st January 2027 are irreconcilable with our vision of science. They are not compatible with the goal of publishing the best science, whoever the authors; they exclude vast swathes of the scientific community; they are not a responsible use of the taxpayers’ money that funds our research; they are also at odds with the standards in Statistics.

Our decision to resign due to this change should not be taken as support for the existing system. The current academic publishing system is built upon vast quantities of unpaid labour and establishes a financial paywall to view the final research articles. Funding bodies are quite rightly pushing back against this, requiring that publicly funded research be freely accessible. From this perspective the move to “open-access” at Statistics and Computing might seem to have some merit on the surface. However, achieving this through APCs simply moves the financial barrier to a different part of the system. The exploitation still remains, and now Statistics and Computing will no longer publish the best science, both due to financial exclusion of those researchers who cannot afford to pay, and those community-minded researchers who refuse to pay on principle.

We would be very supportive of creating a new, genuinely open journal instead, possibly under the auspices of a learned society. We would welcome the opportunity to work with other members of the editorial board to create such a journal, and would be glad to submit future work there. We hope to have conversations with the editorial board and the community about such a move.

We hope to remain in contact with you. Thank you for your confidence,

Pierre Alquier (ESSEC)
Julyan Arbel (Inria Grenoble)
Louis Aslett (Durham University)
Joshua Bon (Adelaide University)
Alice Cleynen (CNRS)
Adrien Corenflos (University of Warwick)
Francesca Romana Crucinio (University of Turin)
Kamélia Daudel (ESSEC)
Ritabrata Dutta (University of Warwick)
Mathieu Gerber (University of Bristol)
Sahani Pathiraja (University of New South Wales)
François Portier (CREST-ENSAI)
Sam Power (University of Bristol)
Robin Ryder (Imperial College London)
Leah South (Queensland University of Technology)
Scott Sisson (University of New South Wales, Sydney)
David Warne (Queensland University of Technology)
Olivier Zahm (Inria Grenoble)

A Canadian journal has issued corrections on 138 case reports it published over the last 25 years to add a disclaimer: The cases described are fictional.

Paediatrics & Child Health, the journal of the Canadian Paediatric Society, has published the cases since 2000 in articles for a series for its Canadian Paediatric Surveillance Program. The articles usually start with a case description followed by “learning points” that include statistics, clinical observations and data from CPSP. The peer-reviewed articles don’t state anywhere the cases described are fictional.

Wha???

Here’s how it came out:

The corrections come following a January article in New Yorker magazine that mentioned one of the reports — “Baby boy blue,” a case published in 2010 describing an infant who showed signs of opioid exposure via breast milk while his mother was taking acetaminophen with codeine. The New Yorker article made public an admission by one of the coauthors that the case was made up. . . .

The move came as a surprise to David Juurlink, professor of medicine and pediatrics at the University of Toronto, who has spent over a decade looking into the claim that infants can receive a meaningful or even lethal dose of opioids via breast milk when their mothers take acetaminophen with codeine. The first such case, published in the Lancet in 2006 by pharmacologist Gideon Koren, was the centerpiece of the New Yorker article. . . .

The Baby boy blue case is “the only such case study, aside from the Lancet case report and the two now-retracted descriptions of the same case in Canadian Family Physician and Canadian Pharmacists Journal,” Juurlink said. “It is the most compelling published description of neonatal opioid toxicity from breastfeeding. And it is wrong.”

And here’s some background:

While the instructions for authors for Paediatrics & Child Health has at times indicated the case reports are fictional, that disclosure has never appeared on the journal articles themselves. . . .

The versions on PubMed Central also do not bear any indication the case reports are fictional.

The surveillance highlights “are intended for paediatric health care providers or physicians in training, and include learning points that briefly translate and disseminate knowledge about the disease or condition,” Elizabeth Moreau, a spokesperson for the Canadian Paediatric Society, told us by email.

To protect confidentiality

The article continues:

The journal decided when it first started publishing the article type “that the cases should be fictional to protect patient confidentiality,” Robinson [editor-in-chief of Paediatrics & Child Health] told us. “Apart from the case that led to the recent New Yorker article, all or almost all were cases of very well recognized conditions (such as congenital syphilis, fetal alcohol syndrome, serious trauma from ATVs, hepatitis C infection) where a single case report would not generate any interest or ever be cited.”

But:

Neither the instructions for authors from 2010 — when Koren and his coauthor Michael Rieder would have written their article — nor the linked list of article types — state the cases are fictionalized, or fictional. A set of instructions dated 2015, and linked from the journal’s author guidelines, indicate the “clinical vignette” should “describe a fictional case.” . . .

In the case of Baby boy blue, “the article was structured as an authentic clinical case, indexed as such, and cited as an actual clinical observation. Readers had no way of knowing it was fictional,” [Juurlink] said. “A narrative that is fictional but published in the format of a genuine case report, without disclosure at the time of publication, is functionally indistinguishable from fabrication in the scientific record.”

I agree with Juurlink on this one. But I also want to make another point, something that Thomas Basbøll and I wrote about in another context.

Setting aside the moral questions involved in presenting fiction as if were fact, and setting aside the specific calamities that arose from the false and, it seems, medically implausible “Baby boy blue” story, I think there’s a bigger problem with these fabricated case studies–even if they are labeled as fictional.

The problem is that, if you make up a case study, you can make it fit your story. A real case study is constrained by reality–and that’s a good thing.

I think it’s a bad idea for the journal to use made-up stories. By using made-up stories, you’re losing a crucial opportunity to learn from clinical judgment.

If you want to change names or circumstances to preserve anonymity, fine. You can still be constrained by what happened in the real case. Making it up from scratch, though, that’s no good. It’s the equivalent of plotting the fitted model without showing any of the data.

I have a question for you–and, if you think it worthwhile, for your readers.

A few weeks ago, I was deposed by Harvard’s lawyers in the lawsuit between Francesca Gino and Harvard. Much of the questioning focused on my replications of research by Harvard Business School professor George Serafeim and my allegations of research misconduct against him and his coauthors.

That experience has led to a lively online debate about two questions:
1. Is fabricating data worse than fabricating results?
2. Is failing to correct a known false report more or less serious than making the false report in the first place?

At the moment, my own thinking is this:
1. Both fabricating data and fabricating results mislead readers. They are simply different paths to the same outcome and thus similarly serious.
2. Failing to correct a false report–once the authors know it is false and material–may actually be more serious. It suggests a conscious decision to leave readers with a claim the authors know to be unsupported.

Your ladder of responses to criticism also seems relevant here, especially categories 6 and 7.

Interesting. This has come up in the past, discussing the moral culpability of researchers who make errors and then avoid acknowledging them. For example this guy at the London School of Economics and Political Science, or this guy at the University of Chicago, or, of course, this guy at the University of California. I don’t think that the first two of those people did any direct research misconduct, but they made major research errors that they never acknowledged–they keep pointing to their discredited work without any note of the problems–and, yeah, that seems like misconduct to me.

Here’s another story for ya. Years ago I had a colleague who showed me a paper he’d just written. It read the paper and realized it had a fatal flaw–not a calculation error, but a misapplication or misunderstanding of a statistical model. I won’t go into the details here; what’s relevant to the story right now is that the paper in question had been accepted by the journal but it had not yet been scheduled for publication. This was before the era of online anything, so the paper really was still in process. I told me colleague he was lucky: he could withdraw the paper and spare himself embarrassment. (The error in the analysis was central to the result in the paper; if you got rid of the error, there was nothing to salvage, so it’s not like he could just send in a corrected version.) To my dismay, my colleague replied, No, the paper is accepted, I don’t want to lose a publication. I asked, Doesn’t it bother you to have them publish something that’s wrong?, and he replied something about the literature being self-correcting. I don’t remember the details of this conversation from decades ago, but I do remember the horrible feeling. I thought about contacting the journal to tell them not to publish, but I figured that ultimately it was their problem for accepting it.

1. New weighting variable: support score = E(2024 vote | other X variables).
2. New weighting method: energy balancing (Huling & Mak, 2024)

Ben Schneider helpfully blogged about energy balancing as well:

Raking and similar calibration methods are based on balancing means or totals for specific variables…The energy balancing method does something different: it calibrates based on an entire multivariate distribution, as measured by an empirical cumulative distribution function (ECDF).

Jared Huling (of Huling & Mak, 2024) helpfully answered questions in the comments. I’m still puzzling over how energy balancing handles empty cells (unsampled regions of the joint covariate space). I need a toy example.

Consider 2 binary variables, so 4 population cells, with known population shares:

       k=0    k=1    total
j=0    .4     .2     .6
j=1    .2     .2     .4
total  .6     .4

Say the sample is missing folks in cell 11:

       k=0    k=1    total
j=0    .5     .3     .8
j=1    .2     0      .2
total  .7     .3

Consider 4 methods:

1. Classical Poststratification: not defined because of division by 0.

2. Raking: match only the margins. Correct when Y | X1, X2 is additive.

       k=0    k=1    total
j=0    .2     .4     .6
j=1    .4     0      .4
total  .6     .4

3. Energy balancing: minimize the Energy-Distance(F_w, F_pop) between the weighted sample distribution of X1, X2 and the population distribution. Correct when Y | X1, X2 is such that nearby cells have similar means.

Say X1 = young/old, X2 = man/woman, Y = percent Democrats, and no old women are sampled.

Raking is correct when additivity holds: old women = young women + (old men − young men)

Energy balancing is correct approximately when: old women = (old men + young women)/2 ?

library(WeightIt)

pop  <- data.frame(X1 = rep(c(0, 0, 1, 1), c(40, 20, 20, 20)),
                   X2 = rep(c(0, 1, 0, 1), c(40, 20, 20, 20)))

samp <- data.frame(X1 = rep(c(0, 0, 1), c(50, 30, 20)),
                   X2 = rep(c(0, 1, 0), c(50, 30, 20)))

dat <- rbind(cbind(pop,  A = 1),
             cbind(samp, A = 0))

W <- weightit(A ~ X1 + X2, data = dat, method = "energy",
              estimand = "ATT", focal = "1",
              dist.mat = as.matrix(dist(dat[, c("X1", "X2")])))

w <- W$weights[dat$A == 0]
tapply(w, interaction(samp$X1, samp$X2), sum) / sum(w)

       k=0    k=1    total
j=0    .381   .309   .69
j=1    .309   0      .309
total  .69    .309

4. MRP: fit a model for Y | X1, X2. The interaction term’s posterior equals its prior, propagating uncertainty around additivity.

Am I understanding this correctly ?

The sausage

So as not to bury the lead (or “lede” if you want a mid-20th-century newspaper vibe), check out the this 3D HMC animation generator.

It can render regular animations or produce anaglyph 3D encoding (red/blue). Unless you have 3D glasses, unclick the “Anaglyph 3D” checkbox at the bottom of the upper left corner control box.

The app let you zoom in and rotate the visualization with obvious controls (explanation in the footer of the visualization). The app also lets you adjust the amount of correlation in the 3D normal distribution as well as step size, number of steps, and animation speed. Looking the long way down a highly correlated “cigar” shape is dramatic.

The 3D effect with glasses is strongest when you rotate the visualization (it’s the usual intuitive controls with instructions at the bottom of the web page) and zoom in a bit. I find that using low 3D depth looks the best. Don’t get your hopes up too much. This isn’t Dr. Strange creating buildings in 3D in a Marvel movie.

If you want to pop it up in an independent browser so you can go to full screen, here’s a link.

• 3D Hamiltonian Monte Carlo Animation

How the sausage was made

I continue to be amazed at the progress of the frontier LLMs. The demo above was the result of handing Claude Opus 4.8 (“hard” thinking mode) the following single prompt with no build up. As with the Galileo inclined plane case study I posted, which Opus one-shotted, I was expecting some back and forth and false starts.

I want to generate a 3D animation for red/blue glasses of the Hamiltonian Monte Carlo algorithm. There is a nice online visualizatuion by Chi Feng here, but it is not 3D https://chi-feng.github.io/mcmc-demo/app.html I just want the main animation—no need to calculate marginals, etc.

To start, we can use a 3D highly correlated (0.9) normal target with unit variance aligned at one end of the cigar (e.g., near (2, 2, 2) looking toward (-2, 2, 2), which will have things zoom over your shoulder and come back).

If you can generate it so that it’ll run in a web browser with controls on step size and number of steps that’d be great, but if not, choose a step size conservatively so it won’t be rejecting very often. I want it to continue multiple iterations in order to see the effect of random momentum on the trajectories. Leave balls behind wherever the sampler actually samples. When it rejects, make the ball bigger. The trajectory should be thick enough to be visible.

If it’s easier to have Python generate an animation that’s also fine. I just want to be able to render it on my desktop to show people during a talk. I just ordered 50 pairs of cardboard red/blue 3D glasses to hand out.

I was wrong. It did it in one shot. After about 10 minutes of cranking away, it produced what you are looking at. The output is a self-contained (i.e., encapsulated) HTML file of 627KB. There are some things I’d change in an iteration (smaller pipes, fewer of them lying around), but I think it’s worth sharing the output of such a simple prompt. Perhaps needless to say, a follow up prompt gave me the HTML I needed to embed the result in this page as an iframe.

I wrote all 692 words of the blog post myself (other than the html embedding), but I’m sure Claude could have done that, too. The LLMs have fewer rhetorical tics when writing technical and scientific material. But it wouldn’t have sounded like me.

Statistical visualization in the mid 2020s?

I wonder what Andrew’s statistics visualization class would look like in 2026 with LLM-powered visualizations this easy to make. Now that the LLMs can reliably one-shot something this complex, I’m finally starting to worry about the future of programmers. Undergraduate enrollments in CS are very volatile and already going back down as they did after the dot com bubble burst. There was huge growth (a factor of two to three) from after the mortgage market bubble burst around 2007 until it started to decline again due to AI.

I saw in a recent issue of the Times Literary Supplement that you have been critical of the “chambermaid” study which purported to show that people were losing weight without changing their diet or exercise. I agree that this study did not show what it claimed.

Along these lines, you might be interested in two articles I recently published with Nicholas Brown:
– How statistical challenges and misreadings of the literature combineto produce unreplicable science: An example from psychology
– This is the reason for external replication

Also I looked you up and saw that you were a scholar of feminism, so you might be interested in my post from a few years ago, How feminism has made me a better scientist. Any thoughts on that would be much appreciated.

I was not able to find your email online–for some reason, it’s often hard to find email contacts for people without current university affiliations–so I’m posting this here on the hope that someone who has your contact information can forward it to you.

Yours,

Andrew Gelman
Professor, Department of Statistics
Professor, Department of Political Science
Columbia University, New York

P.S. I blogged the above because I couldn’t find Tavris’s email. But then someone found her email for me. So I emailed her directly. I’ll keep the post up because it could be of interest to others!

P.P.S. The TLS took down Tavris’s review at her request. But it seems to be reprinted here with slight revision.

Qin Huang, Moyan Liu, and Upmanu Lall write:

Extreme weather events, e.g., droughts, floods, heatwaves, and freezes, are increasing in frequency and intensity, posing severe socio-economic impacts as growing populations heighten exposure to risks that conventional infrastructure cannot fully address. We propose supplementing disaster management with Weather Jiu-Jitsu: a strategy that exploits the chaotic sensitivity of mid-latitude atmospheric dynamics to redirect destructive weather trajectories through small, precisely timed perturbations guided by Finite-Time Lyapunov Exponent (FTLE) diagnostics and deep learning forecast models.

They continue:

Proof-of-concept experiments using the Aurora deep-learning Earth system model show that FTLE-guided nudges applied days before peak impact can shift a hurricane track to avoid landfall on a major city, weaken the peak intensity of a blocking-driven cold extreme, and reduce atmospheric river moisture transport under favorable upstream conditions. Control inputs remain below 2% of total system energy in idealized models, though real-world implementation will require advances in monitoring, attribution, and international governance.

There are some cool ideas here. The big ideas are:

1. Small interventions early on can shift the later progression of a storm, and

2. Chaotic unpredictability can be reduced using high-tech machine learning models.

Both these two things are necessary. The first step is needed to allow this to be done with reasonable cost; the second step is needed to give it a good chance of working.

The other cool thing involves cloud seeding. As I understand it, a big hope of the 1950s was idea of seeding clouds to get rain when you want it–but it didn’t really work, because you can’t get it to rain when the water isn’t there. (I’m sure I’m butchering the science here; sorry!) But this new plan is different because you’d be seeding the clouds over the ocean, and the point is not to get it to rain right there but rather to slightly shift where the rain falls.

I can also anticipate political challenges. For example, suppose a storm is headed toward a major city, but if it were diverted it would destroy a resort frequented by rich and powerful people. This is on top of the existing moral hazard by which owners of property near the water expect to be bailed out after natural disasters.

Here are the research papers backing up the idea:

Targeted adaptive chaos control of regimes and eddy strength in two Lorenz models, by Moyan Liu, Qin Huanga, and Upmanu Lall, Chaos, Solitons and Fractals (2026).

Regime identification and control of extremes in the nonautonomous Lorenz model with chaos and intransitivity, by Moyan Liu, Qin Huanga, and Upmanu Lall, Physical Review E (2026).

Upmanu is a water engineer with big ideas. A bunch of years ago he floated the plan to expand Manhattan’s west side by a few hundred meters by taking the silt that is continuously being dredged from the Hudson River and depositing it on the shore as landfill. That never happened but it still seems like a good idea to me. It’s kind of crazy how they’ll spend billions on a single bridge or remodeled train station or whatever but whiff on the big infrastructure projects.

How Would You Measure and Reward Scientific Impact and Replicable Research Practices?

As NIH continues efforts to strengthen rigor, reproducibility, and public trust in science, we are seeking input from the research community on an important question: Are we measuring and rewarding the activities that matter most for advancing biomedical discovery? NIH wants to hear your perspectives on how scientific impact and rigorous research should be measured and rewarded (NOT-OD-26-087). Comments will be accepted electronically here through our Request for Information (RFI) by August 19, 2026.

My first step would be for the government to stop suppressing its own research. A visible example of this was a report from the Centers for Disease Control and Prevention that appears to have been un-published at the direct orders of the NIH director.

So, yeah, one way to “reward scientific impact and replicable research practices” is to let your own damn employees publish their work.

Beyond that, we have lots of ideas, some of which Erik, Witold, and I discuss in our recent paper, A statistical case for qualified scientific optimism.

P.S. I’m posting this right away, skipping the usual 6-month lag, because the NIH is looking for replies during the next two months.

The suit had a classic name: the Clark Gable. Navy blue and cut just right, it was the creation of Giorgio Armani, the legendary Italian designer.

It was the piece that made Pat Riley, the legendary NBA coach and executive, believe in the power of style. . . .

“I think an audience wants to see somebody on the sidelines who looks like a leader, dresses like a leader, acts like a leader,” Riley said.

It sounded like a bold claim. Sure, a business suit is undoubtedly nicer than the casual “athleisure” look — team-issue polos and pullovers — that NBA coaches adopted during the COVID-19 pandemic. But can a coat and tie really make someone more of a leader?

“It’s a perfectly reasonable thing to think,” said Abe Rutchick, a professor of psychology at California State University, Northridge. “Which is the idea that the clothes we wear have psychological meaning. We put something on, it’s not just clothes. It means something.”

Uh oh, social psychology research . . .

The article continues:

In the early 2010s, during the rise of casual attire, Rutchick and his colleagues examined a similar question and found something intriguing: Wearing formal attire might actually make a person think and act like a leader.

The researchers, using a variety of cognitive tasks, found that wearing formal clothes caused participants to shift from a concrete mode of thinking to a more abstract mindset — they thought of the big picture and looked further into the future. In other words, they thought like someone who was in charge. . . .

The paper, published in 2015, came a few years after another group of researchers found that people who wore a doctor’s white lab coat — and understood its symbolic meaning — had an increased ability to focus and pay attention. . . .

This sounds pretty bad, no joke. The early 2010s were the high-water mark of junk social psychology. This sort of study was one of the main reasons that the replication crisis became a crisis.

I thought journalists had wised up on this sort of thing, but I guess it remains afloat in the business-inspirational world of leadership.

Don’t get me wrong–I have no problem with these “leadership” stories. It’s cool to read about Pat Riley, and I have no reason to doubt that suit-wearing worked well for him. Everyone has to develop their own personal style. My problem is just with the purported scientific claims.

I found the journal article and, yeah, it’s classic replication crisis fodder:

Study 1: N = 60, p = .03
Study 2: “conceptual replication,” N = 60, p = .05 with 18 people excluded because of missing data
Study 3: N = 34, p = .02
Study 4: N = 54, p = .03 after some data were excluded
Study 5: N = 150, a mix of significant and non-significant results, conclusions made based on whether various inferences reached a significance threshold.

This is pretty much textbook bad statistical analysis of the replication-crisis variety:
– Small sample sizes and noisy data so that there’s essentially no power to detect realistic effect sizes (the kangaroo problem);
– Many researcher degrees of freedom in data exclusion, coding, and analysis, the sort of flexibility that makes it possible to achieve statistically significant p-values even in the absence of any signal;
– A bunch of p-values all in the 0.01 to 0.05 range, which is not what you’d expect from a sampling model of independent experiments (or see here);
– Flexible theories that could explain results through many sorts of interactions (the piranha problem);
– No preregistered replications.

That’s just how they did things back in 2015 so I’m not trying to single out these particular researchers. We know better now. We know not to trust this sort of claims. We don’t need to find a Wansink- or Ariely-style smoking gun; nobody’s suggesting there’s fraud here; it’s just standard-issue junk science of the sort that, until recently, was regularly published in major psychology journals and was regularly featured uncritically in major news media.

The only notable thing to me is to see this sort of claim being pushed in the New York Times now, because I had the vague impression that journalists were now aware of the replication crisis. But I guess there’s still a reservoir of credulity for such claims for stories related to the fuzzy topic of business leadership. I’d hope that straight-up sports reporting would have higher standards for the reporting of research on human performance.

P.S. This is an appropriate post for July 4th now that junk science is ensconced in the U.S. government.

𝗪𝗵𝘆 𝗛𝗮𝗿𝘃𝗮𝗿𝗱’𝘀 𝗹𝗮𝘄𝘆𝗲𝗿𝘀 𝘀𝘂𝗯𝗽𝗼𝗲𝗻𝗮𝗲𝗱 𝗺𝗲 𝗶𝗻 𝘁𝗵𝗲 𝗙𝗿𝗮𝗻𝗰𝗲𝘀𝗰𝗮 𝗚𝗶𝗻𝗼 𝗰𝗮𝘀𝗲

My wife called to me. A constable was at the door.

He handed me a subpoena to appear for a deposition in the case of Francesca Gino v. President and Fellows of Harvard College and Srikant Datar.

The subpoena puzzled us. I don’t believe I’ve ever met Francesca Gino, and I am certainly not an expert on her case. Why not call me or email me with any questions?

As directed, I arrived at the offices of Ropes & Gray, Harvard’s white-shoe law firm. I was seated in a conference room with a commanding view of Boston. Thick binders sat on the table. Video cameras were pointed at me, and a microphone clipped to my collar.

One of Harvard’s lawyers opened a binder and began the deposition. She asked about my career, publications, emails, opinions, and LinkedIn posts. Each item was examined, reviewed, noted, and filed away. Page by page. Hour by hour.

The reason for the subpoena became clear.

Harvard’s lawyers asked pointed questions about my allegations of research misconduct against HBS professor 𝗚𝗲𝗼𝗿𝗴𝗲 𝗦𝗲𝗿𝗮𝗳𝗲𝗶𝗺—and they seemed interested in how those allegations compared with the ones against Francesca Gino.

A lawyer later explained the logic. In a case like this, one side may try to show that similar situations have been treated differently.

Here, both Harvard Business School professors have been accused of research misconduct. Yet only Gino lost her tenure and her position at Harvard.

Why?

At the time of my deposition, I had not given that question much thought. But nothing focuses the mind like a deposition.

So, over the next few posts I will consider:

• Do the complaints satisfy Harvard’s standards for research misconduct?
• Is there evidence of a pattern?
• Are the allegations similarly serious?
• And any other questions that emerge.

We discussed King’s encounter with the work of George Serafeim in these two posts:

• This paper in Management Science has been cited more than 6,000 times. Wall Street executives, top government officials, and even a former U.S. Vice President have all referenced it. It’s fatally flawed, and the scholarly community refuses to do anything about it.

• False claims in a widely-cited paper. No corrections. No consequences. Welcome to the Business School.

I have no reason to think that Harvard is worse than other institutions. They just get all the publicity. When bad things happen at the University of Nevada or the University of California, you only hear about it on this blog. When it happens at Harvard or Stanford, the news goes around the world.

I also want to know: How does this subpoena thing work? Can the lawyers hold you against your will? Do they pay you for your time? The only time I’ve ever been deposed, it was for a consulting project and I was being paid. The questions were really stupid and they went on for hours, but it didn’t bother me because I could just keep my mind focused on the check.

P.S. See here for some background on the Gino case.

P.P.S. Commenter K points to further information here.

I’ve been thinking a lot lately about R-hat given that I’m using it for online converging monitoring in our new Walnuts implementation. In that setting, where I use Welford accumulators to update R-hat estimates every iteration, I can’t use split R-hat without way too much buffering. So I’ve been thinking about the effect of splitting, too, and whether we need it. I asked Andrew and he said Kenny Shirley once produced an example where split R-hat diagnosed non-convergence that regular R-hat didn’t, but that example is lost to time and we’ve never seen this kind of behavior with NUTS as far as I know (please give us an example in the comments or via email to Andrew if you have).

Relating R-hat and ESS

My intuition was that we could set a low enough R-hat threshold that it would ensure a high enough effective sample size (ESS) when we crossed it. The relation’s a little tighter than I thought, with

    Rhat^2 ≈ 1 + M / ESS,

where M is the number of chains and ESS is the effective sample size of all chains combined. There’s a multivariate proof in Vats and Knudson, 2021, Revisitng the Gelman-Rubin diagnostic, Statistical Science, page 2 and section 5 for details, but it’s pretty straightforward to get the intuition when you reduce R-hat^2 to (N-1)/N + var(chain-means) / man(chain-variances) as Charles Margossian did in his nested R-hat paper. Vats and Knudson disapprove of Andrew and Aki’s suggested threshold of 1.1 from BDA3, because it is satisfied with a combined ESS of 20 across Andrew’s default 4 chains.

Being me, I tried to validate my intuition with simulations rather than linear algebra. Also, I like to see that things work in practice that theory entails to make sure I’ve understood all the assumptions baked into the theory (one can’t prove anything without assumptions!). When asked to code a simulation using ArviZ, Claude inserted a (2 * M) in the numerator in place of the M. Where did that come from, I asked? It told me it needed the factor of 2 because ArviZ uses split Rhat. D’oh! Of course it does, because we’ve doubled M without increasing ESS.

A worked example

Suppose we have 4 chains with a combined ESS of 400. Then sqrt(1 + 4/400) ≈ 1.005 and sqrt(1 + (2 * 4) / 400) ≈ 1.01. We’ve effectively doubled the number after the 1 by splitting. Unlike Vats and Knudson, I usually don’t need an ESS >> 100, so the 400 required for split R-hat < 1.01 is perhaps a bit too conservative for my tastes. On the other hand, we face a practical problem estimating ESS reliably with fewer than 50 or so ESS per chain. Estimation is challenging because it relies on autocorrelation estimates from the chains themselves, which become much noisier when based on shorter chains. (Side question: Do we not combine autocorrelation estimates across chains to reduce standard error because some chains might not be mixing?) Also, we know this algebra wasn't a coincidence of 4 chains and 400 draws. The Taylor expansion of sqrt(1 + x) is the convergent sequence

    sqrt(1 + x) = 1 + x/2 - x^2 / 8 + x^3 / 16 + ...

When x < 0.1, the first-order approximation, sqrt(1 + x) = 1 + x / 2, is good.

The bottom line for practitioners

We need around twice as many draws to get below a fixed threshold with split R-hat than with the original R-hat.

Just in time for July 4th, Tom Ferguson, Paul Jorgensen, Matthias Lalisse, and Jie Chen share the above graph and write:

What can one Senate race reveal about the hidden machinery of American politics? In Maine, donor patterns expose how campaign finance can shape party competition, political narratives, and the choices voters are asked to make long before ballots are counted. . . .

Platner is strongly supported by Senator Bernie Sanders and other progressives, while many establishment Democrats dislike him. Major media keep printing articles questioning his character. By contrast, Collins’ somewhat contradictory legislative history attracts less coverage. . . .

Our tabulations of the race show that Collins is much closer to a typical Republican pattern (or, to be fair, those of the Old Guard Democratic leaders [Nancy Pelosi and Chuck Schumer, along with Paul Ryan and Mitch McConnell]) in a key respect: the size profile of her donors. . . .

The Republican Senator from Maine is hugely dependent on very large donors. By contrast, Platner strikingly resembles Sanders: he attracts essentially no big money. Recently the numbers of billionaires supporting the candidates has emerged as an issue. A very few have supported Platner with small sums. Almost a hundred (counting spouses) have made contributions of varying sizes to Collins. The overall configuration is as shown [above] and is perfectly obvious.

They also report:

If you put aside contributions that are below the $200 threshold for disclosure, the percentage of money received from Maine donors differs sharply between the candidates. Senate elections have been nationalized for a long time. Contributions from Maine itself make up approximately 20% of all money for Platner; by contrast, Collins’ rate is slightly under 3%. (Not a misprint.) Her biggest contributors include a Who’s Who of prominent financiers in private equity and hedge funds, including Steve Schwarzman of BlackRock, Ken Griffin of Citadel, along with other well known Republican donors, including Larry Ellison of Oracle.

And they give an example of how this works:

A day after a Super Pac backing her received a $2 million dollar contribution from a private equity magnate who, according to press reports, stood to gain munificently from President Trump’s One Big Beautiful Bill, [Collins] provided a crucial vote to spring the bill out of committee. Then she loudly voted against it on the floor.

Another way of looking at this is to ask, why a person living outside of Maine give $100,000+ to Susan Collins? Roughly speaking, the following conditions are needed:
1. The donor has to be rich enough to be able to spare $100,000 as loose change.
2. It has to be legally possible to give this amount of money, or the perceived consequences of violating the law have to be minimal.
3. The donor has to consider Republican Party control of the U.S. Senate has to be important enough to be worth spending $100,000 to make a small change in the probability of this happening.
4. It has to be easy to write the check; that is, the donor does not need to get the agreement of many other people to release the money.
5. Any negative political, social, and economic consequences of revealing oneself to be a strong partisan have to be mild, compared to the perceived benefits of making the donation.

And in recent years these five conditions have increasingly been present:
1. There are more and more super-rich people who can spend $100,000 without blinking an eye.
2. The Supreme Court keeps liberalizing campaign finance laws, also the government has become much more encouraging and tolerant of corruption. On the rare occasions where people are prosecuted, they get off, and even on the rare occasions are imprisoned for corruption, they get pardoned.
3. With political polarization, the two parties are further apart than ever, and party-line voting in Congress has become the norm.
4. The money is being given by individuals, or by companies controlled by single individuals. It’s not like the old days, where, if General Motors made a campaign contribution, they’d need the coordination of some board of directors.
5. This last one is the most interesting. A flip side of partisan polarization is that, if you give a lot of money to the Republicans, it will piss off a lot of Democrats, and vice versa. Political independents might not be so happy either. One way out is that it’s becoming easier and easier to skirt the regulations and campaign in secret. Beyond this, I guess these donors have decided that the Republican business sphere is large enough that they can afford to alienate Democrats and independents. And Black Rock, Citadel, and Oracle are not primarily customer-facing businesses.

The above sketch shows a decision tree.

The circles are uncertainty nodes and the squares are decision nodes. Read the tree from left to right: to start, there is uncertainty of which of the strata i=1,…,I you will be in. In any given stratum, you will have to decide between options 1 and 2, and for each of these decision options there is uncertainty about the payoff.

The goals are:

(a) Conditional on the stratum, pick the best decision. This is the local decision problem.

(b) Averaging over the strata, evaluate the expected value of the tree, that is, the expected value under an optimal decision analysis given the uncertainty.

The challenge is that you don’t know which internal decision is best, because there is uncertainty about the payoffs.

The “optimizer’s curse” is that if, for each stratum in step (a), you make the best decision given available information–that is, you estimate the expected payoff under each of the two decision options and then pick the the one whose expected payoff is higher–then if you use these expected payoffs in step (b) you will systematically overestimate the value of the tree.

The “curse” here is not that the optimizer is making bad decisions, it’s that a naive estimate will be overly optimistic about the net value because you’re selecting on choices that look good.

In 2007, Erwann Rogard, Hao Lu, and I published a paper on the topic, including the above diagram. Here’s our abstract:

The evaluation of decision trees under uncertainty is difficult because of the required nested operations of maximizing and averaging. Pure maximizing (for deterministic decision trees) or pure averaging (for probability trees) are both relatively simple because the maximum of a maximum is a maximum, and the average of an average is an average. But when the two operators are mixed, no simplification is possible, and one must evaluate the maximization and averaging operations in a nested fashion, following the structure of the tree. Nested evaluation requires large sample sizes (for data collection) or long computation times (for simulations).

An alternative to full nested evaluation is to perform a random sample of evaluations and use statistical methods to perform inference about the entire tree. We show that the most natural estimate is biased and consider two alternatives: the parametric bootstrap and hierarchical Bayes inference. We explore the properties of these inferences through a simulation study.

I kinda like the paper. I wouldn’t say it’s one of my all-time favorites, but I think it’s interesting, and I like that we offer two different solutions to the problem.

On the downside, the paper seems to have disappeared without a trace. In 20 years, it’s only been cited three times, and none of them look very impressive:

“Using Alternating Decision Treets,” indeed.

Maybe one problem with our paper was its dry-as-dust title, “Evaluation of multilevel decision trees.”

This all came to mind because Sean Manning pointed me to this post, “The best cause will disappoint you: An intro to the optimisers curse.” Now that’s a good title.

It seems that the term “optimizer’s curse” came from this 2006 paper by James Smith and Robert Winkler, which has a lot of overlap with our article that appeared a year later. Both papers use hierarchical Bayesian analysis. Their paper is better than ours, for sure, and not just in the title, as they make a much better case for the importance of the problem. But we were working independently. Too bad: had we joined forces we could’ve produced something better, as each of the two papers had lots of material that was not in the other. Smith and Winkler consider the problem of choosing among many options with different levels of uncertainty, whereas we consider a multiplicity of binary decisions. These are just two cases of the general principle.

The above-linked post, by someone who goes by the handle “titotal,” is good too. It doesn’t have any new technical material, but it explains the problem in plain English from first principles, goes through some examples, and discusses some of the policy implications.

Say we want to estimate average Platner support in Maine’s likely electorate, E(Y). But we only have survey respondents, R = 1.

The NYT uses survey weights to weight respondents, E(YW | R = 1). In contrast, some pollsters use MRP, fitting a Multilevel Regression model for Platner support, then applying it to the population, E(E_model(Y | X, R = 1)).

Nate discusses 2 Big Changes to how they construct the weights W.

(The polar bear has not yet hiked in ME, but he is training for it. This above is in TN.)

Big Change 1: Support score

A few weeks ago we saw the NYT started weighting on “synthetic 2024 vote”, which is recalled 2024 vote that is validated with the voter file and imputed if needed.

Now they’re also weighting on support score = E(2024 vote | other X variables). Nate explains the motivation:

While a poll can’t weight on dozens of variables, the support score lets us pile a lot of information into a single measure.

This reminded me of the causal inference context, where D’Amour and Franks (2021) “see especially strong performance for propensity weights computed with respect to the prognostic score”, where the prognostic score is E(Y | X, control). In our survey context, this would be a model for Platner support Y. Instead, the NYT use 2024 vote, perhaps for applicability across multiple outcomes Y ?

Big Change 2: Energy balancing

Beyond adding new weighting variables, they’re also changing how they calculate the weights. Nate notes the challenge of weighting on many variables and interactions with typical sample sizes. So they are turning to the R package WeightIt, which implements the energy balancing method from Huling & Mak (2024):

This article introduces a new weighting method, called energy balancing, which instead aims to balance weighted covariate distributions. By directly targeting distributional imbalance, the proposed weighting strategy can be flexibly utilized in a wide variety of causal analyses without the need for careful model or moment specification.

The energy balancing weights do not use outcome Y, but the paper notes that estimates can be improved with a model for Y.

How do energy balancing weights handle the challenge of jointly weighting on many variables with typical sample sizes “without the need for model specification” ?

Therefore i think you should pursue a mixed strategy. On the one hand, you should non-aggressively, soberly, suggest that the groping allegation is exaggerated but likely the result of a good faith misunderstanding. At the same time you should acknowledge that all these accusations have woken you up. You had never fully realized how vulnerable women are, and how the “me-too” campaign reflects decades of oppression and exploitation. You were blindly ignorant of, and insensitive to, this reality. this blind ignorance was all the more inexcusable in that you yourself have a daughter. You absolutely pledge that all your future behavior will reflect this newfound realization. You pledge to enroll (and indeed you should find and enroll in before making this pledge) in a program designed to educate and sensitize men to the pervasive atmosphere of sexual assault and harassment. You pledge to devote the rest of your career to this goal and to change your behavior to reflect this new realization. You pledge never ever again to make gestures that even have a slight chance of being perceived as harassing to females. You apologize profusely for all your hurtful gestures in the past, and recognize that the women who have complained about you are not making their complaints up. You were too physical in the past, you were blind to the vulnerability of women exposed to men in positions of power and influence, you abused that position and their trust even though you were sure at the time that you were doing nothing wrong. You know better now, because you understand women’s vulnerability in ways you didn’t before. You humbly ask Arizona State, or indeed any university that is interested, to give you another chance to show that you are in fact nothing but a caring, active physicist who is now more respectful of women. You are absolutely dedicated to pursuing your academic aspirations without future distractions. Importantly, you should do something dramatic, such as offer 100% of the royalties from your next book to some foundation that assists women who have been victims of harassment.

Jeez, what an asshole, to recommend that the “caring, active physicist” bring his daughter into his P.R. strategy.

In any case it seems that Krauss did not follow his brother’s advice. Not only does Krauss express no remorse about his own behavior, he hedges his bets on Jeffrey Epstein, referring to the financier’s “alleged criminality.” (Elsewhere he wrote that “everyone was a victim, including Jeffrey here.”)

Scroll down below Krauss’s linked post and here are the other things they recommend you read:

“Why We Need to Talk About Transgender School Shooters,” indeed. On the other hand, it seems that this is only the 325th most important thing they needed to talk about, so maybe that need wasn’t so great.

And here’s the second of those links:

I agree with sub-heading on this one. The paradox is that Arizona State, Harvard, and other Epstein-associated universities were themselves “rewarding those who exemplify and cultivate intellectual vices.”

And, yes, I’m saying intellectual vices, not just financial and sexual vices. To the extent that Epstein stood for anything intellectually, it was the principle of recirculating B.S. from well-placed elites (as here). Also the above proposed parade of insincerity (oh, sorry, the “mixed strategy”) is an intellectual vice. For that matter, I think it was an intellectual vice for Biggar to misrepresent the position of someone with whom he had an academic and political dispute.

That’s fine. Biggar can be correct in his larger point even if he does not always live up to these ideals himself, and it’s not his fault that he happened to have published on the same website as someone who is a kind of negative illustration of his point. It’s just interesting to see the juxtaposition.

But, hey, for a mere $4500 you can go on a one-week cruise with this guy (that’s Krauss, not Biggar). I think that part of what you get for this equivalent of 3130 Jamaican beef patties is the right to come up to him on the boat and say, “Hey, Lorrie, what’s your position on the statement, ‘You had never fully realized how vulnerable women are, and how the “me-too” campaign reflects decades of oppression and exploitation. . . . You absolutely pledge that all your future behavior will reflect this newfound realization. . . . You pledge to devote the rest of your career to this goal’?” For $4500, the least he can give you is a straight answer.

Only $1000 for the statistician??? What a cheapskate! Especially given that he said the statistician “did an outstanding job.”

Given all the statistical problems in evolutionary biology, maybe he should’ve allocated more of his research budget to the statistician.

Some background on Trivers is here.