I have a question for you–and, if you think it worthwhile, for your readers.

A few weeks ago, I was deposed by Harvard’s lawyers in the lawsuit between Francesca Gino and Harvard. Much of the questioning focused on my replications of research by Harvard Business School professor George Serafeim and my allegations of research misconduct against him and his coauthors.

That experience has led to a lively online debate about two questions:
1. Is fabricating data worse than fabricating results?
2. Is failing to correct a known false report more or less serious than making the false report in the first place?

At the moment, my own thinking is this:
1. Both fabricating data and fabricating results mislead readers. They are simply different paths to the same outcome and thus similarly serious.
2. Failing to correct a false report–once the authors know it is false and material–may actually be more serious. It suggests a conscious decision to leave readers with a claim the authors know to be unsupported.

Your ladder of responses to criticism also seems relevant here, especially categories 6 and 7.

Interesting. This has come up in the past, discussing the moral culpability of researchers who make errors and then avoid acknowledging them. For example this guy at the London School of Economics and Political Science, or this guy at the University of Chicago, or, of course, this guy at the University of California. I don’t think that the first two of those people did any direct research misconduct, but they made major research errors that they never acknowledged–they keep pointing to their discredited work without any note of the problems–and, yeah, that seems like misconduct to me.

Here’s another story for ya. Years ago I had a colleague who showed me a paper he’d just written. It read the paper and realized it had a fatal flaw–not a calculation error, but a misapplication or misunderstanding of a statistical model. I won’t go into the details here; what’s relevant to the story right now is that the paper in question had been accepted by the journal but it had not yet been scheduled for publication. This was before the era of online anything, so the paper really was still in process. I told me colleague he was lucky: he could withdraw the paper and spare himself embarrassment. (The error in the analysis was central to the result in the paper; if you got rid of the error, there was nothing to salvage, so it’s not like he could just send in a corrected version.) To my dismay, my colleague replied, No, the paper is accepted, I don’t want to lose a publication. I asked, Doesn’t it bother you to have them publish something that’s wrong?, and he replied something about the literature being self-correcting. I don’t remember the details of this conversation from decades ago, but I do remember the horrible feeling. I thought about contacting the journal to tell them not to publish, but I figured that ultimately it was their problem for accepting it.

1. New weighting variable: support score = E(2024 vote | other X variables).
2. New weighting method: energy balancing (Huling & Mak, 2024)

Ben Schneider helpfully blogged about energy balancing as well:

Raking and similar calibration methods are based on balancing means or totals for specific variables…The energy balancing method does something different: it calibrates based on an entire multivariate distribution, as measured by an empirical cumulative distribution function (ECDF).

Jared Huling (of Huling & Mak, 2024) helpfully answered questions in the comments. I’m still puzzling over how energy balancing handles empty cells (unsampled regions of the joint covariate space). I need a toy example.

Consider 2 binary variables, so 4 population cells, with known population shares:

       k=0    k=1    total
j=0    .4     .2     .6
j=1    .2     .2     .4
total  .6     .4

Say the sample is missing folks in cell 11:

       k=0    k=1    total
j=0    .5     .3     .8
j=1    .2     0      .2
total  .7     .3

Consider 4 methods:

1. Classical Poststratification: not defined because of division by 0.

2. Raking: match only the margins. Correct when Y | X1, X2 is additive.

       k=0    k=1    total
j=0    .2     .4     .6
j=1    .4     0      .4
total  .6     .4

3. Energy balancing: minimize the Energy-Distance(F_w, F_pop) between the weighted sample distribution of X1, X2 and the population distribution. Correct when Y | X1, X2 is such that nearby cells have similar means.

Say X1 = young/old, X2 = man/woman, Y = percent Democrats, and no old women are sampled.

Raking is correct when additivity holds: old women = young women + (old men − young men)

Energy balancing is correct approximately when: old women = (old men + young women)/2 ?

library(WeightIt)

pop  <- data.frame(X1 = rep(c(0, 0, 1, 1), c(40, 20, 20, 20)),
                   X2 = rep(c(0, 1, 0, 1), c(40, 20, 20, 20)))

samp <- data.frame(X1 = rep(c(0, 0, 1), c(50, 30, 20)),
                   X2 = rep(c(0, 1, 0), c(50, 30, 20)))

dat <- rbind(cbind(pop,  A = 1),
             cbind(samp, A = 0))

W <- weightit(A ~ X1 + X2, data = dat, method = "energy",
              estimand = "ATT", focal = "1",
              dist.mat = as.matrix(dist(dat[, c("X1", "X2")])))

w <- W$weights[dat$A == 0]
tapply(w, interaction(samp$X1, samp$X2), sum) / sum(w)

       k=0    k=1    total
j=0    .381   .309   .69
j=1    .309   0      .309
total  .69    .309

4. MRP: fit a model for Y | X1, X2. The interaction term’s posterior equals its prior, propagating uncertainty around additivity.

Am I understanding this correctly ?

The sausage

So as not to bury the lead (or “lede” if you want a mid-20th-century newspaper vibe), check out the this 3D HMC animation generator.

It can render regular animations or produce anaglyph 3D encoding (red/blue). Unless you have 3D glasses, unclick the “Anaglyph 3D” checkbox at the bottom of the upper left corner control box.

The app let you zoom in and rotate the visualization with obvious controls (explanation in the footer of the visualization). The app also lets you adjust the amount of correlation in the 3D normal distribution as well as step size, number of steps, and animation speed. Looking the long way down a highly correlated “cigar” shape is dramatic.

The 3D effect with glasses is strongest when you rotate the visualization (it’s the usual intuitive controls with instructions at the bottom of the web page) and zoom in a bit. I find that using low 3D depth looks the best. Don’t get your hopes up too much. This isn’t Dr. Strange creating buildings in 3D in a Marvel movie.

If you want to pop it up in an independent browser so you can go to full screen, here’s a link.

• 3D Hamiltonian Monte Carlo Animation

How the sausage was made

I continue to be amazed at the progress of the frontier LLMs. The demo above was the result of handing Claude Opus 4.8 (“hard” thinking mode) the following single prompt with no build up. As with the Galileo inclined plane case study I posted, which Opus one-shotted, I was expecting some back and forth and false starts.

I want to generate a 3D animation for red/blue glasses of the Hamiltonian Monte Carlo algorithm. There is a nice online visualizatuion by Chi Feng here, but it is not 3D https://chi-feng.github.io/mcmc-demo/app.html I just want the main animation—no need to calculate marginals, etc.

To start, we can use a 3D highly correlated (0.9) normal target with unit variance aligned at one end of the cigar (e.g., near (2, 2, 2) looking toward (-2, 2, 2), which will have things zoom over your shoulder and come back).

If you can generate it so that it’ll run in a web browser with controls on step size and number of steps that’d be great, but if not, choose a step size conservatively so it won’t be rejecting very often. I want it to continue multiple iterations in order to see the effect of random momentum on the trajectories. Leave balls behind wherever the sampler actually samples. When it rejects, make the ball bigger. The trajectory should be thick enough to be visible.

If it’s easier to have Python generate an animation that’s also fine. I just want to be able to render it on my desktop to show people during a talk. I just ordered 50 pairs of cardboard red/blue 3D glasses to hand out.

I was wrong. It did it in one shot. After about 10 minutes of cranking away, it produced what you are looking at. The output is a self-contained (i.e., encapsulated) HTML file of 627KB. There are some things I’d change in an iteration (smaller pipes, fewer of them lying around), but I think it’s worth sharing the output of such a simple prompt. Perhaps needless to say, a follow up prompt gave me the HTML I needed to embed the result in this page as an iframe.

I wrote all 692 words of the blog post myself (other than the html embedding), but I’m sure Claude could have done that, too. The LLMs have fewer rhetorical tics when writing technical and scientific material. But it wouldn’t have sounded like me.

Statistical visualization in the mid 2020s?

I wonder what Andrew’s statistics visualization class would look like in 2026 with LLM-powered visualizations this easy to make. Now that the LLMs can reliably one-shot something this complex, I’m finally starting to worry about the future of programmers. Undergraduate enrollments in CS are very volatile and already going back down as they did after the dot com bubble burst. There was huge growth (a factor of two to three) from after the mortgage market bubble burst around 2007 until it started to decline again due to AI.

I saw in a recent issue of the Times Literary Supplement that you have been critical of the “chambermaid” study which purported to show that people were losing weight without changing their diet or exercise. I agree that this study did not show what it claimed.

Along these lines, you might be interested in two articles I recently published with Nicholas Brown:
– How statistical challenges and misreadings of the literature combineto produce unreplicable science: An example from psychology
– This is the reason for external replication

Also I looked you up and saw that you were a scholar of feminism, so you might be interested in my post from a few years ago, How feminism has made me a better scientist. Any thoughts on that would be much appreciated.

I was not able to find your email online–for some reason, it’s often hard to find email contacts for people without current university affiliations–so I’m posting this here on the hope that someone who has your contact information can forward it to you.

Yours,

Andrew Gelman
Professor, Department of Statistics
Professor, Department of Political Science
Columbia University, New York

P.S. I blogged the above because I couldn’t find Tavris’s email. But then someone found her email for me. So I emailed her directly. I’ll keep the post up because it could be of interest to others!

Qin Huang, Moyan Liu, and Upmanu Lall write:

Extreme weather events, e.g., droughts, floods, heatwaves, and freezes, are increasing in frequency and intensity, posing severe socio-economic impacts as growing populations heighten exposure to risks that conventional infrastructure cannot fully address. We propose supplementing disaster management with Weather Jiu-Jitsu: a strategy that exploits the chaotic sensitivity of mid-latitude atmospheric dynamics to redirect destructive weather trajectories through small, precisely timed perturbations guided by Finite-Time Lyapunov Exponent (FTLE) diagnostics and deep learning forecast models.

They continue:

Proof-of-concept experiments using the Aurora deep-learning Earth system model show that FTLE-guided nudges applied days before peak impact can shift a hurricane track to avoid landfall on a major city, weaken the peak intensity of a blocking-driven cold extreme, and reduce atmospheric river moisture transport under favorable upstream conditions. Control inputs remain below 2% of total system energy in idealized models, though real-world implementation will require advances in monitoring, attribution, and international governance.

There are some cool ideas here. The big ideas are:

1. Small interventions early on can shift the later progression of a storm, and

2. Chaotic unpredictability can be reduced using high-tech machine learning models.

Both these two things are necessary. The first step is needed to allow this to be done with reasonable cost; the second step is needed to give it a good chance of working.

The other cool thing involves cloud seeding. As I understand it, a big hope of the 1950s was idea of seeding clouds to get rain when you want it–but it didn’t really work, because you can’t get it to rain when the water isn’t there. (I’m sure I’m butchering the science here; sorry!) But this new plan is different because you’d be seeding the clouds over the ocean, and the point is not to get it to rain right there but rather to slightly shift where the rain falls.

I can also anticipate political challenges. For example, suppose a storm is headed toward a major city, but if it were diverted it would destroy a resort frequented by rich and powerful people. This is on top of the existing moral hazard by which owners of property near the water expect to be bailed out after natural disasters.

Here are the research papers backing up the idea:

Targeted adaptive chaos control of regimes and eddy strength in two Lorenz models, by Moyan Liu, Qin Huanga, and Upmanu Lall, Chaos, Solitons and Fractals (2026).

Regime identification and control of extremes in the nonautonomous Lorenz model with chaos and intransitivity, by Moyan Liu, Qin Huanga, and Upmanu Lall, Physical Review E (2026).

Upmanu is a water engineer with big ideas. A bunch of years ago he floated the plan to expand Manhattan’s west side by a few hundred meters by taking the silt that is continuously being dredged from the Hudson River and depositing it on the shore as landfill. That never happened but it still seems like a good idea to me. It’s kind of crazy how they’ll spend billions on a single bridge or remodeled train station or whatever but whiff on the big infrastructure projects.

How Would You Measure and Reward Scientific Impact and Replicable Research Practices?

As NIH continues efforts to strengthen rigor, reproducibility, and public trust in science, we are seeking input from the research community on an important question: Are we measuring and rewarding the activities that matter most for advancing biomedical discovery? NIH wants to hear your perspectives on how scientific impact and rigorous research should be measured and rewarded (NOT-OD-26-087). Comments will be accepted electronically here through our Request for Information (RFI) by August 19, 2026.

My first step would be for the government to stop suppressing its own research. A visible example of this was a report from the Centers for Disease Control and Prevention that appears to have been un-published at the direct orders of the NIH director.

So, yeah, one way to “reward scientific impact and replicable research practices” is to let your own damn employees publish their work.

Beyond that, we have lots of ideas, some of which Erik, Witold, and I discuss in our recent paper, A statistical case for qualified scientific optimism.

P.S. I’m posting this right away, skipping the usual 6-month lag, because the NIH is looking for replies during the next two months.

The suit had a classic name: the Clark Gable. Navy blue and cut just right, it was the creation of Giorgio Armani, the legendary Italian designer.

It was the piece that made Pat Riley, the legendary NBA coach and executive, believe in the power of style. . . .

“I think an audience wants to see somebody on the sidelines who looks like a leader, dresses like a leader, acts like a leader,” Riley said.

It sounded like a bold claim. Sure, a business suit is undoubtedly nicer than the casual “athleisure” look — team-issue polos and pullovers — that NBA coaches adopted during the COVID-19 pandemic. But can a coat and tie really make someone more of a leader?

“It’s a perfectly reasonable thing to think,” said Abe Rutchick, a professor of psychology at California State University, Northridge. “Which is the idea that the clothes we wear have psychological meaning. We put something on, it’s not just clothes. It means something.”

Uh oh, social psychology research . . .

The article continues:

In the early 2010s, during the rise of casual attire, Rutchick and his colleagues examined a similar question and found something intriguing: Wearing formal attire might actually make a person think and act like a leader.

The researchers, using a variety of cognitive tasks, found that wearing formal clothes caused participants to shift from a concrete mode of thinking to a more abstract mindset — they thought of the big picture and looked further into the future. In other words, they thought like someone who was in charge. . . .

The paper, published in 2015, came a few years after another group of researchers found that people who wore a doctor’s white lab coat — and understood its symbolic meaning — had an increased ability to focus and pay attention. . . .

This sounds pretty bad, no joke. The early 2010s were the high-water mark of junk social psychology. This sort of study was one of the main reasons that the replication crisis became a crisis.

I thought journalists had wised up on this sort of thing, but I guess it remains afloat in the business-inspirational world of leadership.

Don’t get me wrong–I have no problem with these “leadership” stories. It’s cool to read about Pat Riley, and I have no reason to doubt that suit-wearing worked well for him. Everyone has to develop their own personal style. My problem is just with the purported scientific claims.

I found the journal article and, yeah, it’s classic replication crisis fodder:

Study 1: N = 60, p = .03
Study 2: “conceptual replication,” N = 60, p = .05 with 18 people excluded because of missing data
Study 3: N = 34, p = .02
Study 4: N = 54, p = .03 after some data were excluded
Study 5: N = 150, a mix of significant and non-significant results, conclusions made based on whether various inferences reached a significance threshold.

This is pretty much textbook bad statistical analysis of the replication-crisis variety:
– Small sample sizes and noisy data so that there’s essentially no power to detect realistic effect sizes (the kangaroo problem);
– Many researcher degrees of freedom in data exclusion, coding, and analysis, the sort of flexibility that makes it possible to achieve statistically significant p-values even in the absence of any signal;
– A bunch of p-values all in the 0.01 to 0.05 range, which is not what you’d expect from a sampling model of independent experiments (or see here);
– Flexible theories that could explain results through many sorts of interactions (the piranha problem);
– No preregistered replications.

That’s just how they did things back in 2015 so I’m not trying to single out these particular researchers. We know better now. We know not to trust this sort of claims. We don’t need to find a Wansink- or Ariely-style smoking gun; nobody’s suggesting there’s fraud here; it’s just standard-issue junk science of the sort that, until recently, was regularly published in major psychology journals and was regularly featured uncritically in major news media.

The only notable thing to me is to see this sort of claim being pushed in the New York Times now, because I had the vague impression that journalists were now aware of the replication crisis. But I guess there’s still a reservoir of credulity for such claims for stories related to the fuzzy topic of business leadership. I’d hope that straight-up sports reporting would have higher standards for the reporting of research on human performance.

P.S. This is an appropriate post for July 4th now that junk science is ensconced in the U.S. government.

𝗪𝗵𝘆 𝗛𝗮𝗿𝘃𝗮𝗿𝗱’𝘀 𝗹𝗮𝘄𝘆𝗲𝗿𝘀 𝘀𝘂𝗯𝗽𝗼𝗲𝗻𝗮𝗲𝗱 𝗺𝗲 𝗶𝗻 𝘁𝗵𝗲 𝗙𝗿𝗮𝗻𝗰𝗲𝘀𝗰𝗮 𝗚𝗶𝗻𝗼 𝗰𝗮𝘀𝗲

My wife called to me. A constable was at the door.

He handed me a subpoena to appear for a deposition in the case of Francesca Gino v. President and Fellows of Harvard College and Srikant Datar.

The subpoena puzzled us. I don’t believe I’ve ever met Francesca Gino, and I am certainly not an expert on her case. Why not call me or email me with any questions?

As directed, I arrived at the offices of Ropes & Gray, Harvard’s white-shoe law firm. I was seated in a conference room with a commanding view of Boston. Thick binders sat on the table. Video cameras were pointed at me, and a microphone clipped to my collar.

One of Harvard’s lawyers opened a binder and began the deposition. She asked about my career, publications, emails, opinions, and LinkedIn posts. Each item was examined, reviewed, noted, and filed away. Page by page. Hour by hour.

The reason for the subpoena became clear.

Harvard’s lawyers asked pointed questions about my allegations of research misconduct against HBS professor 𝗚𝗲𝗼𝗿𝗴𝗲 𝗦𝗲𝗿𝗮𝗳𝗲𝗶𝗺—and they seemed interested in how those allegations compared with the ones against Francesca Gino.

A lawyer later explained the logic. In a case like this, one side may try to show that similar situations have been treated differently.

Here, both Harvard Business School professors have been accused of research misconduct. Yet only Gino lost her tenure and her position at Harvard.

Why?

At the time of my deposition, I had not given that question much thought. But nothing focuses the mind like a deposition.

So, over the next few posts I will consider:

• Do the complaints satisfy Harvard’s standards for research misconduct?
• Is there evidence of a pattern?
• Are the allegations similarly serious?
• And any other questions that emerge.

We discussed King’s encounter with the work of George Serafeim in these two posts:

• This paper in Management Science has been cited more than 6,000 times. Wall Street executives, top government officials, and even a former U.S. Vice President have all referenced it. It’s fatally flawed, and the scholarly community refuses to do anything about it.

• False claims in a widely-cited paper. No corrections. No consequences. Welcome to the Business School.

I have no reason to think that Harvard is worse than other institutions. They just get all the publicity. When bad things happen at the University of Nevada or the University of California, you only hear about it on this blog. When it happens at Harvard or Stanford, the news goes around the world.

I also want to know: How does this subpoena thing work? Can the lawyers hold you against your will? Do they pay you for your time? The only time I’ve ever been deposed, it was for a consulting project and I was being paid. The questions were really stupid and they went on for hours, but it didn’t bother me because I could just keep my mind focused on the check.

P.S. See here for some background on the Gino case.

P.P.S. Commenter K points to further information here.

I’ve been thinking a lot lately about R-hat given that I’m using it for online converging monitoring in our new Walnuts implementation. In that setting, where I use Welford accumulators to update R-hat estimates every iteration, I can’t use split R-hat without way too much buffering. So I’ve been thinking about the effect of splitting, too, and whether we need it. I asked Andrew and he said Kenny Shirley once produced an example where split R-hat diagnosed non-convergence that regular R-hat didn’t, but that example is lost to time and we’ve never seen this kind of behavior with NUTS as far as I know (please give us an example in the comments or via email to Andrew if you have).

Relating R-hat and ESS

My intuition was that we could set a low enough R-hat threshold that it would ensure a high enough effective sample size (ESS) when we crossed it. The relation’s a little tighter than I thought, with

    Rhat^2 ≈ 1 + M / ESS,

where M is the number of chains and ESS is the effective sample size of all chains combined. There’s a multivariate proof in Vats and Knudson, 2021, Revisitng the Gelman-Rubin diagnostic, Statistical Science, page 2 and section 5 for details, but it’s pretty straightforward to get the intuition when you reduce R-hat^2 to (N-1)/N + var(chain-means) / man(chain-variances) as Charles Margossian did in his nested R-hat paper. Vats and Knudson disapprove of Andrew and Aki’s suggested threshold of 1.1 from BDA3, because it is satisfied with a combined ESS of 20 across Andrew’s default 4 chains.

Being me, I tried to validate my intuition with simulations rather than linear algebra. Also, I like to see that things work in practice that theory entails to make sure I’ve understood all the assumptions baked into the theory (one can’t prove anything without assumptions!). When asked to code a simulation using ArviZ, Claude inserted a (2 * M) in the numerator in place of the M. Where did that come from, I asked? It told me it needed the factor of 2 because ArviZ uses split Rhat. D’oh! Of course it does, because we’ve doubled M without increasing ESS.

A worked example

Suppose we have 4 chains with a combined ESS of 400. Then sqrt(1 + 4/400) ≈ 1.005 and sqrt(1 + (2 * 4) / 400) ≈ 1.01. We’ve effectively doubled the number after the 1 by splitting. Unlike Vats and Knudson, I usually don’t need an ESS >> 100, so the 400 required for split R-hat < 1.01 is perhaps a bit too conservative for my tastes. On the other hand, we face a practical problem estimating ESS reliably with fewer than 50 or so ESS per chain. Estimation is challenging because it relies on autocorrelation estimates from the chains themselves, which become much noisier when based on shorter chains. (Side question: Do we not combine autocorrelation estimates across chains to reduce standard error because some chains might not be mixing?) Also, we know this algebra wasn't a coincidence of 4 chains and 400 draws. The Taylor expansion of sqrt(1 + x) is the convergent sequence

    sqrt(1 + x) = 1 + x/2 - x^2 / 8 + x^3 / 16 + ...

When x < 0.1, the first-order approximation, sqrt(1 + x) = 1 + x / 2, is good.

The bottom line for practitioners

We need around twice as many draws to get below a fixed threshold with split R-hat than with the original R-hat.

Just in time for July 4th, Tom Ferguson, Paul Jorgensen, Matthias Lalisse, and Jie Chen share the above graph and write:

What can one Senate race reveal about the hidden machinery of American politics? In Maine, donor patterns expose how campaign finance can shape party competition, political narratives, and the choices voters are asked to make long before ballots are counted. . . .

Platner is strongly supported by Senator Bernie Sanders and other progressives, while many establishment Democrats dislike him. Major media keep printing articles questioning his character. By contrast, Collins’ somewhat contradictory legislative history attracts less coverage. . . .

Our tabulations of the race show that Collins is much closer to a typical Republican pattern (or, to be fair, those of the Old Guard Democratic leaders [Nancy Pelosi and Chuck Schumer, along with Paul Ryan and Mitch McConnell]) in a key respect: the size profile of her donors. . . .

The Republican Senator from Maine is hugely dependent on very large donors. By contrast, Platner strikingly resembles Sanders: he attracts essentially no big money. Recently the numbers of billionaires supporting the candidates has emerged as an issue. A very few have supported Platner with small sums. Almost a hundred (counting spouses) have made contributions of varying sizes to Collins. The overall configuration is as shown [above] and is perfectly obvious.

They also report:

If you put aside contributions that are below the $200 threshold for disclosure, the percentage of money received from Maine donors differs sharply between the candidates. Senate elections have been nationalized for a long time. Contributions from Maine itself make up approximately 20% of all money for Platner; by contrast, Collins’ rate is slightly under 3%. (Not a misprint.) Her biggest contributors include a Who’s Who of prominent financiers in private equity and hedge funds, including Steve Schwarzman of BlackRock, Ken Griffin of Citadel, along with other well known Republican donors, including Larry Ellison of Oracle.

And they give an example of how this works:

A day after a Super Pac backing her received a $2 million dollar contribution from a private equity magnate who, according to press reports, stood to gain munificently from President Trump’s One Big Beautiful Bill, [Collins] provided a crucial vote to spring the bill out of committee. Then she loudly voted against it on the floor.

Another way of looking at this is to ask, why a person living outside of Maine give $100,000+ to Susan Collins? Roughly speaking, the following conditions are needed:
1. The donor has to be rich enough to be able to spare $100,000 as loose change.
2. It has to be legally possible to give this amount of money, or the perceived consequences of violating the law have to be minimal.
3. The donor has to consider Republican Party control of the U.S. Senate has to be important enough to be worth spending $100,000 to make a small change in the probability of this happening.
4. It has to be easy to write the check; that is, the donor does not need to get the agreement of many other people to release the money.
5. Any negative political, social, and economic consequences of revealing oneself to be a strong partisan have to be mild, compared to the perceived benefits of making the donation.

And in recent years these five conditions have increasingly been present:
1. There are more and more super-rich people who can spend $100,000 without blinking an eye.
2. The Supreme Court keeps liberalizing campaign finance laws, also the government has become much more encouraging and tolerant of corruption. On the rare occasions where people are prosecuted, they get off, and even on the rare occasions are imprisoned for corruption, they get pardoned.
3. With political polarization, the two parties are further apart than ever, and party-line voting in Congress has become the norm.
4. The money is being given by individuals, or by companies controlled by single individuals. It’s not like the old days, where, if General Motors made a campaign contribution, they’d need the coordination of some board of directors.
5. This last one is the most interesting. A flip side of partisan polarization is that, if you give a lot of money to the Republicans, it will piss off a lot of Democrats, and vice versa. Political independents might not be so happy either. One way out is that it’s becoming easier and easier to skirt the regulations and campaign in secret. Beyond this, I guess these donors have decided that the Republican business sphere is large enough that they can afford to alienate Democrats and independents. And Black Rock, Citadel, and Oracle are not primarily customer-facing businesses.

The above sketch shows a decision tree.

The circles are uncertainty nodes and the squares are decision nodes. Read the tree from left to right: to start, there is uncertainty of which of the strata i=1,…,I you will be in. In any given stratum, you will have to decide between options 1 and 2, and for each of these decision options there is uncertainty about the payoff.

The goals are:

(a) Conditional on the stratum, pick the best decision. This is the local decision problem.

(b) Averaging over the strata, evaluate the expected value of the tree, that is, the expected value under an optimal decision analysis given the uncertainty.

The challenge is that you don’t know which internal decision is best, because there is uncertainty about the payoffs.

The “optimizer’s curse” is that if, for each stratum in step (a), you make the best decision given available information–that is, you estimate the expected payoff under each of the two decision options and then pick the the one whose expected payoff is higher–then if you use these expected payoffs in step (b) you will systematically overestimate the value of the tree.

The “curse” here is not that the optimizer is making bad decisions, it’s that a naive estimate will be overly optimistic about the net value because you’re selecting on choices that look good.

In 2007, Erwann Rogard, Hao Lu, and I published a paper on the topic, including the above diagram. Here’s our abstract:

The evaluation of decision trees under uncertainty is difficult because of the required nested operations of maximizing and averaging. Pure maximizing (for deterministic decision trees) or pure averaging (for probability trees) are both relatively simple because the maximum of a maximum is a maximum, and the average of an average is an average. But when the two operators are mixed, no simplification is possible, and one must evaluate the maximization and averaging operations in a nested fashion, following the structure of the tree. Nested evaluation requires large sample sizes (for data collection) or long computation times (for simulations).

An alternative to full nested evaluation is to perform a random sample of evaluations and use statistical methods to perform inference about the entire tree. We show that the most natural estimate is biased and consider two alternatives: the parametric bootstrap and hierarchical Bayes inference. We explore the properties of these inferences through a simulation study.

I kinda like the paper. I wouldn’t say it’s one of my all-time favorites, but I think it’s interesting, and I like that we offer two different solutions to the problem.

On the downside, the paper seems to have disappeared without a trace. In 20 years, it’s only been cited three times, and none of them look very impressive:

“Using Alternating Decision Treets,” indeed.

Maybe one problem with our paper was its dry-as-dust title, “Evaluation of multilevel decision trees.”

This all came to mind because Sean Manning pointed me to this post, “The best cause will disappoint you: An intro to the optimisers curse.” Now that’s a good title.

It seems that the term “optimizer’s curse” came from this 2006 paper by James Smith and Robert Winkler, which has a lot of overlap with our article that appeared a year later. Both papers use hierarchical Bayesian analysis. Their paper is better than ours, for sure, and not just in the title, as they make a much better case for the importance of the problem. But we were working independently. Too bad: had we joined forces we could’ve produced something better, as each of the two papers had lots of material that was not in the other. Smith and Winkler consider the problem of choosing among many options with different levels of uncertainty, whereas we consider a multiplicity of binary decisions. These are just two cases of the general principle.

The above-linked post, by someone who goes by the handle “titotal,” is good too. It doesn’t have any new technical material, but it explains the problem in plain English from first principles, goes through some examples, and discusses some of the policy implications.

Say we want to estimate average Platner support in Maine’s likely electorate, E(Y). But we only have survey respondents, R = 1.

The NYT uses survey weights to weight respondents, E(YW | R = 1). In contrast, some pollsters use MRP, fitting a Multilevel Regression model for Platner support, then applying it to the population, E(E_model(Y | X, R = 1)).

Nate discusses 2 Big Changes to how they construct the weights W.

(The polar bear has not yet hiked in ME, but he is training for it. This above is in TN.)

Big Change 1: Support score

A few weeks ago we saw the NYT started weighting on “synthetic 2024 vote”, which is recalled 2024 vote that is validated with the voter file and imputed if needed.

Now they’re also weighting on support score = E(2024 vote | other X variables). Nate explains the motivation:

While a poll can’t weight on dozens of variables, the support score lets us pile a lot of information into a single measure.

This reminded me of the causal inference context, where D’Amour and Franks (2021) “see especially strong performance for propensity weights computed with respect to the prognostic score”, where the prognostic score is E(Y | X, control). In our survey context, this would be a model for Platner support Y. Instead, the NYT use 2024 vote, perhaps for applicability across multiple outcomes Y ?

Big Change 2: Energy balancing

Beyond adding new weighting variables, they’re also changing how they calculate the weights. Nate notes the challenge of weighting on many variables and interactions with typical sample sizes. So they are turning to the R package WeightIt, which implements the energy balancing method from Huling & Mak (2024):

This article introduces a new weighting method, called energy balancing, which instead aims to balance weighted covariate distributions. By directly targeting distributional imbalance, the proposed weighting strategy can be flexibly utilized in a wide variety of causal analyses without the need for careful model or moment specification.

The energy balancing weights do not use outcome Y, but the paper notes that estimates can be improved with a model for Y.

How do energy balancing weights handle the challenge of jointly weighting on many variables with typical sample sizes “without the need for model specification” ?

Therefore i think you should pursue a mixed strategy. On the one hand, you should non-aggressively, soberly, suggest that the groping allegation is exaggerated but likely the result of a good faith misunderstanding. At the same time you should acknowledge that all these accusations have woken you up. You had never fully realized how vulnerable women are, and how the “me-too” campaign reflects decades of oppression and exploitation. You were blindly ignorant of, and insensitive to, this reality. this blind ignorance was all the more inexcusable in that you yourself have a daughter. You absolutely pledge that all your future behavior will reflect this newfound realization. You pledge to enroll (and indeed you should find and enroll in before making this pledge) in a program designed to educate and sensitize men to the pervasive atmosphere of sexual assault and harassment. You pledge to devote the rest of your career to this goal and to change your behavior to reflect this new realization. You pledge never ever again to make gestures that even have a slight chance of being perceived as harassing to females. You apologize profusely for all your hurtful gestures in the past, and recognize that the women who have complained about you are not making their complaints up. You were too physical in the past, you were blind to the vulnerability of women exposed to men in positions of power and influence, you abused that position and their trust even though you were sure at the time that you were doing nothing wrong. You know better now, because you understand women’s vulnerability in ways you didn’t before. You humbly ask Arizona State, or indeed any university that is interested, to give you another chance to show that you are in fact nothing but a caring, active physicist who is now more respectful of women. You are absolutely dedicated to pursuing your academic aspirations without future distractions. Importantly, you should do something dramatic, such as offer 100% of the royalties from your next book to some foundation that assists women who have been victims of harassment.

Jeez, what an asshole, to recommend that the “caring, active physicist” bring his daughter into his P.R. strategy.

In any case it seems that Krauss did not follow his brother’s advice. Not only does Krauss express no remorse about his own behavior, he hedges his bets on Jeffrey Epstein, referring to the financier’s “alleged criminality.” (Elsewhere he wrote that “everyone was a victim, including Jeffrey here.”)

Scroll down below Krauss’s linked post and here are the other things they recommend you read:

“Why We Need to Talk About Transgender School Shooters,” indeed. On the other hand, it seems that this is only the 325th most important thing they needed to talk about, so maybe that need wasn’t so great.

And here’s the second of those links:

I agree with sub-heading on this one. The paradox is that Arizona State, Harvard, and other Epstein-associated universities were themselves “rewarding those who exemplify and cultivate intellectual vices.”

And, yes, I’m saying intellectual vices, not just financial and sexual vices. To the extent that Epstein stood for anything intellectually, it was the principle of recirculating B.S. from well-placed elites (as here). Also the above proposed parade of insincerity (oh, sorry, the “mixed strategy”) is an intellectual vice. For that matter, I think it was an intellectual vice for Biggar to misrepresent the position of someone with whom he had an academic and political dispute.

That’s fine. Biggar can be correct in his larger point even if he does not always live up to these ideals himself, and it’s not his fault that he happened to have published on the same website as someone who is a kind of negative illustration of his point. It’s just interesting to see the juxtaposition.

But, hey, for a mere $4500 you can go on a one-week cruise with this guy (that’s Krauss, not Biggar). I think that part of what you get for this equivalent of 3130 Jamaican beef patties is the right to come up to him on the boat and say, “Hey, Lorrie, what’s your position on the statement, ‘You had never fully realized how vulnerable women are, and how the “me-too” campaign reflects decades of oppression and exploitation. . . . You absolutely pledge that all your future behavior will reflect this newfound realization. . . . You pledge to devote the rest of your career to this goal’?” For $4500, the least he can give you is a straight answer.

Only $1000 for the statistician??? What a cheapskate! Especially given that he said the statistician “did an outstanding job.”

Given all the statistical problems in evolutionary biology, maybe he should’ve allocated more of his research budget to the statistician.

Some background on Trivers is here.

The Anthropic Principle in Statistics and Science

The anthropic principle in physics states that our existence implies certain constraints on the natural conditions under which we evolved. In statistics, a corresponding anthropic principle can be used to infer properties of the models we should fit to data. For example, experiments are typically aimed to have a precision sufficient to estimate effects of interest but without overkill; it is rare to have an estimate that is 10 standard errors from zero. We demonstrate through several examples in social and medical sciences how the anthropic principle, combined with Bayesian inference, can be used to improve statistical practice.

Here are a couple of applications of the idea:

• [2000] Should we take measurements at an intermediate design point?

• [2022] A proposal for informative default priors scaled by the standard error of estimates (with Erik van Zwet)

In my talk I’ll discuss these and other examples. I think this anthropic principle is really important, arguably more important in statistics than in physics, which is the field where it originated.

Here’s the zoom information for the talk on Mon 29 June, 4:20pm London time:

https://imperial-ac-uk.zoom.us/j/97341955036?pwd=1kKNbPAwJthKtG55ynXMVF3TLSvIbl.1
Meeting ID: 973 4195 5036
Passcode: J3Ue$f

I’ll be speaking (remotely) at this conference celebrating the 60th birthday of physicist Andrew Jaffe. This seems to be the season for 60th birthday conferences.

I know AJ from when he was visiting the Flatiron Institute last year. We worked together on The Squealer: Sensification of model exploration and model misfit. There’s no connection between the Squealer and the anthropic principle; I decided to speak on the latter topic because I thought it would be of general interest to an audience of physicists.

We’re very excited about this book. It’s the result of several years of effort. You can order from the publisher or from Amazon.

Here’s the book’s webpage, which includes the data and code for the book’s examples and case studies, of which there are many.

Here’s the table of contents:

Part 1: From Bayesian inference to Bayesian workflow
1. Bayesian theory and Bayesian practice
2. Statistical modeling and workflow
3. Computational tools
4. Introduction to workflow: Modeling performance on a multiple choice exam

Part 2: Statistical workflow
5. Building statistical models
6. Using simulations to capture uncertainty
7. Prediction, generalization, and causal inference
8. Visualizing and checking fitted models
9. Comparing and improving models
10. Statistical inference and scientific inference

Part 3: Computational workflow
11. Fitting statistical models
12. Diagnosing and fixing problems with fitting
13. Approximate algorithms and approximate models
14. Simulation-based calibration checking
15. Statistical modeling as software development

Part 4. Case studies
16. Coding a series of models: Simulated data of movie ratings
17. Prior specification for regression models: Reanalysis of a sleep study
18. Predictive model checking and comparison: Clinical trial
19. Building up to a hierarchical model: Coronavirus testing
20. Using a fitted model for decision analysis: Classification competition
21. Posterior predictive checking: Stochastic learning in dogs
22. Incremental development and testing: Black cat adoptions
23. Debugging a model: World Cup football
24. Leave-one-out cross validation model checking and comparison: Roaches
25. Model building and expansion: Golf putting
26. Model building with latent variables: Markov models for animal movement
27. Model building: Time-series decomposition for birthdays
28. Models for regression coefficients and variable selection: Student grades
29. Sampling problems with latent variables: No vehicles in the park
30. Challenge of multimodality: Differential equation for planetary motion
31. Simulation-based calibration checking in model development workflow

Appendices
A. Statistical and computational workflow for Bayesians and non-Bayesians
B. How to get the most out of Bayesian Data Analysis

One way to think of the book is that it’s all the things missing from BDA, like how to set up an informative prior, what to do when your computations aren’t converging, how to work through a series of models fit to the same data, how to design and perform simulated-data experiments . . . and all sorts of other things too.

The core of the book–parts 1 through 3–clock in under 200 pages, and then we have another 300 pages full of case studies demonstrating different aspects of Bayesian statistical and computational workflow. The appendices should be useful to you too, first because the workflow ideas in this book apply to non-Bayesian inference too, and second because BDA still has lots of valuable material in it, so it’s good to know where to look.

This new Bayesian Workflow book could change your life (we hope), and I thank my coauthors, Aki Vehtari and Richard McElreath, with Daniel Simpson, Charles C. Margossian, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, Martin Modrák, Vianey Leos Barajas, for all their care and effort. We thank our employers and various funding agencies for giving us the resources to be able to write this book as a side project along with all our daily responsibilities. And we thank many people for their input on earlier versions of the book, along with the Stan developers making so much of this work possible and the Stan community of users for supplying a continuing series of challenges that have motivated many of the ideas and methods discussed in the book.

I posted this already on the blog and you can see answers to some questions in the comments there. I’m posting it again here because, hey, we don’t come out with a new book every day!

I hope you find the book readable, interesting, and useful.

First, here’s my post from May:

I’m not saying this is the same Scientific American as old. Martin Gardner is long gone, and in the age of social media the articles are shorter. That’s the way of the world. But it’s got serious, interesting articles, a mix of pure science, applied science, policy, and service journalism. The latest in science without the boosterism of so much of science and technology reporting.

Last year, though, the magazine was much more political:

A bit of policy is fine, and there’s a lot of science to global warming, for sure. I wouldn’t want Scientific American to “bothsides” the issue. I’m not saying they need entirely to stick to sports, as it were. But the politicking was getting out of control. I’m glad they’ve returned to their lane.

Then the other day this happened:

Scientific American has been acquired by LabX Media Group, which holds Discover Magazine, IFLScience, and a number of other science publications. . . . And they have started out by firing writers and editors. . . .

I know nothing about LabX Media Group or the new Scientific American management, so I have no sense of whether this is a mere budget-cutting realignment or a full-on Sports Illustrated-style bust-out operation. Martin Gardner is a culture hero and deservedly so, but that was a long time ago, and those days aren’t coming back. Indeed, blogs like these, many of which are Gardner-inspired in one way or another, have taken his place.

It’s funny how magazines, even online, keep disappearing. The model of paying a magazine $50 a year for a subscription and getting a range of interesting material, seems more reasonable than paying $50 each for subscriptions for a bunch of individual bloggers, but, with the exception of the New Yorker, the New York Times, and a few others, we don’t really see much of that.

One way to see this is that I’m not myself a Scientific American reader. I follow all these blogs, many of which are science themed, and each of which, in its own way, goes into more depth than I’d get from a Scientific American article. So there’s this weird thing where I’m concerned about something that I’m not reading anyway. Which is different from Sports Illustrated. Back when Sports Illustrated was a real thing, I’d buy it from time to time. I read it for the articles, as the saying goes.

That said, institutions continue in their own way. I was happy recently to see that Scientific American had pulled itself out of its politicized rut, so it’s a disappointment if it’s now getting taken apart.

If you’re gonna retract something from Max Planck, I’d suggest starting here, with the notorious Manifesto of the Ninety-Three German Intellectuals defending Kaiser Wilhelm’s invasion of Belgium. Here are a couple of retractable passages:

It is not true that the life and property of a single Belgian citizen was injured by our soldiers without the bitterest self-defense having made it necessary.

It is not true that our troops treated Louvain brutally. Furious inhabitants having treacherously fallen upon them in their quarters, our troops with aching hearts were obliged to fire a part of the town as a punishment.

I guess they were the world’s most moral army. “Aching hearts” . . . that must have absolutely sucked. Really mean of those Belgians for defending themselves.

Just to be clear, I’m not saying that Planck should be “canceled.”

Who among us hadn’t retroactively disgraced ourselves with a lachrymose defense of military aggression?

I’m just saying, if you have to retract a paper by Max Planck, I’d retract that one.

P.S. The funny thing is that the above-linked article describes the famous physicist as “almost as widely revered for his character as his physics. In 1933, for example, he bravely confronted Adolf Hitler over Nazi Germany’s discriminatory laws against Jews.” I’ve never read anything about Planck’s life so I don’t know what changed with him between 1914 and 1933. Maybe the loss of the war in 1918 soured him on armed adventures.

I just wrote a long post inspired by a recent post from economist Paul Krugman. Krugman’s post was good, but I’m annoyed that his graph (reproduced above) lists the states alphabetically. Don’t do that! It’s called the Alabama first error.

I would’ve put this as a P.S. on my earlier post but I was afraid that would distract people from my larger point, so I’m just raising the graphical issue here.

If the goal is to have a look-up table, then, sure, alphabetical is fine. But I don’t think that’s the point of that graph. Indeed, if you wanted a look-up table, I’d still prefer a non-alphabetical graph and then you could click to get the numbers in a spreadsheet.

How best to order the states in that graph, then? You could try different things. My first idea is to list in order of average per-capita income by state. (These rankings don’t change much over time; for clarity we could just order by average per-capita income in 2020.)

P.S. All the commenters so far are disagreeing with me, so let me reassess.

I doubt that most readers are looking at this graph to look up individual states. I think the goal is to present the general trend and variation across U.S. states. For this purpose, alphabetical order makes it hard to see systematic patterns that might be clearer using any reasonable ordering.

That said, alphabetical order has the benefit of familiarity, and given that all of you think this is important, I’m willing to believe that my take is a minority view, and maybe the designer of the graph is better off going with the majority.

So I’ll alter my recommendation. Instead of saying, “Don’t alphabetize,” I’ll say, “Supplement with another graph showing the states in a more informative order.”

Mitzi and I were swotting up on structural equation models (SEM) for our class this past Monday at the Modern Modeling and Methods (M3) conference at Fordham University. It was a lot of fun and now I think I understand SEM notation. I really like these applied conferences and this was a group of psychometrician, econometricians, and sociometricians. Many if not most of them thought about models in terms of SEM, so we thought we should figure it out. But I was left with a concern you may be able to help me sort out.

The example

The first worked example in Ken Bollen’s seminal 1979 textbook on SEM is a study of how industrialization relates to democracy. It comes from his paper,

• Bollen, Kenneth A. (1979). “Political Democracy and the Timing of Development.” American Sociological Review, 44(4).

and was reprised in his book

• Bollen, Kenneth A. (1989). Structural Equations with Latent Variables. Wiley.

I had the pleasure of sitting across from Ken at the invited speakers dinner at the conference, so I’m glad I looked into SEM before that. Good news for the SEM devotees—he released a completely revised guide to SEM a few months ago.

• Bollen, Kenneth A. 2026. Elements of Structural Equation Models. Cambridge University Press.

The data and parameters

The data consists of eleven covariates (called “indicators” in SEM) for each of 75 countries. Four of the covariates are related to democracy in 1960 (y1, y2, y3, y4), the same four measurements were taken again again in 1965 (y5, y6, y7, y8) , and there were three measurements of industrialization in 1960 (x1, x2, x3).

The SEM model the original researcher came up with here assumes three latent scalars per country, industrialization in 1960 (IND60), level of democracy in 1960 (DEM60), and level of democracy in 1965 (DEM65). These latent parameters are related in the following way: democracy in 1960 is a regression on industrialization in 1960, and democracy in 1965 is a regression on both democracy in 1960 and industrialization in 1960.

The covariates are then modeled like a seemingly unrelated regression in econometrics. The four democracy 1965 parameters are treated as regressions on the latent level of democracy in 1965, and similarly for the democracy in 1960, and industrialization in 1960.

Rather than independent errors, a SEM model explicitly indicates with arrows which pairs of observations are allowed to have non-zero correlation in the covariance matrix for the observations. The three industrialization observations are assumed to have zero correlation—there are no arrows between any of the three measurements in the SEM diagram. Each of the four measurements in 1960 is assumed to covary with the same measurement taken in 1965. In addition, the second and fourth measurement in each year are assumed to be correlated with each other, which leads to a box-like structure.

The SEM diagram

Here are the arrows in the diagram, where I’m not using their standard LISREL notation, but writing them in R expression syntax to indicate what is regressed on what. In their graphical notation, just replace ~ with <-. All three latent variables and all eleven measurements are indexed by country.

IND60
DEM60 ~ IND60
DEM65 ~ DEM60, IND60

x1, x2, x3 ~ IND60
y1, y2, y3, y4 ~ DEM60
y5, y6, y7, y8 ~ DEM65

The covariance structure is indicated by stating which pairs of measurements are modeled with non-zero correlation. The first four just pair the measurements of the same thing across 1960 and 1965.

y1 <-> y5
y2 <-> y6
y3 <-> y7
y4 <-> y8

The last pair of correlations are within 1960 and within 1965.

y2 <-> y4
y6 <-> y8

Together, these induce an odd box structure, where y2 is correlated with y6 and y4, both of which are correlated with y8, but y2 and y8 are assumed to have zero correlation.

y2 <-> y6
^      ^
|      |
v      v
y4 <-> y8

Stan implementation

We didn’t get this far in my half of the class, so I will share here the Stan Playground example where I fit Bollen’s example (you can get the data and the Stan model through the Playground link:

•   Stan implementation of Bollen’s SEM example.

It gets the right answer compared to lavaan/blavaan, which is nice. In the Stan code, xi is IND60 and eta1, eta2 are DEM60, DEM65. The relation among the latent parameters are modeled directly as regressions. The correlations among the observations are modeled using soft zeroing, where I just put a tight prior around zero on the structural zero elements, because Stan doesn’t give you a good way of setting up structural zeroes in a covariance matrix (Sean Pinkney or Ben Goodrich might know how to do this?).

This makes me curious how the lavaan package in R manages this. There’s a Bayesian version of lavaan built on top of Stan, blavaan. The first example right at the top of the home pages for both the lavaan and blavaan is Bollen’s democracy model. I guess it’s like the Scottish lip cancer data set for spatial modeling or Fisher’s iris data for regressions.

My questions

Consider a simple diagram among measurements like the following.

x <-> y
y <-> z

This says there can be non-zero correlation between A/B and also between B/C, but the correlation between A/C is zero. It’s a simplified case of the box we saw in the actual example. These arrows implies the correlation matrix looks as follows.

|        1  rho[x,y]         0 |
| rho[x,y]         1  rho[y,z] | = Omega
|        0  rho[y,z]         1 |

Given that the correlation matrix Omega must be positive definite, this limits the range of rho[x,y] and rho[y,z]. For example, we can’t have rho[x,y] = rho[y,z] = 0.9, or rho[x,z] would have to be greater than zero to maintain positive definiteness.

Q1: Why doesn’t SEM instead say that the correlation rho[x,z] is just the minimum value it can be given rho[x,y] and rho[y,z]? I’m suggesting that we instead treat the above diagram as implying no additional correlation between x and z other than that implied by the correlation between x and y and the correlation between y and z? That is, why try to shrink rho[x,z] all the way to zero? From the text, it feels like the motivation is to enforce zero correlation in the model. But all this is doing is simplifying regressions—it won’t actually enforce zero correlation among the measurements that are modeled with zero correlation. I wished I’d asked Ken this question at dinner, but I’ll ping him about this blog post and hopefully get a response.

Of course, in the pragmatic Bayesian workflow, we’d use posterior predictive checks to evaluate whether there’s unmodeled correlation between x and z.

Q2: I’m also curious what Andrew and others think about enforcing structural zeroes in correlation between measurements as opposed to just estimating a dense covariance matrix and inspecting where the correlations fall.

In a world where the Department of Justice and the president are either indifferent to or actively support rich criminals, what can be done? Fortunately, there is a range of legal tools that ordinary citizens can use to pursue civilly the sort of corruption that would ordinarily be prosecuted criminally.

The shareholders potentially cheated by Mr. Wiederhorn could sue the Trump inaugural committee under the federal civil RICO law — written to destroy the Mafia — for seemingly helping to secure Mr. Wiederhorn’s freedom. Companies that follow the law can sue rivals, like Binance, that do not, under California’s Unfair Competition Law. And investors scammed by Mr. Milton can sue the political committees he donated to if they were “unjustly enriched” by his scheme. . . .

When regular citizens can’t act themselves, they can pressure their local prosecutors to do so. Recall Mr. Homan’s $50,000 in cash from undercover F.B.I. agents. This Justice Department may not continue the investigation. But Mr. Homan’s personal business is headquartered in Virginia, and it would be awfully interesting to find out whether Mr. Homan reported that money on his state tax returns. If he didn’t, he may well have committed a crime. . . .

He concludes:

Criminals and government officials are barely hiding their schemes, and their brazenness is meant to make us feel helpless, to think that nothing can be done. That is false. We already have the legal tools to fight corruption. We just need to use them.

This is inspirational and I hope someone does all of this.

My point in the present post is that getting justice can require a lot of effort.

Here’s an example. The other day I was talking with someone about research fraud, and he characterized the Michael Lacour story as the biggest scandal ever in political science. I disagreed. It was my impression that Lacour had been forgotten (here’s some background), but what about the time that the American Political Science Association gave an award to a plagiarized book? Here’s the story. I’d never heard of any of the people involved in that episode, but it incensed me that APSA had done this.

I wasn’t the only angry person. Indeed, I’d heard about the Frank Fischer case from Alan Sokal, who’d emailed an academic official at Rutgers University, where the plagiarist worked, but there was no useful response. So I decided to take a whack at it. I sent off this email to the people on the committee that had given that award:

Dear APSA Public Policy Section:

I learned recently that you gave your 2017 Aaron Wildavsky Enduring Contribution Award to Frank Fischer for his 2003 book Reframing Public Policy. I was surprised to hear this, given that the book appears to have plagiarized material. For background, see this document by Krešimir Petković and Alan Sokal:
https://chronicle-assets.s3.amazonaws.com/5/items/biz/pdf/plagiarism_fischer.pdf
and this note by Petković:
https://chronicle-assets.s3.amazonaws.com/5/items/biz/pdf/Petkovic_Experiment_with_CPS.pdf
and this news article for further background:
https://www.chronicle.com/article/alan-sokal-takes-aim-at-an/124969

Petković, a political science graduate student in Croatia, found places in Fischer’s 2003 book where he had used materials from previously published work by others without giving full attribution. In addition to copying without attribution (as Petković writes, Fischer mentions the book he copied from, but nowhere near the copied passage), Fischer also makes mistakes such as misspelling authors’ names and reproduces errors that arose in the original sources.

Two of the works from which Fischer copied in his 2003 book without appropriate attribution are:

Majone, Giandomenico, 1989. Evidence, Argument, and Persuasion in the Policy Process. New Haven: Yale University Press.

Walsh, David, 1972. Sociology and the Social World. In: Filmer, Paul, Phillipson, Michael, Silverman, David and Walsh, David, New Directions in Sociological Theory. London, Collier-Macmillan: 15-35. [Also published by MIT Press, Cambridge, Mass., 1973.]

I am not an expert in this area and have no intention of pursuing any formal process here. Indeed, I am not even a member of APSA. However, I am a political scientist and, as such, am distressed to see APSA promoting plagiarism.

My recommendation is that you retract the award. If that is too difficult, one thing you could do is retroactively also give this award to Majone (1989) and Walsh (1972). It does not seem fair that they did the work and someone else gets the award, no? I do not know Prof. Fischer and am making no judgment regarding the quality of his writing. It may be that it is indeed an enduring contribution to the field; if so, all authors of this enduring contribution should be recognized.

Yours,

Andrew Gelman
Professor, Department of Statistics
Professor, Department of Political Science
Columbia University, New York

P.S. I have also cc-ed the members of APSA’s Committee on Professional Ethics, Rights, and Freedoms.

From APSA’s guide to professional ethics:

“7. Political scientists, like all scholars, are expected to practice intellectual honesty and to uphold the scholarly standards of their discipline.

7.1 Plagiarism, the deliberate appropriation of the work of others represented as one’s own, not only may constitute a violation of the civil law but represents a serious breach of professional ethics.

7.2 Departments of political science should make it clear to both faculty and students that such misconduct will lead to disciplinary action and, in the case of serious offenses, may result in dismissal.”

A few months later I followed up:

Hi all. I was just wondering what happened with this. As I wrote last year to **, I am not submitting a formal grievance or complaint. I just wanted to let the committee be aware of this situation so that they can have the opportunity to fix it.
So I was interested to find out how things have progressed, as it seems to be an embarrassment to APSA to have given a major award for a book with plagiarized material!
Andy

After several months I hadn’t heard back from the committee so I pinged them in June. A couple weeks later they got back to me and said they couldn’t do anything because it had not been submitted as a formal complaint.

Fair enough. I didn’t think it would be right for me to file the complaint myself, given that I’m not at all knowledgeable about this area of political science.

Meanwhile, the books that had been plagiarized, Majone (1989) and Walsh (1972), never got that award. Doesn’t seem fair to me!

Anyway, my point is that it takes work to pursue these things, and it’s more my inclination to point out the problem than to go through the political and administrative steps needed to rectify the problem.

I’m not dissing “the political and administrative steps”–I have a lot of respect for people who can do these things!–it’s just not something that I’m good at.

Here’s another example. I once had a colleague who plagiarized my work. When I realized what was going on, I was stunned. But then, looking back, I realize that I’d been warned of this behavior years earlier, indeed my memory flashed back to a time that I’d seen something else he’d plagiarized from me, and I’d just kind of filed that image in my mind and forgotten it. My collaborator and I had a good thing going, and, hey, nobody’s perfect, so it was easier to look away. When I confronted him about the plagiarism–this was a long time ago–he kind of wriggled around, saying that he didn’t want to share credit with me on the project I’d been working on with him–at one point I was dictating formulas to him over the phone–but we could jointly write a separate article on the topic. This just pissed me off, but, ultimately, he won, in the sense that he correctly calculated that I was rational enough not to want to get involved in a major scandal early in my career. Yes, he’s the one who would’ve looked bad had I raised a formal complaint, but it wouldn’t have done my reputation any favors to be seen as a complainer. Also, though, I won, in that I stopped my involvement in this project and I moved on to better collaborators.

The episode bothered me (which is why I keep talking about it), but my cost-benefit analysis led to the decision to not file a formal complaint. That’s the decision-theory analysis. The game-theory analysis is that my colleague could see ahead to the next move: he know I was rational and that it would be a net loss to me to make a fuss about his actions, and I expect that this minimax analysis led him to the conclusion that he’d be safe in plagiarizing me. Yes, he was taking a risk to his reputation in doing so, but it was a calculated risk, in his mind less than the expected benefit to his reputation of taking full credit for this part of our joint research.

What should be done?

I’m not sure. In academic scandals, maybe it’s best just to move on. So what if some obscure political scientist got some award that he didn’t deserve? So what if some researcher publishes substandard work because he decides to not credit a collaborator? Worse things happen every day in academia. Indeed, if you want to talk about the worst scandal in modern political science, I might give the nod to Samuel Huntington’s book, The Clash of Civilizations and the Remaking of World Order, not because of plagiarism or anything like that, but just because arguably it’s had a large and malign influence in the world. Given all the problems in social science, plagiarism is the least of our concerns. So, although it annoys me, ultimately I think the appropriate strategy is to just let it happen, to talk about it but not to worry about seeking justice.

When it comes to business and government corruption, though, I agree with Ballou that something should be done. Legislatures should be writing laws, local and state governments should be prosecuting, lawyers should be suing, etc. These guys are stealing, giving and taking bribes . . . this is the kind of thing that degrades the entire economic and political system.

So, again, I hope some people make some of the moves that Ballou recommends. They should just be aware that it will take a lot of effort and persistence.

It is a hard problem. Quotas on submissions per author are one avenue forward, which TMLR just announced it would adopt. Not surprisingly, many reviewers are also turning to AI to help. The question becomes how to design AI review protocols to help reduce some of the noise, through preliminary filtering or flagging or helping guide human attention to parts of a paper that are most likely to be problematic. 

But what sorts of checks should an AI review assistant run on a paper? It’s useful to separate basic integrity violations AI could flag, like is there evidence of plagiarism, fake citations, missing code/data to reproduce main results (which are comparatively less controversial) from “epistemic filters,” like does the paper pass replicability checks, robustness checks, preregistration checks, statistical significance checks, etc. There’s a temptation to blur these things in proposing how to apply AI to review. It’s easy to assume that the metascientists have already established that practices like replicability or preregistration are truth-indicating and we can just implement them at scale (and indeed, ML researchers are citing open science and other reform arguments to back their proposals).

But if there’s one lesson to be learned from the aftermath of the replication crisis, it’s that there is no small, stable, non-conflicting set of detectable signals of good science that will find the good stuff and reject the bad. There are heuristics that can be useful prompts for deliberation – get in the habit of preregistering, make sure you can replicate your results, test the sensitivity of your results to choices you made along the way – but things get weird when we start treating them like universal requirements. Authors shift attention away from unrewarded signals, like better theory or exploratory work, and become preoccupied with rigor signaling through their methods. The result is not necessarily more thoughtfulness. 

And so even if the AI review tools we create are simply intended to inform human reviewers about what checks a paper passed, what we implement will have important policy implications by incentivizing more work like that in the future. I don’t think we are in a good position to predict what happens if suddenly we require multiverse robustness or statistical significance in a field like machine learning, which has in many ways been all about iterative improvement and “frictionless reproducibility” rather than individual results passing all the robustness checks.

The answer is not to avoid using AI in review until we can find a non-gameable set of credibility qualities to have AI focus on, as some have recently argued (though I agree with the linked paper that we need more rigor in how we go about motivating review tools). Non-gameability sounds nice, but any automated review policy that allocates attention will be gameable, because ensuring good science is not so simple as finding the right checklist. The relevant question is instead what assumptions and downstream incentives we are willing to tolerate. To this end, at the very least we should get in the habit of spelling out the assumptions we’re making, so that the trade-offs of focusing on particular proxies become explicit.

I wrote up this view recently in a paper called “Stop Treating Metascientific Heuristics as Quality Filters in AI Review.” Here’s the abstract: 

AI-implemented checks for reproducibility, robustness, preregistration, claim scope, and other intended proxies for scientific credibility can extend human reviewers’ capabilities. However, treating metascientific heuristics–whose theoretical grounding remains contested or incomplete–as necessary and sufficient signals for filtering out bad science is counterproductive to scientific progress. The emerging literature blurs the line between integrity filtering, based on necessary but insufficient signals of validity like reproducibility of stated results or lack of fake citations, and epistemic filtering, which uses machine-detectable signals to judge scientific quality. Drawing on critical metascience, we show that commonly proposed signals of research quality are insufficiently justified as general indicators of scientific value. The answer is not necessarily to ban AI in review, given the deluge of submissions venues are facing. Instead, in recognition of how any use of automated signals–even when deployed with human oversight–will shape attention and create incentives upstream, developers of AI review tools should explicitly specify their assumptions about how proxy signals inform on scientific quality in the context of specific review decisions. This approach treats AI review contributions as contestable decision policies that will shape future research, acknowledging the value-laden nature of scientific judgment and surfacing relevant tradeoffs. 

Rather than arguing for or against any particular proxies, I’m more interested in the methodological and philosophical mindset we should bring to the new questions raised by AI review. To demonstrate what I mean by more explicit motivation, I analyze an example review decision problem and set of detectable signals in the appendix, drawing on an analysis of how statistical significance and exact replication success relate to signal-to-noise ratios measured under error from a recent paper by Eric van Zwet, Andrew, and Witold Więcek. The takeaway is that the value of a proxy will depend on how you define the latent state you care about (e.g., whether the direction of an effect was correctly estimated, how big the true signal-to-noise ratio is), what you assume about the generating process (i.e., how the proxy noisily reflects the latent state), and what you assume about the decision-maker’s choice of actions and utility function. By suggesting this approach, I am *not* suggesting that one can validate a new review tool’s utility before its been deployed. The point is that there will be trade-offs no matter what, and the best we can do is be concrete about the kinds of  assumptions that have to hold for proxies to be useful in review, so the community can debate what risks they are willing to accept. 

In this sense, my argument is very much along the same lines as Devezer et al’s argument that those proposing reform procedures should adopt more formal methodology to avoid unwarranted overgeneralization. Once checks become part of review infrastructure, they stop being neutral diagnostics and become policy levers. Let’s start treating them as such in research on AI review.

Campos labels this as, “The Meritocracy!”, and clearly he’s being ironic: his point is that it seems unlikely that these two twenty-somethings are really the people with the most merit needed to run this zillion-dollar company. All things are possible, but it would be an amazing coincidence if, among all the possible financial executives out there, that these two would happen to be the best.

And, sure, I get that.

But now I want to point to my old post on the topic, Meritocracy won’t happen: The problem’s with the “ocracy.”

The short version is that the news item, “Howard Lutnick gives top Cantor Fitzgerald jobs to his sons Brandon and Kyle,” is a very clean example of meritocracy. Lutnick Sr. had the merit (in whatever sense) that took him to the top of the heap, and he used that merit to get jobs for his kids: that’s the “ocracy” part.

If all that merit did was get you top jobs and lots of money, that’s not meritocracy, that’s just merit-based employment and pay. What makes it “meritocracy” that the people with the merit don’t just get nice jobs, they also get to be in charge of everything (”ocracy”). And one thing you do when you’re in charge is take care of your kids!

As Mark Palko discussed over ten years ago, our society seems to have become more tolerant of nepotism. Or maybe the point is that nepotism has always been a thing, but in recent years there’s been more of an effort by rich people and the news media to portray nepotistic hires as having special merit of their own. This is not to say that children of the successful cannot make great contributions themselves—John Quincy Adams comes to mind, also Julian Lennon had that cool song a few decades ago where he sounded just like his dad, so that’s something too. And then there was Oliver Wendell Holmes, Jr., who surpassed his famous father in achievements. And Alexander of Macedon didn’t do so bad either.

Anyway, “meritocracy” implies that the people with merit rule society, and they’ll use their power to help their kids.

Nepo babies aren’t a counterexample to meritocracy, they’re a central part of it.

New preprint To select or not to select: predictively consistent priors instead of model selection with Anna Elisabeth Riha, Leevi Lindgren, David Kohns, Paul Bürkner and me. arXiv.2606.22850

tl;dr: Model selection is not a substitute for building good models in the first place.

Abstract: Bayesian modelling workflows often consider multiple candidate models of varying complexity. Model selection is commonly used to navigate potential trade-offs between model complexity and generalisability to new data. We study when model selection is unnecessary or can even be harmful for predictive performance in finite data regimes and find that the need for selecting simpler models can depend on prior choice. We formalise predictively consistent priors, which keep prior predictive implications stable as model complexity increases. Across examples and numerical experiments, including adding covariates in linear and logistic regression, forward variable selection, and nonlinear modelling, flexible models with predictively consistent priors typically match or outperform selected simpler models in out-of-sample predictive performance. When selection helps, it can indicate poor joint prior implications, such as excessive prior mass on implausible predictive values. Based on our findings, we propose replacing the notion of sparsity or parsimony at the level of model components with specifying priors that remain sensible in predictive space as models become more complex.

These ideas have been around, but there was no single easy paper to refer to explaining and illustrating some important aspects of model selection. Sure, model selection can reduce overfitting, but even better is to use big models and predictively consistent priors.

This is a long (76 pages) slow science paper. I had been showing variants of some plots in my talks years ago, but polishing the explanations and adding more theory took a long time. Anna, Leevi, David, and Paul all did great work on this.

Pretend it is 1980 and we are at the US Census Bureau. We just revamped the occupational coding system, and it’s so much better ! We want 1980-style codes on all our old data that only had 1970-style codes. Let’s trade in our peasant blouses for some shoulder pads.

Say we have double-coded training data (n = 10,000) with:

• O_1980 = occupation coded in the 1980 coding system
• O_1970 = occupation coded in the 1970 coding system
• E = education, either high or low
• I = income, either high or low

We want to impute O_1980 for the single-coded full dataset (N = 1,000,000) with only O_1970, E, and I.

Consider everyone with the a specific occupation according to the 1970 codes, e.g. Accountants. Say there are 200 accountants in the double-coded training data and they have either high income and high education or low income and low education. They have either OCCUP1 or OCCUP2 according to the 1980 codes.

From BDA2 Table 9.1:

Say we use standard regression software to fit p(O_1980 | O_1970 = Accountants, E, I). It will flag the predictors E and I as perfectly collinear, because in the double-coded training sample, education and income are perfectly correlated.

Suppose you drop education and use only income. The single-coded data actually has some low education and high income folks. The model only uses income, so 90% of them get OCCUP1. But suppose I drop income and use only education. My model only uses education, so only 10% of them get OCCUP1. Who is correct ?

As the authors say:

the truth is that we have essentially no evidence on the split for these units… the occupational split for the ‘E=low, I=high’ units should vary between, say, 90/10 and 10/90. … If some variable should or could be in the model on substantive grounds, then it should be included even if it is not ‘statistically significant’ and even if there is no information in the data to estimate it using traditional methods.

Carroll/Langer: Credulous, scientist-as-hero reporting from a podcaster who should know better

7 steps to junk science that can achieve worldly success

A suggestion for Freakonomics and Sean Carroll: Interview Nick Brown

Two researchers in the Harvard psychology department published a paper reporting that they could make people heal faster by telling them that more time had passed. Nick Brown and I looked at this paper carefully and didn’t think that it offered good evidence for its claims. Meanwhile, the paper was promoted uncritically in various media outlets.

As I wrote a couple years ago, to the extent that healing is important, I think it’s important not to overstate evidence for speculative claims about what works. Individual and societal resources are limited. If you want to say something like, “Sure, this is pie-in-the-sky research, but if it works it would be wonderful (‘kind of amazing,’ as physics podcaster Dean Carroll might say), so it deserves our attention, respect, and funding as a high-risk, high-return possibility” . . . go for it. That argument could be made. But then that argument should be made. Don’t fudge it by acting as if there’s evidence that isn’t really there.

Nick and I published an article in a psychology journal discussing the problems with the paper in question, framing it as a more general exploration of how scientific errors can propagate. One of the authors of the original paper then published an article in that journal arguing that we had gotten it wrong and that they really did have strong evidence. Nick and I didn’t find their response convincing on scientific or statistical grounds, but we thought it could possibly be rhetorically effective: just as a piece of writing, if you read it in isolation, it might make you think that we were full of crap. So we closed the loop by replying in the journal, basically restating what we’d said in our earlier article.

The four articles are in different places online and I thought it could be helpful to have all of them in the same place. So here they are:

Peter Aungle and Ellen Langer (2023), Physical healing as a function of perceived time:

In this study we wounded study participants following a standardized procedure and manipulated perceived time to test whether perceived time affected the rate of healing. We measured the amount of healing that occurred across three conditions using a within-subjects design: Slow Time (half as fast as clock time), Normal Time (clock time), and Fast Time (twice as fast as clock time). Based on the theory of mind–body unity—which posits simultaneous and bidirectional influences of mind on body and body on mind—we hypothesized that wounds would heal faster or slower when perceived time was manipulated to be experienced as longer or shorter respectively. Although the actual elapsed time was 28 min in all three conditions, significantly more healing was observed in the Normal Time condition compared to the Slow Time condition, in the Fast Time condition compared to the Normal Time condition, and in the Fast Time condition compared to the Slow Time condition. These results support the hypothesis that the effect of time on physical healing is directly affected by one’s psychological experience of time, independent of the actual elapsed time.

Andrew Gelman and Nicholas Brown (2024), How statistical challenges and misreadings of the literature combine to produce unreplicable science: An example from psychology:

Given the well-known problems of replicability, how is it that researchers at respected institutions continue to publish and publicize studies that are fatally flawed in the sense of not providing evidence to support their strong claims? We argue that two general problems are: (a) difficulties of analyzing data with multilevel structure and (b) misinterpretation of the literature. We demonstrate with the example of a recently published claim that altering patients’ subjective perception of time can have a notable effect on physical healing. We discuss ways of avoiding or at least reducing such problems, including comparing final results to simpler analyses, moving away from shot-in-the-dark phenomenological studies, and more carefully examining previous published claims. Making incorrect choices in multilevel modeling is just one way that things can go wrong, but this example also provides a window into more general problems with complicated designs, cutting-edge statistical methods, and the connections between substantive theory, experimental design, data collection, and replication.

Peter Aungle, Daniel Chen, and Nicholas Holmes (2026), Beyond Statistical Myopia: Replying to a Misguided Critique of Mind-Body Research:

In response to Gelman and Brown’s recent critique of Aungle and Langer (2023), we argue that their article illustrates how narrow statistical reasoning and selective literature review can misrepresent and undermine credible scientific findings. Using their discussion of perceived time and physical healing as a case study, we identify three general problems: (a) a failure to accurately characterize the methods and results of the study they critique, (b) misinterpretations and omissions in their review of the relevant literature, and (c) a tendency to generalize from isolated statistical issues to sweeping claims about the invalidity of mind-body research. We adopt Gelman and Brown’s recommended model and find that the main effect remains robust. We also document errors in their interpretations of other cited studies and demonstrate that they ignore decades of rigorous, well-replicated research on placebo effects and health mindsets. By examining their critique in detail, we highlight how methodological skepticism, when untethered from accurate reading and balanced appraisal, can mislead rather than clarify.

Nicholas Brown and Andrew Gelman (2026), This is the reason for external replication: Response to Aungle et al. (2026):

In an earlier article we addressed a controversy regarding a form of mind-body healing, arguing that a recent paper had overstated evidence from experiments and from literature review. In reaction, one of the authors of that paper disputed our claims. Here we explain why we remain skeptical.

The short answer is that, no, we don’t see any evidence that manipulating people’s subjective experience of time will help them heal better, nor do we see evidence that telling people that they’re exercising will get them to lose weight without their being any changes in their diet or exercise, or various other things claimed in that original paper. I do think it’s possible for researchers, through a combination of sloppy statistics, forking paths, and inaccurate literature review, to create an impression of a strong body of evidence even when nothing is going on–this was a point made eloquently in the classic 2011 article by Simmons, Nelson, and Simonsohn. And I think this combination is enough not just for people to mislead others, but, more importantly, to fool themselves, which can then allow them to spread misunderstanding in the scientific literature, the popular press, and, yes, NPR, Ted, and podcasts.

The whole thing makes me sad, to see researchers caught in a loop of misunderstanding so that, even after their mistakes are pointed out to them, they double down and remain confused. There’s no way that the authors of the above papers will agree with me on this point, and maybe they will find all this to be condescending, but I’m completely sincere here. It makes me sad to see people aim their careers in this direction. The good news is that over the years I’ve received many many emails from young researchers who see this sort of thing going on in their labs and want to do better. I guess the best way to get a grip on this problem is to see how others have been trapped in it.

We are organising a focused workshop on Rethinking the Role of Bayesianism in the Age of Modern AI from October 26 to 30, 2026, bringing together researchers exploring the frontiers of Bayesian machine learning and deep learning. The meeting will take place in Edinburgh, Scotland, UK, and will be hosted by the University of Edinburgh’s School of Informatics.

This workshop follows in the footsteps of the meetings held at Dagstuhl in 2024 and MBZUAI in 2025. This year, the meeting is growing and becoming an official event of the International Society for Bayesian Analysis (ISBA)’s new section on Bayesian AI. We are planning to maintain the collaborative and interactive spirit of the previous meetings, with a programme that includes talks, panel discussions, poster sessions, and ample time for interaction among participants representing a wide range of perspectives and expertise.

Looks interesting!  They should invite Aki for sure.

Someone pointed me to a series of applied economics papers:

1. George Borjas and Nate Breznau, Ideological bias in the production of research findings:

Our study exploits an opportunity to observe 158 researchers working independently in 71 teams during an experiment. After being asked their position on immigration policy, they used the same data to answer the same empirical question: Does immigration affect public support for social welfare programs? . . . teams composed of pro-immigration researchers estimated more positive impacts of immigration on public support for social programs, while anti-immigration teams estimated more negative impacts. The differences arise because different teams adopted different model specifications. . .

The results include an unusual labeling of statistical significance:

Usually it’s one star for p < 0.05, two stars for p < 0.01, as here:

or here:

These are not intended to be authoritative references; they just turned up in a quick search. The point is that 0.05 is the usual standard. Using 0.10 is a way of manufacturing a “statistically significant” result when you don’t have it in your data (as here). In the case of the Borjas and Breznau paper, the data were too variable to get a conventionally strong result, but they still wanted to get it published, and so they shifted the stars. I’m surprised that the reviewers didn’t catch it!

Don’t get me wrong. I don’t think people should be using statistical significance, at any level, as a threshold. To get a sense of my perspective you can read our paper, Abandon Statistical Significance. Even if you have an estimate that’s just one standard error from zero, that’s still evidence of the direction of the effect, as long as no selection is going on.

2. Katrin Auspurg and Josef Brüderl, Fragile Evidence for an Ideological Bias in the Production of Research Findings: Comment on Borjas and Breznau:

Although we were able to reproduce B&B’s numerical results, our reanalysis shows that the reported association is not robust. Specifically, the association hinges on a coding error. Data from four teams that contradict the ideology hypothesis were excluded from the analysis due to idiosyncratic variable coding. Correcting this error renders the ideology effect no longer statistically significant. Also, B&B employed a different outcome variable and weighƟng scheme to that used in a previous paper based on the same data. These two analytical decisions further contribute to the observed ideology effect. Correcting the coding error or using the same specification as in the previous paper renders the ideology effect indistinguishable from zero. . . .

They also go with the 10% significance level, I guess to be consistent with the original paper?

3. Nate Breznau and George Borjas, A Lack of Robustness in Robustness Checking from Auspurg and Brüderl:

In our published paper, we explicitlyacknowledged the limitations of our findings which are based on secondary data and a small sample. After examining Auspurg and Brüderl’s claims, we conclude that they have not presented any new evidence that warrants any correction to our conclusions. . . .

This rejoinder includes the table at the top of this post, in which the significance level has now crept up to 0.20.

I’m anticipating a few more rounds of this, culminating in a table by Breznau and Borjas in which anything with a two-sided p-value of less than 0.5 is given a star. Everybody’s a winner!

P.S. Just kidding in the title of the post. This “p < 0.20" thing isn't really the new rule in econ; it's just something from this one paper. It may be that its authors got some special exemption from the 0.05 threshold.

Lennon’s an interesting case in that he’s had a certain amount of success–early books being published by serious commercial presses and getting respected reviews, and these books made it into stores to the extent that readers such as me came across them), he gets asked to write for the London Review of Books (all they ever publish of me is letters!) and he has a comfortable job teaching at an Ivy League university–but his fiction nowadays . . . ummm, “disappears without a trace” would be putting it too strongly, but readers have to go and search for it. There are just too many people out there who can write well and would like to write for a living, and too few people who want to pick up a book and read a story. The numbers don’t work out.

The above is all background to a weird and kind of mysterious story, which is that there’s someone online who hates Lennon’s guts, but not for any personal reason, just professional grievances of some sort. The person in question is Colin Fleming, and he seems to be, like Lennon himself, a moderately successful writer, which, as discussed, seems like a frustrating position to be in. Fleming has a low opinion of Lennon’s work. That’s fine; literary judgment is subjective. But he’s so angry at Lennon, which just seems odd to me. Lennon’s just some guy, right? Fleming’s blog reminds me of a wacky book from fifty years ago by disaffected journalist Richard Kostelanetz (see some discussion here). I find something fascinating about these cul-de-sacs of literature and publishing–but it’s disturbing to see it happening real time, directed at a real person.

If you want to draw connections, you can note that Lennon once reviewed a book by James Lasdun who once wrote a book about how someone had stalked him. Fleming doesn’t appear to be a stalker; he’s just really angry in a way that seems disproportionate to whatever set him off. At least, that’s my perspective; Fleming seems angry that Lennon has reached literary heights while writing really bad stuff, but, as I see it, Lennon is just getting by–publishing four stories in the New Yorker over a twenty-year period isn’t enough to pay the bills–and I think he’s an excellent writer. I get that Fleming is angry, but it doesn’t seem to me that he’s picking an appropriate target.

P.S. Just incidentally, I think Fleming underestimates the difficulty of coming up with a good title. Coming up with a good title is harder than it looks (unless you’re Donald Westlake). When people can do it, they deserve our respect. When they can’t, they deserve our sympathy, not our mockery. Even some great books have mediocre titles.

P.P.S. Just for fun, here’s a review by Lennon of a recent book by Stephen King.

Eric Novik came by my talk the other day and we were chatting about a number of things, including how much we forget as the years go by. I remarked that I used to be very comfortable with Fourier analysis and was able to use it as a research tool—see section 2.2 of my Ph.D. thesis, and it also came up in my research leading to R-hat (although it didn’t make it into the writeup)—but at this point I only understand Fourier analysis on a conceptual level. It’s not one of these things that stuck with me.

In response, Eric pointed to this app that he created (with chatbot assistance) to help him my understand some things about Fourier series. Maybe it will be useful to some of you too. The source code is here.

This came to mind with respect to three once-famous people: Gray Davis, Grover Norquist, and a rabbi.

Act 1 (2021-2022): I receive emails from some sort of, ummm, I don’t want to call it a “scam” exactly . . . let’s call it a “networking event,” featuring luminaries such as “Gray Davis – Of Counsel, Loeb & Loeb. Fmr. Governor, California. [Los Angeles],” “Grover Norquist – President, Americans for Tax Reform. [Washington, D.C.],” and “David Wolpe – Rabbi, Sinai Temple. [Los Angeles].”

It seemed to be a great opportunity–just look at the email:

Hello Andrew,

We’ve heard a lot of great things about you, which is why you’ve been selected for membership. Dialog members–ranging from scientists to elected politicians, CEOs, artists, economists, media figures, and political dissidents–regularly convene to intellectually challenge each other in off-the-record conversations exploring pressing issues. We think you’d add an exciting perspective!

On the other hand, they were charging $16,846, which, as you may have heard, would cover the cost of a lot of Jamaican beef patties.

If they’d really heard a lot of great things about me, and they thought I’d add an exciting perspective, you wouldn’t think they’d charge me for the privilege, right?

I asked the organizers, who replied:

I absolutely get it; the majority of those who are invited to Dialog typically only attend conversations or gatherings as the keynote speaker, and if money is involved, it’s typically because they’re being paid to attend.

To keep Dialog fully independent and off the record, it is 100% participant funded–everyone who attends pays to do so.

Wow! So Gray Davis, Grover Norquist, and the rabbi were paying thousands of dollars to mingle with each other? It kinda makes you wonder. One of the other listed members was as the “Turki Al Faisal Al Saud, Former Minister of Intelligence, Saudi Arabia”–no, I’m not kidding there! I wonder if they let him take the bone saw on the plane? I bet he had a great conversation with “Zeke Emanuel – Vice Provost for Global Initiatives, Professor & Chair, Department of Medical Ethics and Health Policy, University of Pennsylvania.” And what about “Lawrence Summers – President Emeritus & Professor, Harvard University. Fmr. Secretary of the Treasury, United States”: did he really pay? It’s hard for me to imagine Larry paying for anything out of his own pocket. Maybe he got some friendly Harvard donor to fork over the money?

As I discussed in the above-linked blog post, I could see reasons why Gray Davis or Grover Norquist might want to talk with me, and I could see reasons why I might want to talk with Gray Davis or Grover Norquist, but I can’t figure out why each of us needs to spend $16,846 to do it. We could just talk on the phone for free!

Act 2 (2026): This arrives in the inbox:

I am reaching out on behalf of the WIRED team. We are working on a story about Dialog, the private, invite-only organization co-founded by Peter Thiel.

WIRED has obtained internal Dialog records, exposed by its website, including a membership directory and the registration list for the group’s 2026 retreat. Your name appears in them.

We wanted to give you the opportunity to comment before we publish. We’d welcome any response, including whether you’d confirm your affiliation with Dialog and anything you’d like to say about the group or your involvement.

Our deadline is 1pm EST, but if you’d need more time to prepare a response, please let me know as soon as possible.

As many of you know, I never check my email before 4pm. It’s actually daylight time here in New York, not standard time, but either way it’s before 4.

In any case, another email arrived soon after:

My sincerest apologies for this mixup! Please ignore our previous email. Your name was mixed up with a list of Dialog attendees we are trying to reach. We’re actually reaching out because we saw your 2022 blog post about being invited to the event, wanted to mention it in our story, and thought you should have the opportunity to comment on it, if you wanted to.

That evening I saw the message and replied:

Hi, sure, feel free to quote me. I stand by what I wrote before. I’ve never actually attended the Dialog event, as I have better uses for my $16,000.

The news article appeared soon after, under the title, “Leak Exposes Members of Peter Thiel’s Secretive ‘Dialog’ Society,” with subtitle, “More than 200 of the world’s elites registered for a retreat whose agenda runs from panels on cult-building and sex to prepping for World War III. An associated app offers matchmaking.”

Wow–I had no idea! I have to say, the idea of seeing Gray Davis, Grover Norquist, and a rabbi talking about cult-building and sex . . . ok, still not worth $16,846, but maybe there’s some entertainment value there.

Act 3 (2026): The story was picked up by other news organizations. I know this because a few of them contacted me directly and asked if I had anything more. I forwarded them three of the emails I’d received back in 2021 and 2022. There was also a story in the Hollywood Reporter (Palko pointed me to it) mentioning anti-free-press warrior Peter Thiel and a bunch of movie stars and executives, the most notable of whom was Benj Pasek, one of the composers of the music for La La Land.

It’s hard for me to picture Benj Pasek forking over $16,846 for the opportunity to mingle with Gray Davis, Grover Norquist, a rabbi, and the head of Saudi intelligence. But maybe his agent paid for it? I dunno.

Act 4 (2026): Here’s what I’m wondering. What do Gray Davis, Grover Norquist, and the rabbi think about all this? Each of them is a bigshot in his own field (failed politician, political lobbyist, religious leader), but none of them is important enough to be mentioned in any of these news articles.

How humiliating!

There was a time when the name Gray Davis meant something, a time when Grover Norquist had armies at his command, a time when a rabbi could call down thunderbolts. And now they’re just anonymous names in a list. What a comedown. Here’s my advice to these three guys: Fire your publicist.

I hope at least that they enjoyed the conferences. $16,846 is real money!

The only thing I don’t get is why the news organizations are making such a big deal about all of this. It’s an annual conference where rich guys spend thousands of dollars to be in each others’ company. No joke, it doesn’t sound much different from a country club.

And there’s this whole bit about the membership list being a secret. I don’t get why this is supposed to be a big thing either. Country clubs keep their membership lists secret too–it’s part of the whole exclusivity cachet. They’re not the public library, y’know!

Summary

Gray Davis, Grover Norquist, and a rabbi got the worst of all worlds. They had to go to a boring conference, they paid $16,846, they got no press coverage out of the deal, and they didn’t get any Jamaican beef patties.

I have no idea what food they serve at the Dialog conferences. I’m guessing it’s standard crappy upscale catering food, nothing nearly as good as you could get for $2.85 at Golden Krust here on 125 St.

I’ve been planning for at least a couple years to generate a case study around Galielo’s use of an inclined plane instrumented with water clocks to estimate the terrestrial gravitational constant. Here are some photographs of a replica in the Museo Galileo (click to blow them up). And here’s a video simulation of the experiment. We replace his clever pendulum apparatus explained in the video and the web page with simple Bayesian statistics so we can actually estimate the gravitational constant.

The case study

Here is a draft.

Bob Carpenter. 2026. Estimating g from Galileo’s Water Clock: A scientific Bayesian inverse problem with Stan and CmdStanPy. GitHub.

I list myself as the author here because I’m responsible and AIs can’t own copyright in the U.S., but 100% of the text and code was written by Claude Opus 4.8 (medium or high effort, but I can’t recall which). I used the desktop app, which doesn’t allow sharing, but you can try it yourself.

The prompt

Here’s the sloppy prompt I used, which I just typed in without much thought in a couple minutes to get a feel for what it could do on its own.

I would like to generate a case study written in Quarto and using CmdStanPy to demonstrate solving scientific Bayesian inverse problems. I want to use a simulation of Galileo’s water clock experiment, which can be used to estimate the gravitational constant. I would like you to start by generating the mathematical model description in LaTeX, the model code in Stan to solve the inverse problem, and a simulation driver in Python using CmdStanPy and plotnine for plotting. Please just `import plotnine as pn` and use `pn.geom…`, etc. All I need in the output now is a call to `.summary()` on the fit returned by `.sample()`. Wrap this all up in a quarto document for me from which I can generate HTML by calling `quarto render galileo.qmd`.

It was done before I got back to my desk with a cup of coffee (well under five minutes). So not quite the several hours Andrew said it took him to write his case study on the New York Knicks basketball team, which he posted earlier today. Of course, this was much simpler and I didn’t have to think through any details before generating it.

Is it right?

What Claude produced looks really good to me. If a student had done this, I’d given them an A. I can’t object to the way it described Galileo’s experiment, wrote the math, wrote the Stan code, wrote the Python simulation, or plotted the raw data as Andrew is always urging us to do.^*

The source

You can find the source .qmd file on my GitHub:

https://github.com/bob-carpenter/case-studies/tree/master/galileo-gravity

It’s short, so I would have just included it, but the blog software blocked my post after considering it an attack on the site. To get it to render with resources embedded, I had to ask Claude a follow-up question and manually insert a single line of config into the .yaml header for the markdown document.

Putting this blog post together took longer than writing the prompt and checking the results.

^*   Maybe Claude runs a little simulation of Andrew like I do. Andrew himself claims to run a simulation of Jennifer Hill—it’s the basis of his
handy statistical lexicon entry for “WWJD,” which he told me stands for “What would Jennifer do?” Unfortunately, neither the lexicon entry nor its underlying link explains the acronym.

Nowadays, I guess that much of this business has been supplanted by machine gambling that you can do on your phone in the comfort of your own home. But the market for gambling must be far from being tapped: I imagine that there are many millions of potential gambling addicts out there, available to be hooked by some form of gambling or another.

As a statistician, I have mixed feelings about gambling. Ever since I was a kid, I’ve thought that probability is cool, and I like to bet. When we were kids we had a toy roulette set that we would play (just betting chips, not real money) and I’ve enjoyed poker and informal sports betting. The last time I’ve bet on anything was about 20 years ago, but that’s just more me getting older than anything else.

At the same time, there are all these addicts, and all the people who might not be addicts but who still degrade their standard of living, not to mention reward evil people (even if they’re pleasant as invididuals, they’re in an evil business; sorry, Nate!). And it just keeps getting worse.

To a statistician, this is all an endlessly fascinating topic: the odds and all that, but also whatever it is in people’s brains that motivate them to spend thousands of dollars on lottery tickets, etc.

As Schüll writes in her book, the popularity of machine gambling (which she says is the source of the majority of casino gambling profits in Vegas) is particularly puzzling in that people are just pulling the lever over and over again, without the sense of human context or any feeling of agency.

There’s also the interaction between the players and the people who make money from the machines:

For extreme machine gamblers, the experience of play is an end in itself–an “autotelic” zone beyond value as such, in that “no other reward than continuing the experience is required to keep it going.” Conversely, for the gambling industry the zone is a means to an end; although it carries no value in and of itself, it is possible to derive value from it. . . . In effect, gamblers’ drive to remain indefinitely suspended in the zone is rerouted, via the technological detours of the gambling industry, toward a destination of complete depletion.

It’s not just “the technological detours of the gambling industry,” it’s also politics: the industry doing what it takes to keep all this going, a gradual effort over many decades that continues to this day.

Later, Schüll summarizes:

Gambling addicts play machines to suspend themselves in a state of equilibriated affect.

This seems pretty accurate.

I would just add two things.

First, this equilibrium is not flat. It’s periods of stress, punctuated with the occasional excitement of winning and the frequent relaxing calm of losing. The best analogy I can think of is the way that a baby is calmed, not by lying completely still, but by being rocked in a somewhat irregular fashion.

Second, stakes matter. That “state of equilibriated affect” can only be achieved when real money is involved. I guess this is related to the phenomenon of habituation in drug exposure. Schüll talks with someone who started on a zero-stakes poker video game but them moved to the machines that take real dollars. We discussed this general idea recently in our post, Why isn’t it possible to play a fun and serious game of poker not for money?

It’s a good thing that babies don’t work that way–you can rock them a reasonable amount and they’ll be happy. No need to keep upping the stakes until the crib does a loop and the baby flies out the window. Although I guess that might happen if there were money in it.

With Leonard’s reputation as a Western author growing, [Detroit-based advertising agency] Campbell-Ewald saw fit to match Leonard with their truck division, writing copy geared toward the same rough-and-tumble demographic that, essentially, would read like a Western paperback. “Truck ads I had an easier time with,” he later admitted. “You could be straightforward with a truck . . . I’ve never been any good at similes and metaphors.” Much like his father before him, Leonard was soon sent traveling around the country for company “field work,” gathering customer testimonials from satisfied truckers. “I would call on the Chevrolet dealer, who would then introduce me to a truck owner who had some fantastic story to tell about his trucks,” he would later claim, prompting the owner to “say something colloquial,” in the hopes of shaking loose some down-home phrases to tinker with. However, Leonard’s favorite–“You don’t wear that sonofabitch out, you just get tired of looking at it and buy a new one”–proved too gritty for Chevy.
— from Cooler than Cool: The Life and Work of Elmore Leonard, by C. M. Kushins.

As the above quote illustrates, this is an interesting, well-researched, and well-written biography, much better than the biography of John D. Macdonald that we discussed a few years ago. Kushins begins with a brisk and effective overview of Leonard’s childhood and then moves quickly into the career, bouncing between the style and themes of Leonard’s stories and books; the details of writing schedule, agents, and contracts; and enough of his activities outside of the writing to give a sense of how his life fit together. Thanks to Leonard’s long and stable career, Kushins is also able to spread the details uniformly through the decades, unlike for example any biography of J. D. Salinger, which won’t have much to say for the final decades of that writer’s life.

The main weakness of Kushins’s book for me is that it doesn’t talk so much about the novels themselves. There’s a lot on how they were written and on their general themes (good guys and bad guys, the roles of the women characters, religious themes, some other things) and on their style (notably, Leonard’s move from Westerns to crime capers and his ear for dialogue), and on movie adaptations and helpful literary agents and how he did his research and where many of the character names came from and all sorts of fascinating things–overall I enjoyed the biography and I recommend it–, but I would’ve liked to see more actual literary criticism, some detailed discussions of the novels and what made them work, as well as, sometimes, what didn’t.

I first learned of Elmore Leonard around 1981, it must have been from a book review in the Washington Post. Phil and I read a bunch of his books with pleasure–my favorite is Swag, which I actually read a few years later–and I also learned about George V. Higgins, an author to whom Leonard was often compared. Over the years, I’ve read almost everything by Higgins that I could find.

Elmore Leonard vs. George V. Higgins . . . what to say? Leonard had a long and successful career, whereas Higgins started at the top and worked his way down. And on a sentence-by-sentence level, Leonard was a better writer: Higgins had a lot of clunky sentences and was notorious for not rewriting. But I think that, of the two, Higgins was more of an artist. There’s something special about Higgins that makes me really love his writing, despite the flaws. Leonard was great too–I think Swag is a close-to-perfect crime novel–but, I don’t know, I don’t have the same feeling of being transported. I want to say that Leonard is Wings and Higgins is the Velvet Underground . . . no, that’s not quite right . . .

What else? Both Leonard and Higgins wrote about loquacious lowlifes. Leonard wrote with more affection, Higgins with more cynicism, but both had a habit of playing favorites with their characters, liking some and finding others irritating. Which can lead to some absolutely wonderful things, such as the pitch-perfect final line of Swag.

The other thing is . . . oddly enough, neither Leonard nor Higgins had great plots, or great characters. A crime novel needs a plot, and both authors’ plots were serviceable, often excellent in the details (for example, the robberies in Swag and The Friends of Eddie Coyle), but for both authors the plots weren’t much more than vehicles to allow for stunning set-pieces of dialogue and the development of themes of friendship, betrayal, etc. As to the characters: it might seem odd to describe these authors’ characters as empty, given that they were portrayed by great actors in memorable films, but . . . ok, let me put this more carefully . . . I wouldn’t say the main characters in their books are one-dimensional, but rather that they are blank. Not completely blank, of course–they have characteristics–and they have a lot more personality than the killers in Agatha Christie books–but not what I’d call memorable characters.

If it’s not the plots, and it’s not the characters, then what are we reading Leonard and Higgins for? The juicy dialogue, sure, but also the situations. Swag doesn’t have an elaborate plot, but the setup of these two criminals with rules for robbery, that’s great. Similarly with The Switch: it’s a great setup. Or The Digger’s Game, with all the events spooling out with a sense of inevitability. I guess you could label all of this as “plot,” but these are not cool plots in the manner of The ABC Murders.

OK, so here you have it: Leonard and Higgins place real (if sometimes blurrily-defined) people into compelling situations, and they make it all run on sharp, hilarious, compelling dialogue. This is actually very cinematic! It’s the setup more than the plot, but the setup only works because you’re throwing (some version of) real people into it.

And why do I find Higgins more compelling than Leonard? Because with Higgins the stakes are higher. Not just that it’s life and death–lots of bodies hit the floor with Leonard too–but, even in the presence of humor, Higgins’s stories are ultimately more serious.

OK, back to the Kushins book, which, again, I like a lot. I’m just bummed that he doesn’t engage with Leonard’s writing, even to the extent that I do above, or to the extent that those book reviewers did, 45 years ago. I’m not saying that Kushins has to say that Leonard isn’t as good as Higgins–he’s a Leonard fan, and I’d expect him to make the case for the author–; I’d just like some discussion of the novels themselves along with all the fascinating details of how they were constructed.

I often enjoy literary biographies and I appreciate that new ones keep being published, given that they must not sell a lot of copies! That said, it seems likely that Elmore Leonard will outlast some other great once-bestselling authors. Younger readers still appreciate his books, at least for now. Higgins, though, unfortunately I think he has no chance. He’s an innovator and I don’t think he’ll ever be completely forgotten, but I think his books are a little too hard to read to ever sustain a revival.

Maybe it helps to write in genre. Readers of crime and science fiction seem loyal to past bestsellers in a way that we don’t always see with mainstream literature.

P.S. I also recommend this review by J. Robert Lennon of a few of Leonard’s books. I don’t agree with everything Lennon says, but that’s fine; it’s what I was looking for, which is a serious engagement with what makes Leonard’s books work. My main disagreement with that review is that Lennon says that Leonard’s strength is creating memorable characters, whereas I think that, as with many crime and suspense novelists, what Leonard does best is to create memorable situations and then work out their logical implications.

Congratulations to the R Core Team on receiving the 2026 Rousseeuw Prize for Statistics.

R has made creative, open-ended statistical analysis and graphics accessible to generations of statisticians and applied researchers. It has also been central to statistical research, methodology, and applications during decades when statistics became more computational and more important across science, engineering, business, and public health.

One of the great strengths of R is that it is not just a software platform. It is also a community. The system of R packages allows anyone to implement a new method and share it with the world, helping make statistical research more open, useful, and alive. R has also been the medium for major developments in statistical graphics, transforming applied statistics and the way people work with data.

The volunteers who have developed, guided, and maintained R and the R community are richly deserving of this major award.

I agree with the committee that the R team is an excellent recipient of this award. I say this for several reasons:

– Most obviously, R is super-useful and it’s changed statistics, both by enabling more complicated and reliable analysis and by establishing a common language for statistical coding.

– R integrates statistical modeling with graphics, which traditionally (but, in my opinion, mistakenly) have been thought of as in opposition.

– R is open source. This might sound like no big deal, but its predecessor was Splus, which was a commercial package. Before that came S, which was open but was not set up to expand in a scalable way.

– With its system of packages, R became modular: different groups of users (including me!) could write their own packages and develop new and useful tools without needing to get tangled in core R issues. For example, we have cmdstanr, which lets you run Stan programs from R. This is super-useful for Bayesian workflow.

– R is a programming language, not a menu-based set of commands. This is no big deal now, given that the natural comparison to R is Python, but, back in the day, when R’s competitors were Sas, Spss, Stata, etc., it was a big deal that with R you write programs, you don’t just push buttons. A big deal for workflow in statistics and data science.

– Regarding the R community . . . ok, this gets complicated. Still and all, the R core team is very helpful to outsiders and has been a clear net benefit to the communities of developers, statisticians, and users.

I’m sure I’m missing a few things. My only disagreement with the award citation is that it doesn’t mention S, the statistical software environment developed by John Chambers and others at Bell Labs back in the 1980s. R is a rewrite of S. With lots of improvements, but I do think the S team deserves credit for setting up the template.

As a member of the BayesComp 2027 conference committee, I would like to share the announcement of the call for invited session proposals for the upcoming BayesComp conference, which will be held in College Station, Texas, on May 18–20, 2027.

The scientific committee is currently soliciting proposals for invited sessions. Each invited session will consist of three speakers, and proposals should focus on timely, important, and broadly engaging topics in Bayesian computation and related areas.

The submission deadline (as of now) for invited session proposals is August 15, 2026.

Proposal form: https://forms.gle/wpYvkkjKGZ5vHqhF6

Additional details are available in the official announcement:

The LOC for BayesComp 2027 is pleased to announce that the next edition of BayesComp will take place in College Station, TX during May 18–20, 2027. The scientific committee is now opening calls for invited sessions. Each invited session will consist of 3 speakers. Proposals should highlight timely, important, and broadly engaging topics in Bayesian computation and related areas. Each speaker may be listed as a speaker in only one invited or contributed session proposal.

Lu is the first author on the Pathfinder paper and continues to do interesting work on Bayesian statistics and computing. Based on what I’ve heard about past BayesComps, the conference should be really interesting.

One way I celebrated was by reading Lax & Phillips 2009, Gay Rights in the States: Public Opinion and Policy Responsiveness. It’s on-theme, an example in the MrPlew paper (which I also still need to digest), and I wanted examples of using MRP in later analyses.

Lax & Phillips 2009 studied the relationship between state-level public opinion and state adoption of policies affecting gays and lesbians. Andrew blogged about this work in Nov 2008, Jan 2009, and June 2009 when he wrote:

Fancy statistical analysis can indeed lead to better understanding. Jeff Lax and Justin Phillips used the method of multilevel regression and poststratification (“Mister P”…

The paper’s appendix includes a NYT article and an almost-rainbow-colored plot:

Lax & Phillips 2009 used MRP to estimate state-level public opinion E(y | s). Let

• y_i = 1 if person i supports laws to protect against discrimination in job opportunities (for example), = 0 otherwise
• s[i] = state where person i lives, e.g. NY
• L_s = 1 if state s has laws to protect against discrimination in job opportunities, = 0 otherwise

Their Multilevel Regression (“MR” of MRP) model had race, gender, age, education, state, and poll effects:

They modeled the state effect with state-level predictors (% religious conservatives, % Democratic voters in 2004):

Then they Poststratified (“P” of MRP) to the population:

Then they used the MRP estimate of public opinion as a predictor of whether the state adopts the policy:
Pr(L_s = 1) = logit^-1(a + b * y_s^pred)

From their Figure 1:

Questions:

1. (How) did Lax & Phillips 2009 incorporate uncertainty in the MRP estimate of public opinion y_s^pred in their later analysis of its effect on policy adoption L_s ?
   Footnote 7 says they incorporated uncertainty for non-MRP estimates:

   if we use an opinion index based on disaggregation instead of MRP estimates, correcting for reliability using an error-in-variables approach (eivreg in Stata)…

2. Are results sensitive to whether policy adoption L_s is a state-level predictor in the MRP model ?

1. The Knicks’ stunning series of come-from-behind victories to win the NBA title in 5 games;

2. The martingale property of probability forecasts;

3. An example of learning from simulation;

4. How we (sometimes) do research in probability and statistics.

I don’t know enough about this blog’s audience to know which of the four topics will appeal to most of you. For the internet as a whole, it’s #1; for most of you, it might be #3.

I’m interested in all four, which is why I’m writing this all up right now. I’m embarrassed to say that it took several hours to do this. I was originally planning to post this Sunday morning after the game but it took time for me to get to the task. Most of the effort came from writing the code, not from writing the text. And there’s actually not much code, as you can see if you scroll to the end of this post. The main effort was not figuring out the syntax or even debugging (although there was some of that) but in working out what I wanted to be coding in the first place.

On the plus side, this is research I’ve been wanting to do for awhile, so (a) I don’t think this effort is wasted, even beyond whatever educational and entertainment value if has for you, and (b) I learned a bit from this already. Looking at data is always good; experimenting with simulation is always good.

Ok, here goes.

The NBA finals

Hey, remember this, from game 4 of the recent NBA finals:

Or the trajectory of the game that came after:

Just for completeness, here are the traces for games 3, 2, and 1, also courtesy of ESPN:

In game 4, the Spurs at one point were estimated to have a 99.6% chance of winning. But, as you might have heard, they lost.

Extreme win probabilities

Were those stated win probabilities too extreme?

On one hand, sure, unusual events happen on occasion. If you have a 0.4% chance of losing, that’s something that should happen 1 in 250 times, and there were a lot more than 250 basketball games just in this past season. On the other hand, very unusual event are supposed to happen only very rarely, and there was a point in the third quarter of game 4 where ESPN’s algorithm gave the Spurs a 97.1% chance of winning, a point in game 1 where the Spurs were given a 94.1% chance. There was a moment in game 2 where the Knicks were assigned a 98.2% chance of winning, and, sure, they did win that one, but given that the final score was 105-104, after being tied 97-97 and 104-104, it seems in retrospect that this 98.2% was a bit overconfident.

Should we be suspicious of these probabilities? One way to ask this question is to check calibration: if we collect all game situations where a team has a 99.6% of winning, are they winning 99.6% of the time?

On the other hand, I’m picking the most extreme values of these win probabilities. You should get calibration of win probabilities at any time, and it’s ok to condition on them, but only to condition on what came before.

That is, if we look at win probabilities at the end of the first quarter, or at the end of the first half, or at the end of the third quarter, they should be calibrated. And if you look only at win probabilities only when they’re greater than 99%, they should be calibrated. And if you look only at win probabilities when they are the maximum for the game so far, they should be calibrated. But it’s not clear to me that you should expect calibration for win probabilities selected to be the maximum for the entire game, because if the win probability at time t is p(t), and you condition on the event p(t) < p(t_0) for t > t_0, that could provide information. It’s tricky.

The martingale property of probability forecasts

We wrote about this in section 1.6 of our 2020 article, Information, incentives, and goals in election forecasts:

And it also came up in some blog posts:

from 2020: Do we really believe the Democrats have an 88% chance of winning the presidential election?

from 2020: More on martingale property of probabilistic forecasts and some other issues with our election model

from 2024: “Unusual Betting Patterns With Several Temple Games”: It’s martingale time, baby!

also from 2024: It’s martingale time, baby! How to evaluate probabilistic forecasts before the event happens? Rajiv Sethi has an idea. (Hint: it involves time series.)

I’d expect ESPN’s win probabilities to be closer to calibrated than prediction-market odds or model-based election forecasts. Prediction markets depend on the bettors and there’s no reason to expect calibration, at least not until the market is fully mature in some way. Model-based election forecasts are based on approximate models that have known pathologies (for example here), so they won’t be universally calibrated. ESPN’s probabilities won’t be calibrated either–they too are based on an imperfect model–but I assume it’s model has been trained on tons of data so I don’t think it should be far off.

If someone could send me the moment-by-moment estimated win probabilities from some large database of basketball games, we could take a look.

In the meantime we can get some intuition by simulating from a mathematical model where we can compute win probabilities exactly.

Simulating the process

Assume a simple Brownian motion with drift, where the score differential y(t) starts at y(0) = 0 and then takes a continuous random walk so that y(t) ~ normal(delta*t, sigma*sqrt(t)). We’ll scale t to be in minutes, so the game goes from t=0 to t=48, with the winner being determined by y(48). The drift is then delta=point_spread/48, because this is the expected final score differential before the game has started. And we’ll set sigma=2, which seems reasonable: 2*sqrt(48)=13.8, so that the sd of the final score differential is approximately 14 points.

One cool thing about this model is that the win probability can be trivially computed given the score differential at any point in the game.

How wrong can you be?

To demonstrate, I’ll show the results–the score and the win probability during the game–for 18 independently simulated games. For simplicity I’ll assume the point spread is 0, so the two teams are always assumed to be evenly matched. And I’ll step through the game 10 times per minute, thus approximating the game as a sum of 480 independent increments.

The code is below; here are the results:

I don’t know enough about basketball to have a sense of how plausible these are as game outcomes (setting aside the lack of discreteness in the score; we used a continuous model so that we could more easily compute the relevant probabilities analytically). They don’t look too much like the Knicks-Spurs game except for that one simulation near the lower left of the plot, where the “Spurs” led by 10 points into the third quarter, maxing out with a win probability of 95.6% before eventually losing.

To get a broader picture, I simulated 10,000 games. (Just as a reference point, there are 30 NBA teams, so there are 82*30/2=1230 regular season games each year.)

For each game, I computed “max_p_wrong”: the highest win probability assigned to the game’s eventual loser. In my simulation, every game starts with a 50/50 probability–remember, for simplicity I’m always assuming a point spread of 0–so max_p_wrong must be somewhere between 0.5 and 1. Here’s what comes out:

So, extreme wrong probabilities are not unheard of. How common are they? Out of these 10,000 games, 61 had max_p_wrong greater than 99%. That is, in 0.6% of games, the eventually-losing team exceeds the threshold of 99% win probability during some point in the game.

This result should go up if we move to continuous updating. But we’re already updating 10 times a minute. Increasing this schedule to 50 times a minute increases Pr(max_p_wrong > 0.99) to 0.0075, and increasing to 100 times a minute takes it to 0.0076, so my guess is that this is roughly the continuous limit.

OK, just to check, I’ll simulate 100,000 games, and now Pr(max_p_wrong > 0.99) is 0.0072 with 10 updates a minute, or 0.0084 with 50 updates per minute. So I’ll go out on a limb and say that if we were to compute the exact probability under continuous updating, we’d get 0.0085.

This was a surprise. Before doing this simulation, I was assuming that the probability of p_win exceeding 99% in for the eventual loser at any time in the game would be more than 1% because of selection. I guess my intuition was wrong. Maybe it has to do with the fact that I’m conditioning on which team wins. (Of course, if you go the other way, the probability of p_win exceeding 99% for the eventual winner is 100% in the continuous limit, because with epsilon of a second left in the game the winner will almost certainly be known.)

So, yeah, the above graph is kind of interesting. Under our model, most games won’t stray too far into retrospectively-embarrassing probability estimates, but it can happen sometimes.

It would be interesting to compare the above graph with what you’d get from a database of game-odds data from ESPN or whatever.

Just to be clear: there’s no reason to think that the above graph represents any sort of universal property of martingales. It’s a very specific model! But you have to start somewhere. Also, the existence of various central limit theorems makes me hold out the hope that this could be a general result under some appropriately restricted class of continuous martingale processes. It’s a research question!

A surprising uniform distribution

To get some further understanding of the process, I gathered the win probabilities after the end of each of the three quarters for the 10,000 simulated games. Below are histograms of these probabilities and calibration plots:

Unsurprisingly, the calibration is fine. After all, the probabilities are computed from the same model that the data are drawn from. Indeed, even the apparent anomaly in the lower-left plot is just a small-sample artifact which disappears when we up the number of simulations to 100,000.

More interesting are the histograms. It makes sense that, as the game goes on, the distribution of win probabilities starts at 0.5, then gradually bunches up at 0 and 1. Indeed, at the end of the fourth quarter the win probabilities are exactly 0 and 1.

But it’s funny how the distribution of win probabilities is exactly uniform at halftime. There must be a direct mathematical argument giving intuition for that result; it’s too perfect to just be an accident.

Lots more research to be done here:

– Generalizing beyond the continuous model to allow discrete scoring changes.

– Generalizing beyond the random walk; there’s no reason the model needs to be Markovian.

– Are there general statements that can be made about these distributions of win probabilities under arbitrary martingale processes? I’m guessing there are some results. At least, there should be some inequalities and limit theorems.

– Looking at real data from basketball, other sports, and other realms, including election forecasts and prediction markets.

Our ultimate aim here is to come up with a general measure of departure from the martingale property of probability forecasts. We want something that can be applied to any dataset, obviously with more precision as the series get longer, more finely-spaced in time, and when replications are available (as in those thousands of basketball games).

P.S. Here’s the R code to make the above simulations and graphs:

set.seed(123)

blank_plot <- function() {
  plot(0,0,xlab="",ylab="",xaxt="n",yaxt="n",bty="n",type="n")
}

winprob <- function(y, t, T, delta, sigma) {
  ifelse(t==T, (sign(y)+1)/2, pnorm((y + delta*(T-t))/(sigma*sqrt(T-t))))
}

N_games <- 10000

sigma <- 2
T <- 48
N_time_points <- T*10
t <- seq(0, T, length=N_time_points)
t_gap <- T / (N_time_points - 1)

y <- array(NA, c(N_time_points, N_games))
p_win <- array(NA, c(N_time_points, N_games))
y_final <- rep(NA, N_games)
p_wrong_halftime <- rep(NA, N_games)
max_p_wrong <- rep(NA, N_games)
for (j in 1:N_games){
  spread <- 0
  delta <- spread/T
  increments <- rnorm(N_time_points - 1, t_gap*delta, sigma*sqrt(t_gap))
  y[,j] <- c(0, cumsum(increments))
  p_win[,j] <- winprob(y[,j], t, T, delta, sigma)
  y_final[j] <- y[N_time_points,j]
  p_wrong_halftime[j] <- ifelse(y_final[j] > 0, 1 - p_win[N_time_points/2,j], p_win[N_time_points/2,j])
  max_p_wrong[j] <- ifelse(y_final[j] > 0, max(1 - p_win[,j]), max(p_win[,j]))
}

N_plots <- 18
par(mfrow=c(N_plots/3,6), mar=c(3,3,1,1), mgp=c(1.5,.5,0), tck=-.01)
t_range <- range(t)
y_range <- max(abs(y[,1:N_plots]))*c(-1,1)
p_range <- c(0,1)
for (j in 1:N_plots){
  par(mar=c(3,4,1,0))
  plot(t_range, y_range, xaxs="i", bty="l", xaxt="n", main="Score over time", xlab="time", ylab="score", type="n")
  axis(1, seq(0,48,12))
  abline(0, 0, col="gray")
  lines(t, y[,j], col="blue")
  
  par(mar=c(3,2.5,1,1.5))
  plot(t_range, p_range, xaxs="i", yaxs="i", xaxt="n", yaxt="n", main="Pr(win) over time", xlab="time", ylab="score", type="n")
  axis(1, seq(0,48,12))
  axis(2, c(0.5, 0, 1))
  abline(0.5, 0, col="gray")
  lines(t, p_win[,j], col="red")
}

print(mean(max_p_wrong > 0.99))

par(mfrow=c(3,3), mar=c(3,3,1,1), mgp=c(1.5,.5,0), tck=-.01)
hist(max_p_wrong, breaks=seq(0,1,0.01))
blank_plot()
blank_plot()

for (i in 1:3) {
  hist(p_win[N_time_points*i/4,], xlab="ESPN's Pr(win)", main=paste("Histogram of win prob from end of quarter", i), cex.main=.9)
}

for (i in 1:3){
  in_quarter <- (t >= T*(i-1)/4) & (t < T*i/4)
  p_win_vector <- as.vector(p_win[in_quarter,])
  y_final_vector <- as.vector(matrix(y_final, nrow=sum(in_quarter), ncol=N_games, byrow=TRUE))
  N_bins <- 20
  boundaries <- seq(0, 1, length=(N_bins+1))
  lo <- boundaries[1:N_bins]
  hi <- boundaries[2:(N_bins+1)]
  p_in_bin <- rep(NA, N_bins)
  mean_in_bin <- rep(NA, N_bins)
  freq_in_bin <- rep(NA, N_bins)
  for (k in 1:N_bins) {
    in_bin <- p_win_vector > lo[k] & p_win_vector < hi[k]
    p_in_bin[k] <- sum(in_bin)/length(in_bin)
    mean_in_bin[k] <- mean(p_win_vector[in_bin])
    freq_in_bin[k] <- mean(y_final_vector[in_bin] > 0)
  }
  plot(mean_in_bin, freq_in_bin, pch=20, xlim=c(0,1), ylim=c(0,1), xaxs="i", yaxs="i",
       xlab="ESPN's Pr(win)", ylab="Empirical Pr(win)",
       main=paste("Calibration from quarter", i), cex.main=.9)
}

I’m writing to ask for your help circulating a PhD opening in my group at Chalmers, the AI and Global Development Lab (www.aidevlab.org). The position is in Earth Observation, Data Science, and AI for poverty estimation, the Data Science and AI division (Department of Computer Science and Engineering). We are looking for candidates with a strong grounding in data science, computer science, deep learning, statistics, or similar— remote sensing experience and causal inference are welcome bonus.

Ad and application portal: https://www.chalmers.se/en/about-chalmers/work-with-us/vacancies/?rmpage=job&rmjob=14818&rmlang=UK
Deadline: 20 June 2026.

Here’s the description of their center:

The AI & Global Development Lab fuses AI with Earth Observation to illuminate the causes and consequences of human development across time and space.

Our interdisciplinary team, comprising data scientists, computer scientists, and social scientists, develops methods to better understand the multi-scale dynamics of pressing global issues, including poverty, conflict, sustainability, and the effectiveness of policy interventions.

By analyzing satellite imagery from 1984 to the present, AI search agent swarms for large-scale knowledge discovery, and other planetary-scale sources, we are reconstructing historical and geographical development trajectories at a level of detail never before possible, working to offer new insights into the changing face of development worldwide.

We also invite you to visit PlanetaryCausalInference.org for more information about the causal arm of our project.

They call it “Planetary causal inference,” which seems to fit the themes of this blog.

Just as the labour movement had never been quite sure whether the capitalist system was on its last legs and needed only a final push to be toppled, or was healthy enough to be milked over and again, so the cultural-intellectual left had never quite decided whether it liked increasing prosperity or not.

I like the above quote, and I would add something analogous for conservatives, that they have never been quite sure whether the capitalist system is an amazing wealth machine with even low-income people being rich on an absolute scale, or whether the system is so fragile that people can barely afford to pay their taxes and that any particular tax or regulation will bankrupt the system. Unfortunately, try as I might, I can’t manage to phrase this as aphoristically as Wheatcroft did.

I suppose that every political movement must balance between triumphalism and alarmism. For another example, environmentalists will announce their progress in protecting the environment and warn of all the horrible things that will happen if more isn’t done. From the other direction, business groups will say that we can’t afford to protect the environment (we want jobs, not owls) but at the same time insist that the environment is better than ever.

The political science research project all this would be to study these ideologies more systematically and see which groups follow different patterns in their statements.

The Anti-Debate is a new format for debate where participants build on each other’s insights, so that greater complexity can emerge.

Despite its name, the Anti-Debate is not anti-debate. It actually starts out like a traditional debate, with opening statements and rebuttals. But then it goes further — guiding participants to explore how they might integrate their perspectives into a bigger picture. Hence our tagline: First Debate, Then Elevate.

Sounds reasonable to me. They refer to the concept of steel-manning, and I’m skeptical of that, but I agree that standard debate formats have problems (just read The Topeka School!) and I’m very open to this sort of alternative.

The organizer, Winter Ku, referred to my posts on “the statistical skepticism about betting markets versus polls (self-reinforcing prices, thin volume), and more recently the integrity and harm concerns in your ‘Uh oh prediction markets’ writing, e.g. manipulation, the absence of insider-trading rules, and the gambling-like risks to vulnerable users,” and it seemed like it would be fun to have a chance to speak on this with several hundred people who might well be inclined to disagree with me. At the very least, I’d get some good questions, lots of pushback, and I’d probably change my mind about a few things.

The anti-debate was to be held at Manifest, an annual festival about prediction markets and forecasting at the same California location that had this blogging workshop a couple months ago. Unfortunately I was only invited to the Manifest thing a couple days ago and I wasn’t able to fly out on such short notice.

I hope the anti-debate goes well without me! Actually, it’ll probably go better without me than with me. I think I’m a careful and interesting writer with lots of good ideas, but I don’t know how well I’d do in a live debate. I imagine I’d get flustered. On the other hand, sharing objections to prediction markets, in front of a crowd coming from a much different perspective than me, but open to listening, could possibly do some good, as well as being a learning experience for me.

So maybe next year! I don’t know if they’ll put the anti-debate up on youtube or whatever; if so, it would be interesting to see the arguments on both sides.

P.S. I came across this entertaining and meandering report from Peter Miller describing this Manifest conference. So now I know what I missed!

I don’t have much to say about the main argument of the article–it’s a topic we’ve gone over all too much in past comment threads–also, as non-user of chatbots, I’m really the worst person to ask for an opinion on the topic. Indeed, the other day I was contacted by a reporter for a story about “vibe analytics” where people use chatbots to write code to perform data analysis. I shared my thoughts for a few minutes but then referred the reporter to Bob and Jessica, as they both have thought a lot more about this than I have. I continue to (a) think that it can make sense to consider chatbots and ping-pong playing robots as having human-level intelligence, and (b) agree with Gary Smith that it remains a big problem when people think chatbots have a level of understanding that they don’t actually have. But, again, my thoughts on this shouldn’t count for much.

But there is one thing in this new article that I did want to comment on. It was just an aside, not the main point by any means, but interesting:

“All intelligence, human or artificial, must extract structure from correlational data.”

Is this true? I don’t know about that, for two reasons. First, I can’t think of many cases where I (that is, my human intelligence) have extracted structure from correlational data. Setting aside my professional life as a statistician and social scientist, when have I done this? I’m not sure. Yes, I’ve estimated parameters from correlational data–for example, if I’m playing sports I make inferences about the abilities of other players based on what they’ve done on the field in the past. But that’s not structure, exactly. There is structure in the world, like the difference between cats and dogs. You can dress a dog up like a cat but it’s still a dog. Essentialism and natural kinds and all that. But that’s not anything I extracted from correlational data: I know it because people told me.

One way that I’ve extracted structure from correlational structure is that as a kid I heard lots of talking and read lots of books and I extracted lots of structure of the language from that. But that’s just one example–an important example, sure, but I don’t know that it’s a characteristic of “all intelligence.”

Another way to look at this is that, as a community, we’ve extracted a lot of structure in the world–it’s called doing science–and some of this is from correlational data (Kepler figuring out planetary orbits, Galton and his table of heights, etc.) but lots of the structure we’ve extracted comes either from logical reasoning (Newtonian mechanics, relativity theory) or from experimentation–they say Galileo did a bit of that.

This doesn’t invalidate the argument made in the linked article–after all, there’s no reason a computer program can’t do pure theory or conduct experiments–; I just thought it was interesting. Speaking in some fundamental sense, it seems to me that experimentation, not just observation, is a crucial part of how we often extract structure. We experiment a lot when speaking. On the other hand, sometimes, as with Kepler or with someone learning a language from reading books, the information is all, or almost, correlational.

It’s an interesting thing to think about. We could throw this at a chatbot and see what it would say–or, more precisely, we could see what it could extract from what humans have said about related topics. But humans have said a lot; it’s a mark of intelligence to be able to read a million books and then extract their key points.

P.S. After reading a bunch of comments, I realize that I kind of missed the point of the passage I was quoting.

My argument above is that intelligence doesn’t learn about structure only by extracting structure from correlational data. Intelligence also learns about structure from logical reasoning and experiment.

But my argument doesn’t refute the quoted line, “All intelligence, human or artificial, must extract structure from correlational data.” That quote doesn’t posit that intelligence only learns from correlations. It just says that learning from correlation is part of the mix, and I agree with that.

So, as long as that passage is interpreted as saying that “extracts structure from correlational data” is necessary for “intelligence,” I’m ok with it. My problem was my interpretation (or misreading) that correlational analysis was sufficient.

Dmitri responded:

When I was 13, there were two things I wanted to understand more than anything else in the world: jazz and quantum mechanics.

Eventually I realized they are kind of similar. In both cases, you start with this fabulously complicated 19th-century language — Lagrangian and Hamiltonian mechanics in the one case, and romantic harmony in the other. Then you “twist” it. In the one case, you turn variables into operators, while in the other you add this scale-based improvisational component. But they are both difficult in kind of the same way because you have to learn this whole other language, and then apply this massive conceptual twist.

But quantum mechanics is genuinely mysterious — there’s some basic stuff we don’t know. General relativity is just straightforward geometry, no mysteries to solve.

All I can say regarding the connection between jazz and quantum mechanics is . . . wow. I wish I could play music, hold music in my mind, and read music. I guess that with a lot of effort I could make some progress in all three of these, but I can’t see myself putting in the time, so I’ll just be wistful about it, and I’ll continue to listen to lots of music and read a lot of music.

Here are some relevant posts (on music, not on quantum mechanics or jazz):

– In music, literature, and technical writing, the relation of large-scale structure to the local action

– Books by Charles Rosen and Jeremy Denk on piano playing and the nature of music

– Playing music, listening to music, background music, talking about music

– How Music Works by David Byrne, and Sweet Anticipation by David Huron

– Why do we prefer familiarity in music and surprise in stories?

– The revelation came while hearing a background music version of Iron Butterfly’s “In A Gadda Da Vida” at a Mr. Steak restaurant in Colorado

– Luc Sante reviews books by Nick Hornby and Geoffrey O’Brien on pop music

– This guy is to music as I am to statistical graphics

– “Song for Aki”: Prof reportedly clears a half million bucks by requiring online students to pay $89.99 each for his self-published course notes

– Why is modern poetry so hard to read? Adam Kirsch offers a clue.

– Causality and Crime: In science as in genre storytelling, the thrill of the unexpected can only come with reference to (and in confounding) some preexisting norm.

And, finally:

– My desert island discs

In psychological test norming, nonrepresentativeness in background variables in the normative sample can lead to bias in the normed score estimates. Because representativeness is difficult to establish in practice, adjustment methods are needed to combat this bias. As a candidate adjustment method, we investigated generalized additive models for location, scale, and shape with multilevel regression and poststratification (GAMLSS + MRP), the combination of MRP and continuous norming with GAMLSS. This adjustment method was then compared to current adjustment methods in continuous norming using weighted regression: GAMLSS + P (with poststratification) and cNORM + R (with raking). The results of our simulation showed that GAMLSS + MRP was generally more efficient than GAMLSS + P and cNORM + R. Furthermore, GAMLSS + MRP was better than the current methods at reducing bias in samples where the nonrepresentativeness was age-dependent. We argue that GAMLSS + MRP is a valid adjustment method in continuous norming and recommend this adjustment method to mitigate bias in nonrepresentative normative samples. To facilitate the use of GAMLSS + MRP in practice, we provide a step-wise approach for the implementation of GAMLSS + MRP. We illustrate this approach by deriving normed scores from the normative data of the third Schlichting language test.

I don’t recall how I came across this paper, and I haven’t actually read it, but I wanted to share it with you, just because it’s cool to see the different ways that multilevel regression and poststratification (MRP) can be used.

Ultimately, MRP is the inevitable consequence of three things:

1. We are interested in generalizing to populations of interest.

2. Available data are typically unrepresentative of the population. This is the case even with simple random sampling–Hello, random variation! Hello, small-area estimation!–and is even more so with selected samples, nonresponse, dropout, etc. In some settings such as medical experimentation there’s not even an attempt to get a representative sample: you’re directly aiming to include in the study the groups of people who might get the greatest benefit from the treatment.

3. When adjusting for differences between sample and population, many variables can be relevant–for example, demographic and geographical variables in a survey of people–and so simple adjustments such as raw poststratification or non-multilevel regression adjustment won’t do the job.

Put this together and you’ll want to do MRP (or, more generally, RPP). It’s not just for survey research. It comes up everywhere in statistics and machine learning, whenever there is a concern with population prediction, or generalization, or transportability, or whatever you want to call it.

It can seem like a hassle that to do this you need to know (or estimate, or postulate) a distribution of predictors in your population, but (a) this is often work that’s well worth the effort, if you really care about the population, (b) dependence of the result on the choice of population is important, and where this dependence is strong you should be aware of it, and (c) if you want to take the easy way out you can always bootstrap to get inference for the hypothetical population of which your data are considered to be a random sample.

The website holds:

– All data and code used in the book
– Free sample chapters
– Bonus material

These aren’t quite the same methods for causal inference that I’m inclined to use (for my own approach, see chapters 18-21 of Regression and Other Stories), but their presentation is clear and has code, and it’s always good to see another perspective.

I’m sure the readers would appreciate your posts. I think they appreciate almost all the posts that aren’t mine. If I could do a customer satisfaction survey, I expect that the average approval of non-me posts is higher than that of my posts. The trouble is that my posts appear 400 times a year so they have no scarcity value.

I hope folks ask about evaluating MRP models. We’ve seen:

• Individual-level Loss(y_i, yhat_i) may not be great for choosing models for MRP. (“individualism doesn’t work”)
• Weighted-to-the-population individual-level loss also isn’t great. (“individualism doesn’t work (even when weighted)”)
• Cross-validation noise can swamp important model differences. (“Individualism and the CV Noise Problem”)
• In probability samples, splitting a cluster between training and test fits models with more information than we should, but not splitting a stratum between training and test fits models with less information than we should. How does this apply to MRP ? (“dCV for MRP ?”)

At Andrew Gelman’s 60-ish Birthday workshop Aki gave a great talk about loo’s 10ish birthday. The loo R package computes approximate leave-one-out (loo) cross-validation. Aki covered a huge range of work across the Bayesian workflow. He said there will soon be a new version of their paper about evaluating MRP models, Kennedy et al. 2024.

Kennedy et al. 2024 pivot from the usual individual-level Loss(y_i, yhat_i) to a population-level Loss(E(Y), E(yhat_i)). We don’t have the true E(Y), so they replace it with a classical poststratification estimate (see the post on poststratification). To avoid overfitting, this classical estimate should be calculated on different data than the MRP model itself.

They use leave-one-cell-out (LOCO) cross-validation, a version of leave-one-group-out (LOGO) that we mentioned in “design-based cross validation (dCV)”. In “dCV for MRP ?” we asked if we should be assessing how well the MRP model predicts new groups (e.g. new cells).

Should MRP workflow include LOCO-CV ?

Mayor Giuliani took the unusual step of naming the Manhattan Detention Complex, the Lower Manhattan central lockup known informally as the Tombs, after a still-living person: Kerik. Giuliani’s police commissioner at the time, Kerik had previously served two years as his correction commissioner, after first getting to know the mayor as his bodyguard and driver and moving up through the ranks under his patronage.

Naming the jail facility after Kerik became somewhat awkward a few years later in 2006, when he was charged with the first of a series of state and federal crimes ranging from receiving undisclosed and improper gifts to lying to White House officials.

Then-mayor Michael Bloomberg recognized the awkward optics, and Kerik’s name came off the building. “After Bernie Kerik pleaded guilty, it was not appropriate to have that facility named after him,” Bloomberg said. “I informed the [Correction] commissioner of my decision and he expeditiously changed the naming of the sign.”

And here’s the funny part:

Nearly 20 years after Kerik’s name was stripped from the Tombs, in July the DOC quietly reinstalled signage designating the building at 125 White Street the “Bernard Kerik Courts.”

“The late Bernard Kerik served as First Deputy Commissioner of the NYC Department of Correction from 1995 until 1997 and served as Commissioner from 1997 to 2000,” a DOC spokesperson told Hell Gate when asked about the new signage. “The Manhattan Detention Center was previously named in his honor and signage was re-installed on the DOC side of the Manhattan Courts upon his passing.” . . .

Kerik’s professional biography is long, fascinating, and so chock-a-block with outrageous and alarming episodes of moral failure that his life takes on a sort of mythic scale, a tall tale of rolling skullduggery.

An extremely incomplete accounting might include: abandoning his daughter and her mother in Korea; commandeering a Battery Park City apartment donated for the use of tired police and rescue workers after 9/11 to conduct one of two simultaneous extramarital affairs; acting as a sex-police enforcer for a Saudi hospital; taking multi-million-dollar payouts from Taser; tasking police under his command to do book research for him and harass Fox News employees suspected of stealing his lover’s jewelry later found in her bag; and acting as the interim interior minister of Iraq, where he took a quarter-million-dollar, no-interest personal loan from an Israeli billionaire with Defense Department contracts. . . .

In 2009, Kerik pleaded guilty to eight federal corruption charges including tax fraud and lying to White House officials about having helped a company suspected of mob connections get a license in exchange for free renovations to his Riverdale home. For those crimes, Kerik did three and a half years in federal prison. . . .

So, yeah, if you’re gonna name something after this guy, it might as well be a jail! “Named in his honor,” indeed.

This is appropriate in the same sense that is was appropriate for them to name an airport near D.C. after someone who overthrew democratic governments in multiple foreign countries.

But BATF doesn’t take the bait

Also, amusingly I found this news article suggesting that the headquarters of the Bureau of Alcohol, Tobacco, and Firearms be named after a politician whose most famous act was to kill someone while under the influence of alcohol. I don’t think they did it, though. According to Wikipedia, they named it after Ariel Rios, an ATF undercover special agent who was killed in action in 1982. The BATF just doesn’t have the sense of irony possessed by the NYC Department of Correction.

I’ve been thinking a lot about scores (gradients of the log density function) and how they can be used for convergence monitoring. We know that the expected value of the score is zero. Stein generalized this with Stein operators. In the monomial case, the Stein operators give you functions in increasing degrees, all of which have zero expectation in the posterior. Here theta is the variable being sampled and S is the score function, so that S(theta) is the gradient of the target log density evaluated at theta.

    Order 0: S(theta)

    Order 1: 1 + theta .* S(theta)

    Order 2: 2 * theta + theta^2 .* S(theta)

This leads to a natural test for convergence of first, second, and third moments. Just compute Monte Carlo estimates of these quantities and see if they’re zero. We’d want to standardize for standard deviation to make the result scale-free like R-hat. To develop some intuitions, in a standard normal distribution p(theta) = normal(theta | 0, I), we have S(theta) = -theta, and thus S(theta) converges to zero at the same rate as our variable theta converges to its true value; the order 1 test is 1 – theta^2, which we know has expectation zero because theta^2 has a ChiSquared(1) distribution with expectation of 1). The order 1 case corresponds to equipartition in physics and the form D + theta’ * S(theta) also naturally has zero expectation as shown in the viral theorem in physics in the 1870s.

Diving into this a bit more led me back to Jackson Gorham and Lester Mackey’s work on Stein’s method. They haven’t been sitting still since introducing the basic idea, which kernelizes the idea above. Mackey et al. have produced an absolutely wonderful summary of this body of work in two forms. The first is a dense, 41-slide deck with all the key definitions and results. I’d suggest at least skimming this first.

Lester Mackey. April 2026. Stein’s Method, Learning, and Inference.. GitHub.

Mackey along with Chris Oates and Qiang Liu, who have also worked heavily in this area, put together a definitive monograph. They’ve presented a great deal of difficult material in a way that I can digest (though it’s going to be rough going if you’re not well versed in sampling and how MCMC is traditionally measured and evaluated).

Qiang Liu, Lester Mackey, Chris Oates. March 2026. Probabilistic Inference and Learning with Stein’s Method. arXiv.

In particular, they go over Stein variational inference, which seems to me like it would be the ideal way to perform quasi Monte Carlo-like inference for statistical models if we could only get a robust version to scale. The idea’s to initialize a bunch of points, then use optimization to minimize a kernelized Stein discrepancy of the empirical distribution of those points to the true distribution.