I’ve been thinking a lot about scores (gradients of the log density function) and how they can be used for convergence monitoring. We know that the expected value of the score is zero. Stein generalized this with Stein operators. In the monomial case, the Stein operators give you functions in increasing degrees, all of which have zero expectation in the posterior. Here theta is the variable being sampled and S is the score function, so that S(theta) is the gradient of the target log density evaluated at theta.

    Order 0: S(theta)

    Order 1: 1 + theta .* S(theta)

    Order 2: 2 * theta + theta^2 .* S(theta)

This leads to a natural test for convergence of first, second, and third moments. Just compute Monte Carlo estimates of these quantities and see if they’re zero. We’d want to standardize for standard deviation to make the result scale-free like R-hat. To develop some intuitions, in a standard normal distribution p(theta) = normal(theta | 0, I), we have S(theta) = -theta, and thus S(theta) converges to zero at the same rate as our variable theta converges to its true value; the order 1 test is 1 – theta^2, which we know has expectation zero because theta^2 has a ChiSquared(1) distribution with expectation of 1). The order 1 case corresponds to equipartition in physics and the form D + theta’ * S(theta) also naturally has zero expectation as shown in the viral theorem in physics in the 1870s.

Diving into this a bit more led me back to Jackson Gorham and Lester Mackey’s work on Stein’s method. They haven’t been sitting still since introducing the basic idea, which kernelizes the idea above. Mackey et al. have produced an absolutely wonderful summary of this body of work in two forms. The first is a dense, 41-slide deck with all the key definitions and results. I’d suggest at least skimming this first.

Lester Mackey. April 2026. Stein’s Method, Learning, and Inference.. GitHub.

Mackey along with Chris Oates and Qiang Liu, who have also worked heavily in this area, put together a definitive monograph. They’ve presented a great deal of difficult material in a way that I can digest (though it’s going to be rough going if you’re not well versed in sampling and how MCMC is traditionally measured and evaluated).

Qiang Liu, Lester Mackey, Chris Oates. March 2026. Probabilistic Inference and Learning with Stein’s Method. arXiv.

In particular, they go over Stein variational inference, which seems to me like it would be the ideal way to perform quasi Monte Carlo-like inference for statistical models if we could only get a robust version to scale. The idea’s to initialize a bunch of points, then use optimization to minimize a kernelized Stein discrepancy of the empirical distribution of those points to the true distribution.

And there’s a contest! Here’s Alexandre:

One listener will get to bring their real-world workflow problem onto the recording for the three of them to work through live!

How to enter:
1. Like, comment or repost this announcement post so other practitioners see it
2. Review the show on Apple Podcasts (https://lnkd.in/gbTagKAS) or Spotify (https://lnkd.in/guYMdMUv), and upload a screenshot here: https://lnkd.in/dPpuSKf5
3. Submit your problem (100-200 words) on the form

Prizes:
1. Grand prize: your problem worked through live by Gelman, Vehtari, and McElreath during the recording + a signed copy of Bayesian Workflow
2. The two runners-up will get signed copies
3. Five LBS Patreon subscribers (random draw) will get ebook copies!

Contest closes June 10!
If you’ve ever wanted to come on the show and ask Andrew, Aki or Richard “okay but what would you actually do here?” — this is your shot!

And here’s the Google form for the contest.

I’ve been on Alexandre’s podcast twice before:

• Modeling the US Presidential Elections (with Merlin Heidemanns), 1 Nov 2020

• Regression and Other Stories (with Jennifer Hill and Aki Vehtari), 30 Jul 2020

It was fun both times. I assume this new one will be too.

For a writer to be a public intellectual was a choice, and a choice that not every writer made. Robert Penn Warren and John Updike wrote some public-intellectual sorts of things that were trifles compared to their fiction; T. S. Eliot wrote some essays but they weren’t so readable, perhaps no surprise given that his poetry is pretty tangled too! Gore Vidal and Norman Mailer became more celebrated for their nonfiction than for their fiction, which may well have annoyed them.

But who are the public intellectuals of today? Not the authors. Lorrie Moore, Jonathan Franzen, etc., write the occasional essay but they don’t have the moral or intellectual authority of Wilson or Orwell or even a lesser light such as Clive James. Paul Krugman and, formerly, Steven Levitt, have had some influence but they are coming from academia; their authority doesn’t come from their ability to put words together, in the manner of a Mencken, Didion, etc. The literary critic Jackson Lears writes about politics for the London Review of Books, but he seems to me more of a well-connected bullshitter than anything else, and not readable enough to be worth reading for style alone, as was sometimes the case with Wolfe.

OK, so one thing is that literary fiction is much less important in our culture as an arts and entertainment medium. But it’s not like filmmakers or the equivalent have picked up the slack. Tarentino or Scorsese might make the occasional pronouncement, but they’re not all-purpose expert. David Mamet might like to be an all-purpose expert, but he’s just a playwright and screenwriter with a side gig as a loud opinionator: you see lots of people like that on the internet and, sure, Mamet and Joyce Carol Oates and J. K. Rowling and all the rest have every right to try to transfer their literary fame into political influence, but really they’re acting like celebrities here, not thinkers of the order of Mary McCarthy or, yes, Gore Vidal.

Another direction to look among modern writers would be political pundits like Josh Marshall or David Brooks, commentators who have reached wide influence through their ability to say interesting or provocative things (even if, in the case of Brooks, they sometimes get their facts wrong and don’t seem to care) in a direct and clear style. But I’d say that Marshall and Brooks correspond not to Edmund Wilson but rather to other news pundits of the past, like Walter Lippman or Joseph Alsop: they weren’t promulgating theories in the manner of Wilson, Orwell, Sontag, Wolfe, etc., but rather reporting on ideas that are out there. Which is valuable too, just not quite the same thing.

OK, my answer to the “Who’s the modern-day Edmund Wilson?” question is no secret–it’s in the title of this post. Scott Alexander is, like Wilson and those others, a prolific writer with strong political opinions, who feels comfortable opining about–and gets respect for his opinions on–all sorts of things. Like Wilson, he’s an autodidact and a bit of an outsider. Alexander’s topics are different from Wilson’s–more on science, less on literature–but he has the same feeling of well-earned authority. And a fun writing style. Bloggy rather than magaziney–times have changed–but a distinctive and readable style in any case.

Alexander and Wilson also share an interesting balance between political extremism and curmudgeonly centrism. Wilson was heavily influenced by communism–no surprise for a New York writer in those days–and you see in a lot of his writing a pushing back and forth among different ideologies. He was cranky but no ideologue. Alexander has been heavily influenced by far-right internet writings–no surprise for a tech-adjacent blogger in the past decade or so–but doesn’t strongly identify with the far right (indeed, he unequivocally recommended voting against Donald Trump); you can see the influence still there even as it battles with others. Just as Wilson was always writing under the auspices of communism, as it were, even–indeed, especially when–he was arguing against it, similarly, Alexander is always writing under the auspices of the far right, operating on that playing field, as it were, even when he has his differences.

Wilson and Alexander are people with eccentric personas (I don’t mean “eccentric” in a bad way here), and I think this is to the benefit of their writing, even if sometimes it leads them into places they might not have intended to go.

I don’t want to make too much of this. Neither Wilson nor Alexander are unique literary figures; as noted above, Wilson was part of a long tradition of writers who were public intellectuals, and Alexander’s not the only blogger out there. Other influential bloggers include Josh Marshall (mentioned above), Tyler Cowen and Alex Tabarrok, and others. Actually not so many others anymore. Now we’re seeing another kind of thought leader, which is people who are loud on twitter, and I guess they’re important too, but I think of their role more as spreading ideas, not in the mode of writers like Orwell or Didion who express new thoughts and for whom the way that the thought is expressed is as important as its content.

P.S. Another way that Scott Alexander is a public intellectual is that people care about his positions; he has some intellectual and moral authority. You can see this in the comments: some people like Alexander, some people don’t, but there’s a sense that he’s an important figure and that his opinions matter.

We’re having to reconsider what authorship means. Can a person create literature or express their position on a subject without writing a single sentence themselves? When do we really care who strung the words together?   

Some people think detection is a waste of our collective time because we will never reach an equilibrium. AI-generated text will keep shifting toward what passes the detector. Human writers will continually update their beliefs about what features are indicative of AI-writing, but will also be influenced to write more like AI by reading so much AI text. There’s no stable target, just an endless cat and mouse game that incentivizes being savvy enough at any given time to avoid getting flagged. Meanwhile people are being morally scorned and suffering reputational damage for being caught on the wrong side of things. This may disproportionately affect some writers (like non-native english speakers) who are finally seeing the playing field leveled a bit. 

On the other hand, there are situations where it really is important to know who strung the words together. Education is the most obvious one. It’s just very hard to teach someone to think if they’re not writing down their ideas themselves. 

The problem is that outside of select scenarios like teaching, what we really tend to care about is who controlled the ideas, and this is not equivalent to who strung the words together. Some would argue that the latter is becoming increasingly irrelevant given that AI can write more fluently than many people and many people prefer AI-generated text. 

Of course the reason we’re seeing detection used to filter paper submissions is because the ideal process–where the content of each paper is carefully considered on its own merits–is increasingly untenable given the huge surge in submissions in some fields. It’s easy to pump out credible-seeming papers with minimal human oversight using AI, and enough people are doing this to create serious problems. 

Mostly my response is that if we are going to debate the value of detection we should be willing to make our assumptions explicit. So let’s walk through a toy model to think about what we’re really conjecturing about.

One way to think of the latent state that we actually care about in paper review is the author type. Let’s say type A authors come up with their ideas and do a lot of the writing themselves. Type B authors rely on AI to do much of the thinking for them, and also use AI to do much of the writing. Type C authors come up with their own ideas, but engage in extensive prompting to get AI to write everything they want to say for them.*

For each paper, we choose to either pass or reject, conditional on the output of a Pangram check. Let’s say we only care about whether it flags 100% AI generated or not, so the signal s is binary, where s=1 means AI detected.

Based on available Pangram audits, if a text is actually written heavily by AI there is a very high chance it flags as AI-generated: beta=P(s=1|AI written) with beta very close to 1. If a text is not written by AI, there is a very small chance it flags as AI-generated: alpha=P(s=1|human written). Pangram’s internal audits put alpha around 10^−4 but other audits find essentially zero false positives for medium-to-long passages. 

So P(s=1| A)=alpha, and if we assume Types B and C use AI to a similar extent for the writing, then \beta=P(s=1|B) = P(s=1|C). The posterior probability that a flagged paper is from a Type B author is then:

P(B|s=1) = (beta × p_B)/(alpha × p_A + beta × p_B + beta × p_C), and since alpha is tiny and beta is close to 1, P(B|s=1) ≈ p_B/(p_B + p_C)

The relevant considerations become what we think the author population looks like, and how costly we think a false positive versus a false negative are. 

As a starting point, let’s say that for our conference submissions this year, Type C is the rarest, at 20%, and Type A and Type B equally split the remaining mass at 40% each. Let’s also say that we consider rejecting an acceptable paper, c_FP, to be twice as bad as passing an unacceptable one c_FN. 

The optimal decision rule is to reject if c_FN​ * P(B|s=1)>c_FP * ​P(A or C|s=1), or equivalently P(B|s=1)>c_FP/(c_FN+​c_FP​​)

With c_FP=2 and c_FN=1, this means we reject if P(B|s=1) > 2/3.

Under the prevalence assumptions above, P(B|s=1) is approximately 2/3, so we are right on the boundary. From the standpoint of making the right decisions for this particular conference cycle, it’s not obviously bad. But if Type C is a little more common, e.g., we shift a little mass from p_A to p_C to make p_C 0.25, then P(B|s=1) is 0.62, then we shouldn’t desk reject only based on the flag. Similarly if we were to decide that falsely rejecting an acceptable paper is three times as bad as passing an unacceptable one, we shouldn’t rely on it alone. 

This model is obviously very simple. But it shows us what kinds of things we have to make assumptions about in the most basic case. Obviously I don’t really know how many people are using AI blindly to write papers, nor how many people are relying heavily on AI to write up their own ideas. You should take my numbers with a grain of salt. Personally I can’t imagine how relying on AI to do all the writing when I came up with the ideas would ever feel efficient, because I tend to have strong opinions on how things are said. But I can accept I am probably more of a control freak than many others. And AI overreliance is easy to slip into. Maybe papers chairs from recent ML conferences (or arXiv moderators) have estimates on bad-actor rates based on what they are seeing. 

What this exercise can’t tell us is how scientific progress is impacted by the warping of incentives that can happen when we use AI-detection as a filter. Classic principal-agent problems suggest that when we care about something hard to observe—like scientific quality or long-term epistemic value—but must rely on observable proxy signals to judge authors’ outputs, we should expect authors to shift more effort toward improving exclusively on those proxies. Avoiding m-dashes and ‘not this, but this’ constructions and whatever else currently ups the posterior probability of AI-generation is orthogonal to the actual thinking that research requires. What if relying more heavily on AI to write up our ideas is a good idea for science in the long run, in terms of more clearly communicating the ideas or saving a lot of time, so that we can get more good ideas out in the same amount of time? Then too much emphasis on detection might slow us down. However, I’m doubtful we are currently anywhere near a state of the world where discouraging writing with AI is as costly for scientific progress as spending time reviewing and reading many more questionable AI-generated papers is. The bigger threat at the moment is the slop overwhelming our ability to find the good stuff.

*We could also posit Type D authors that get AI to generate the ideas, but then write the papers themselves to evade detection, or are extremely good at getting AI-written text to evade detection. But this seems much less likely so I’m ignoring it.

The interaction depends on the model of y given x1 and x2, while the correlation depends on the model for x1 and x2. These are two completely different parts of the model. And yet, they often seem connected.

I have the general impression that I’d be more likely to expect a positive interaction of x1 and x2 when predicting y, if x1 and x2 are positively correlated in the population.

For example, when predicting income from height and sex, being taller and being male both predict higher income, also they interact–the coefficient for height is higher for men than for women–and of course the two predictors, height and male, are positively correlated in the population.

I’m not sure how to think about this connection or even whether it’s a real pattern! But there might be something there so I wanted to share it with you.

The issue of interactions comes up in the context of the concept of intersectionality, which is a form of interaction that comes up in sociology. It started for me with this email from Elin Waring:

I’ve been working on data on intersectionality and retention of students in STEM majors. My little group is specifically looking at data from Lehman College and trying to model graduation with a STEM degree. There are a lot of details, but basically we have come to the conclusion that the right way to describe this is with a discrete time competing risk model (the competing risks being graduation with a STEM degree and graduation with a non-STEM degree). I won’t go into all the details. We have data for between 1 and 20 semesters enrolled for students starting as freshman. For us, intersectional identity is defined by 5 variables that yield 32 distinct combinations or strata as used in the next articles.

In trying to think about how to account for intersectional identities we came across the “MAIHDA Method.” I was wondering if you had seen this discussion before or have any thoughts about it.

Evans, Clare R., George Leckie, and Juan Merlo. 2020. “Multilevel versus Single-Level Regression for the Analysis of Multilevel Information: The Case of Quantitative Intersectional Analysis.” Social Science & Medicine (1982) 245:112499. doi:10.1016/j.socscimed.2019.112499.

They essentially argue for treating the strata as random effects in a multilevel model where with the individual components of the combinations introduced as fixed effects describing the combinations.

The next article criticizes that approach and argues for fixed effects all around.

Wilkes, Rima, and Aryan Karimi. 2024. “What Does the MAIHDA Method Explain?” Social Science & Medicine 345:116495. doi:10.1016/j.socscimed.2023.116495.

Responded to here:

Evans, Clare R., Luisa N. Borrell, Andrew Bell, Daniel Holman, S. V. Subramanian, and George Leckie. 2024. “Clarifications on the Intersectional MAIHDA Approach: A Conceptual Guide and Response to Wilkes and Karimi (2024).” Social Science & Medicine 350:116898. doi:10.1016/j.socscimed.2024.116898.

I was wondering if you have any thoughts about this? For me, intersectionality as a theoretical approach does mean that it makes sense to look at the strata rather than thinking of the strata as just the most complex level of creating statistical models of the intersection of the variables. But then it seems as though treating this a random effect more or less undermines its centrality to the theory. And is treating both the strata and the individual characteristics as variables at the same level basically a way to decompose?

In the end, I feel like the pro-MAIHDA people retreat to “we are just descriptive” in a way that isn’t very helpful. That said, they are right that this seems to have some traction in the world of health disparity research.

I replied that I’d never heard of any of this method before. I couldn’t actually muster the energy to read the above articles, as all this debate seems to be missing the key issues. I don’t really care if something is called a fixed effect or a random effect (see here); my current preferred way of thinking of these problems is by framing as a generative model.

Regarding intersectionality, the natural way I would see it is that this would show up as an interaction term, the idea that the interaction is more than the sum of its parts? For a simple example, if there are 5 binary variables and each has the same effect on its own (which they wouldn’t, this is just a simple hypothetical example), then you could create a variable which is the total number of identities, thus a number from 0 to 5, and “intersectionality” would show up as a super-linear or convex relation between the outcome and this total predictor?

Waring responded:

Sure, but the idea you suggested about intersectionality itself isn’t right. You can’t just sum the number of identities, everyone has identities and the idea is that it is not just about concentrated disadvantage of having all or some specific identities. If we have 5 dichtomous identity/group variables everyone has 5 dimensions of identity. Intersectionality is about the idea that something like “white, native born. woman, high income” shapes what happens because of how those come together to shape (in the case of my analysis) whether, as an undergraduate, you persist in STEM fields.

I replied as follows:

Yes, I was actually thinking this when I wrote that! I was imagining that each of the 5 factors has an “off” and “on” setting, and intersectionality kicks in when there are multiple “on” settings, where “on” represents the group that faces more difficulty (nonwhite, non-native born, female, low income, gender nonconformist, etc.). Once you allow arbitrary possibilities for intersectionality, then my simple superadditive model wouldn’t fit. On the other hand, if you were to allow all 32 possibilities to take on any value, then realistically you would not be able to estimate anything much at all: this is the usual problem in sociology of approximating a complex social structure by a simple model that explains most of the variance. For predicting persistence in STEM (or any academic field), one possible factor that could enter in a complicated way is conservative political ideology, in that for many attitudes and behavior its predictive effect goes in the opposite of the “on” categories listed above, but grad students, in STEM and other fields are predominantly politically on the left. I could well imagine that conservative political ideology, like the other “on” categories, is predictive of not persisting in STEM but that this could interact in unexpected ways with those other categories.

From a statistical perspective, my main message is to choose such a model based on its explanatory power and recognizing that it’s an approximation, rather than using methods such as statistical significance or Bayes factors which in different ways are driven by sample size, as we discussed in this 1995 paper.

Another interesting statistical feature of this and similar discussions is that it’s natural for the discussion to go back and forth between the correlation between two predictors in the data (or the population) and the interaction between their predictive effects, as discussed at the top of this post.

I’m not sure if this interaction thing is a general pattern that has some statistical explanation, or just a faulty intuition of mine based on just a couple of special cases. But I have noticed a general confusion that when people talk about interactions, often they seem to be talking about correlation between the predictors.

I haven’t thought about the Monkey Cage for awhile. At some point it became attached to the Washington Post, and they kept telling me that my posts were too bloggy and not journalistic, so I ended up contributing much less often. Then it left the Post and got renamed as Good Authority. My last post was in 2020.

Sometimes, though, I come across old posts on this blog that link back to my Monkey Cage articles. Often the links don’t work because the site kept changing ownership. So then I have to do some googling to find my original post. Annoying! But I guess my bad for not saving everything I write locally.

Anyway, I recently found this archive of everything I wrote for the Monkey Cage. It looks like there are over 1000 posts! It’s hard for me to believe I wrote so much for that site. I’d forgotten almost all of this. To be fair, some of these posts are pretty short.

P.S. The Good Authority people need to fix their webpage. The top of it looks like this:

That’s all fine except that I clicked on the tab for “Israel-Hamas” and got the following articles:

Only one of these is about Israel or Hamas! Or maybe they’re trying to draw a not-so-subtle analogy by including a post on white South Africans in the “Israel” category.

I get so tired of people dumping on decision theory because real world decisions are complex. If decision theory is so deeply flawed, I’d love to know what alternative methods the critics advise for trying to evaluate and improve decision making in some real world setting. Should we give up on modeling completely because some cause problems for our assumptions? What happened to the epistemic value of attempting to formalize goals so as to better understand what components we think are at play? Do we really want to go back to talking about man as a creature of instinct and habit and leave it at that?

I agree, and this reminds me of a discussion from twenty years ago (!) about the transition from viewing people as “rational animals” to viewing people as “irrational computers.”

Here’s Thomas Jefferson from 1823:

We believed . . . that man was a rational animal, endowed by nature with rights, and with an innate sense of justice; and that he could be restrained from wrong and protected in right, by moderate powers, confided to persons of his own choice, and held to their duties by dependence on his own will.

He’s coming from a liberal (in the U.S. politics) perspective, with the idea being that rationality is a way to move forward from outmoded feudal arrangements. Not that this was so easy–Jefferson owned slaves!–, but nobody said that rationality was easy, just that it’s a way forward.

This association, in which the left was associated with utopian rationality and the right was associated with sensible acceptance of irrationality, continued for another century. Consider, for example, the contrast between the rationalist and socialist George Barnard Shaw and the Catholic conservative G. K. Chesterton. This association of rationality with the left continued through the New Deal period in the U.S. and the idea of the Soviet Union as being scientifically socialist. The second world war pitted Soviet central planning and “Fordist” American organization against the blood-and-soil Axis powers.

Sometime during the mid-cold-war period there was a shift, at least in the U.S. and its allies, where science and technology was associated with the military-industrial complex and gained a conservative tinge, while the left embraced an anti-technology, back-to-the-land vision. “Humanism” moved from a conservative, roll-back-the-tide, Chestertonian position to a liberal, fight-the-Man position.

Nowadays things are a mess: conservatives support military and police hardware, coal, nuclear power, bitcoin, data centers, and gas guzzlers more generally, but conservatives also oppose vaccines and scientific more research more generally, and Biblical creationism hasn’t gone away either. And, with conservatives in charge of the country and much of public discourse, liberals are often defining themselves based on what they oppose.

I’m with Jessica in that I see no conflict between humanism and rationality. Rationality is an ideal or a way of being, not an algorithm. Yes, we’re animals, and rationality is one of our very useful tricks. I wouldn’t want to abandon rationality or define ourselves against it, any more than I’d want to abandon running or singing or any of the other things that we can do so well, when we do them well.

Donna Spiegelman shares this presentation she gave at the recent American Causal Inference Conference. I like what she has to say.

Here are the two parts of the stable treatment value assumption:

1. No interference between units. As Spiegelman says, nowadays it’s not hard to model spillovers. As I say, untangling spillovers is an ill-posed inverse problem that can be solved using Bayesian inference with reasonable priors. Serious practical work has moved past the demonstrate-that-spillover-doesn’t-matter stage to the just-model-the-spillover-directly stage.

2. Deterministic potential outcomes. As Spiegelman says, in the real world, outcomes are stochastic. Jonas and I talk about this in our Russian roulette paper.

The part that I’m less sure about is Spiegelman’s claim that adjustments for pre-treatment variables usually don’t matter. I’m persuaded that they usually don’t matter in the epidemiology and biostatistics applications she’s worked on, but I think that in social science, such adjustments can be important. Especially if there are big treatment interactions and your population is a lot different from your sample.

In any case, I recommend you look through Spiegelman’s slides, as she offers a refreshing perspective compared to our usual obsessive focus on the details of causal identification:

Artist Sophie Gelman made the below:

Yair Ghitza gave a talk about Andrew’s influence on polling. Yair is Chief Scientist at Catalist and coauthor of excellent papers about MRP we’ve cited in this blog series: Ghitza and Gelman 2013 and Ghitza and Gelman 2020. The discussion after his talk included mention of Nate Cohn’s May 18, 2026 NYT article about weighting with “synthetic past vote”.

Let’s use our notation from “is a mismeasured X better than none at all ?” and “more adventures in mismeasured X” (see also “more on recalled vote”):

• Y = current support
• X = true 2024 vote, unknown
• X* = recalled 2024 vote

And notation from “weights and MRP for voters”:

• V = current registered voter
• V2024 = record of voting in 2024 (a coarsened version of X that only tells of whether someone voted)

Suppose we want E(Y | V=1), current support among current registered voters. Using MRP, we might want to estimate this via E(E(Y | X, sample, V = 1) | V = 1). But we’ve got at least 2 challenges:

1. We can’t directly estimate E(Y | X, sample, V = 1) because we only have recalled vote X*.
2. We need p(X | V=1) for the outer expectation. But past election results give p(X).

Nate’s article proposes:

1. Create synthetic past vote, X**, which aims to improve recalled vote X*:
   1. Impute X** if X* is missing and there is a record they voted (i.e. V2024 = 1).
   2. Validate: set X** to “nonvoter” if there is no record they voted (i.e. V2024 = 0).
2. I interpret this to mean they estimate p(X** | V = 1) ? See “weights and MRP for voters” for ideas.
   

   “Synthetic past vote is weighted to match our estimate for how today’s registered voters …voted in the last election.”

    

What do you think ?

But, yeah, with respected news sources repeatedly falling for ridiculous scams like the hyperloop or the idea that UFOs are space aliens or mind-body healing etc etc etc., I guess we shouldn’t be shocked that they will uncritically report on a junk survey. It’s that insatiable need for fresh content.

One of my general problems with the concept of “unintended consequences” is that it so often seems to be used either as an argument against a proposed reform (recommending to not do this seemingly good thing because of its unintended consequences; what Albert Hirschman called the “perversity thesis” in his classic book, The Rhetoric of Reaction) or as a way to get evildoers off the hook by arguing that their bad actions were actually the unintended consequences of somebody’s good intentions.

I have a similar problem with Hanlon’s Razor (“Never attribute to malice that which is adequately explained by stupidity”). Often Hanlon’s Razor applies, that’s for sure, but I also think it can be a way to let people off the hook.

Also, often the simpler explanation is the right one. In the motivating example for Noem’s Razor, someone attributed the lethal behavior of the immigration police in Minneapolis as a “Sad case of poor incentive design (ICErs create expensive externalities bc of legal, reputation. etc costs of processing bad detentions and arrests. Textbook amateur mistake.”–but it seemed to me more likely that those police were doing what the government wanted. So the incentives (by which the agents can break the law without fear of consequences) worked directly. There’s no evidence that the consequences were unintended. The political consequences may well have been not as desired, but I see that as more of a political miscalculation than anything else.

As the economists say, when there’s a policy that seems like it doesn’t make sense, think more carefully about the incentives. And of course this policy applies much more generally, as in the literature on regulatory capture.

I don’t buy the argument that the nice guys are the real assholes. I think the assholes are usually the real assholes.

That doesn’t mean I think that all purported do-gooders are actually doing good–see here, for example. People need to be evaluated based on what they do, not what they say.

About 15 years ago you kindly linked an article I wrote on “rational turbulence.” I’d like to let you know that I have recently summarized much more research along these lines in a short book Rationally Turbulent Expectations.

I have published it as cheaply as color printing allows and also posted all chapters for free on ssrn, starting with this overview.

The main finding, summarized in the first few pages of Chapter 4, is that–once we allow for even tiny doubts about the the stability of an iid process–Bayesian learning has calm and turbulent phases, with fast learning more turbulent. It explains why differences in opinion between two reasonable people often widen before they narrow. I think this deserves broader attention in that people can learn to disagree more respectfully.

Hardly anyone will listen to me but many listen to you, so I am hopeful you will persuade yourself of this quickly and help persuade others.

I clicked through and took a look. Lots of things there resonate with various ideas I’ve discussed various times without ever fully thinking them through. For example:

“Words like ‘ahead’ and ‘forward’ that point to space in front of us also point to future time. However, that isn’t the only way to align directions”: This reminds me of the idea discussed here that there’s a logic of causality going forward in time and a logic of inference going backward in time. Generative models in statistics are a way of going from one of these to the other.

“Most surprises are outliers from a still intact trend. We tweak the next round of forecasts and move on. Occasionally the surprises get under our skin. They shock us less by their size than their persistence. They make us suspect that what we thought of as a rare outlier is now the new norm”: This reminds me of the ideas of Shewhart, Deming, etc., on quality control, an approach to statistics which I think is important and underrated.

“Contemporary finance theory rests on an unstable truce between two opposing schools. . . . The Rational Expectations school treats the market as a knowledge machine that assesses risks correctly and prices them appropriately. The Behavioral Finance school treats the market as a ship of fools prone to long stretches of complacency and short bouts of panic”: This reminds me of what I’ve called the two modes of reasoning in microeconomics: people are sometimes considered to be rational, so that the role of economists is to observe and analyze behavior and, from that, deduce values and motivations; and sometimes people are considered to be irrational, and the role of economists is to set them straight. Either way, the economist (or “freakonomist”) is portrayed as a culture hero, either in protecting us from pinheaded academics who don’t trust the ordinary Joe to make his own damn decisions, or in helping people avoid deadweight losses all around them.

Osband frames the economy in terms of turbulence, which fits in well with the idea that economics occurs on the phase transition of equilibrium. “Turbulence” seems like an appropriate term.

I don’t have the energy to read the whole book–I guess there’s an economic message there too!–so I can’t say more, but I’m happy to spread the word, and if you’re interested you can take a look into it yourself.

These arguments typically follow the patterns of anti-reform arguments noted by Albert Hirschman: perversity, futility and jeopardy. From wikipedia:

– According to the Perversity Thesis, any purposive action to improve some feature of the political, social, or economic status quo only serves, perversely, to exacerbate the very condition one wishes to remedy (compare: Unintended consequences).

– The Futility Thesis holds that attempts at social transformation will be unavailing, that they will fail to “make a dent” in the problem, and the motives of those who keep attempting futile reforms are suspect.

– The Jeopardy Thesis states that the risk of the proposed change is too great as it imperils some previous, precious accomplishment.

Just because these are standard forms of argument, it doesn’t mean they’re wrong. In any given case–indeed, in many actual cases–one or more of these points can be correct. Just cos a proposed reform sounds kinda reasonable, it doesn’t mean that it would be a good idea if implemented.

But I see a broader themes in political science anti-reformism, a cynical attitude that we’ve described in the past as, A quick rule of thumb is that when someone seems to be acting like a jerk, an economist will defend the behavior as being the essence of morality, but when someone seems to be doing something nice, an economist will raise the bar and argue that he’s not being nice at all. In political science this comes out in the form of, Ha ha those silly reformers don’t understand the real world. Among economics writers you sometimes see the argument that insider trading laws shouldn’t be enforced, either because insider trading is going to happen anyway (futility, in Hirschman’s terminology) or that insider trading is good in itself because it makes the market more efficient (jeopardy). There’s also the argument that environmental regulation is bad because it can be evaded (futility), that it leads to the dreaded “regulatory capture” (perversity) or that, being a government-enforced law, will interfere with a preferable market solution (jeopardy). Political science anti-reform arguments are a bit different from economics anti-reform arguments in that the economists often seem to want to remove politics and government as much as possible, whereas political scientists will argue that all problems should be settled at the ballot box.

I’m not saying that the political scientists or the economists making anti-reform arguments are themselves cynical. There’s no reason you can’t sincerely believe that campaign finance laws will be futile or that insider trading laws can’t work. But I do think they can have knee-jerk reactions against anything that is promoted as a good-government reform. Skepticism is fine, but just cos something is promoted as a reform, that doesn’t automatically make it bad either.

I’m not sympathetic to the general argument that he bad guys are gonna win anyway, so let’s not even try to stop them.

All this has become clearer to me in the current political climate in which corruption has pretty much become legalized (see here for one of many high-profile recent examples). Partisn redistricting is out of control, all sorts of lies are heavily promoted by public officials and on social media, nonpartisan news sources are being attacked by the government, campaign finance laws are routinely violated with no consequence, but meanwhile ordinary people are at risk all over the country for getting fired because of saying something insufficiently worshipful about Charlie Kirk or saying something that some campus watchdog doesn’t like. This is political polarization–anything goes because the view is that the other side is worse–and I’m not saying that any particular reforms, whether they be campaign finance restrictions, anti-gerrymandering laws, or an end to court rulings that effectively legalize bribery, will stop this trend–but I do think that the ending of political restraint has been facilitated by the efforts made in recent decades to take away the rules.

In short: Yes, political reforms don’t always deliver what they promise. But a continuing flow of reforms may be necessary to counter trends in lawlessness.

To put it another way, the fact that bad actors can work their way around reforms is often taken as an argument against reform: if a reform is gonna be evaded, why do it? But I’d argue it the other way: if bad actors are going to try to get around the rules, we should make it effortful for them to do so, and risky too. When corruption is effectively legalized, as it is now, the barriers go down and more people will do corrupt things, indeed from an economic and political standpoint they’re pushed in that direction. When corruption is forbidden, yes, it will still happen, but you’d expect to see less of it. And there’s a big difference between “some politicians are corrupt” and “the government is up for sale.”

Aki, Richard, Lizzie, and I put together a special issue on Statistical Workflow for the Philosophical Transactions of the Royal Society. I guess “royal” isn’t as impressive as it used to be, but still.

Statistics and data analytics play an increasingly important role in and across science and policy. But much of what is done by the best practitioners–their “workflow”–is tacit knowledge only glanced over in textbooks and research articles. In this new collection covering a wide range of disciplines, leading statisticians and researchers discuss the motivations and details for their workflows.

The four of us did this project because we were all interested in Bayesian workflow, and we wanted to learn more about statistical workflow in general, not just the Bayesian part.

Here’s what’s in the issue:

• Statistical workflow, by Andrew Gelman, Aki Vehtari & Richard McElreath
• Unsupervised machine learning for scientific discovery: workflow
  and best practices, by Andersen Chang, Tiffany M Tang, Tarek M Zikry & Geneva I Allen
• PCS workflow for veridical data science in the age of AI, by Zachary T Rewolinski & Bin Yu
• Simulations in statistical workflows, by Paul-Christian Bürkner, Marvin Schmitt & Stefan T Radev
• An automatic finite-sample robustness metric: when can dropping a little data change conclusions? Part I: definitions and experiments, by Ryan Giordano, Rachael Meager & Tamara Broderick
• An automatic finite-sample robustness metric: when can dropping a little data change conclusions? Part II: theory and intuition, by Ryan Giordano, Rachael Meager & Tamara Broderick
• Building a Backdrop of Meaning in Magnitude (BoMM) as part of research workflow, by Megan Dailey Higgs
• A preliminary data analysis workflow for meta-analysis of dependent effect sizes, by Elizabeth Tipton, James Pustejovsky & Jingru Zhang
• A four-step simulation-based workflow for ecological analysis and science, by EM Wolkovich, T Jonathan Davies, William D Pearse & Michael Betancourt
• Scientific workflow in experimental economics, by Anna Dreber & Séverine Toussaert
• Hidden processes of workflow in cognitive developmental psychology, by Lauren N. Girouard & Susan A. Gelman
• Reproducible workflow for online AI in digital health, by Susobhan Ghosh et al.
• Model checks for Bayesian estimation and forecasting of health coverage indicators in low- and middle-income countries, by Leontine Alkema et al.
• Closing the gap between statistical and scientific workflows for improved forecasts in ecology, by Victor Van der Meersch, James Regetz, T Jonathan Davies & EM Wolkovich
• Machine learning workflows in climate modeling: design patterns and insights from case studies, by Tian Zheng et al.

Lots of good stuff here, and lots of different perspectives. Thanks to all the authors. The issue is here, and all the papers should be freely available.

If you have any thoughts on the articles in the volume, or on any other statistical workflow topics, just let us know right here in the comments box.

My site is kappazoo.com, and it’s still a work in progress. I would rather have emailed after I had the new goodness-of-fit code done, but I saw that you’re doing a summer workshop (on Andrew’s blog) [editor: Modern Modeling Methods (M3)], so thought I’d mention it now.

It may be billed as a work in progress, but it’s a complete draft with no missing sections that provides a really nice overview of rating/crowdsourcing models. These are the models that dragged me into statistics, namely Bayesian rating models formulated as noisy measurement models. The first model of this kind that I or Eubanks could find was Phil Dawid and Allan Skene’s (1979) paper on rating.

Eubanks works through a great deal of workflow without calling it that. There are multiple model evaluation and comparison measures used and explained with connections to information-theoretic notions like entropy.

There’s a long discussion of Cohen’s kappa statistic, which is a commonly reported statistic measuring inter-rater agreement. As Eubanks notes, it doesn’t deliver on its promise of adequately measuring inter-rater agreement. The discussion is quite good here and complementary to the discussion from me and Becky Passonneau in our paper on rating in NLP, though our conclusions are the same.

I was surprised to see that Eubanks has a section comparing item-response theory (IRT) models with difficulty. I’ve been trying to convince people this is important for years. It took me around ten years to figure out how to move from Dawid and Skene’s IRT-0-like model to an IRT-1-like model, which we report in in our latest paper on crowdsourcing with difficulty parameters (which also works through a lot of Bayesian workflow in considering different models). I can’t identify what took so long—it seems so obvious to me now.

The seemingly irrational attachment to the ideas or findings can seem easy to dismiss as a bad thing. Yet there are also times when having some level of personal commitment makes one more effective at certain roles that scientists must play. For example, being too transparent about our own uncertainty is not always effective when presenting research to others, because the audience can become distracted and stop listening entirely, even if you have some useful insight to convey. My question is, What does our ability to now use AI to generate implementations, presentations, and even the ideas we work on themselves add to the mix?

I got a glimpse of this recently. May ended up being workshop month for me, with at least one each week. I saw a lot of presentations. A couple of these showed me something I hadn’t yet seen, at least outside of student presentations: talks comprised of obviously AI-generated slides. If you’ve tried to use the non-design optimized versions of models like GPT or Claude to create slides yourself, you will know what I mean. Almost every slide has content organized in a grid. There’s too much text—full sentences or nearly so in multiple places, headers and footers, and stylized phrasing everywhere, like “principal design levers” and “load-bearing assumptions” and “actionable pathways”. 

These were not presentations by overwhelmed junior faculty or researchers I’d never heard of. They were by prominent researchers who are respected in their fields. 

Needless to say, they were not very effective talks. The slides tended to have too much going on to parse in time, with way too much text. The vague phrasing was distracting, making me wonder what exactly the presenter meant by terms like “governing frictions” or “strategic bottlenecks” and whether they write like that in their papers too. Part of the problem is that the presenter tends to use their own language as they present, rather than reinforcing what’s on the slide, so you have two competing streams of information that feel like they’re from two distinct viewpoints, one which is quite confident and willing to summarize and even exaggerate, the other more reserved. 

It makes sense that you’re more likely to hold at a distance what you didn’t come up with yourself, subconsciously at least, even if you think you’re selling it. In one case, the speaker also described how some of the results themselves were discovered by AI, which probably further contributes to the impression that they hadn’t fully committed to what they are presenting. 

This has me wondering what the impact on diffusion of ideas will be as it becomes more standard practice to rely on AI for implementation in scientific production and communication. It’s funny how reserving skepticism for your own results often comes up in discussing epistemic virtues, but when speakers present as if holding their work at arm’s length, the result is not so informative. As we rely more heavily on AI in all stages of research, will we face more challenges in getting others to adopt our ideas? 

It’s also another reminder of how few people thinking about AI for science seem to have considered all the personal stuff that goes into the practice of science, with lots of irrational investment and fixation and stubbornness and pride to drive the loop of discovery and validation and communication. Scientific discovery may be an “ocean,” to borrow an analogy associated with Leibniz, but surfing it requires strapping oneself to a board and committing to seeing where it gets you, not just keeping it in sight while you splash around somewhere else. 

This also leads to a practical question of how you instill a sense of ownership, or at least commitment, to ideas that were partly produced by AI. My own experience is that it takes a lot of time to verify AI produced results before I get to the level of confidence I’d have if I’d done it myself. For complex tasks there will inevitably be decisions made along the way, e.g., about how to parameterize certain things in implementation or to deal with edge cases or other exceptions. Each of these has to be reconstructed before I can really feel that I stand behind the output. 

Is there an alternative? It makes me think of the “baking guilt” that housewives supposedly felt after cake mixes came on the market, because they only required adding water. There was a loss of a sense of personal contribution and emotional ownership. The solution, which persists today, was to have them add an egg. Some psychoanalysts went so far as to interpret this as symbolic of their fertility. For AI-aided science, the closest thing to adding an egg seems to be having agents explain at length to you what was done, which can still mean a big improvement over implementing everything yourself, but not as much of a boost as it first seems.

At any rate, interpreting the new challenges of AI-generated presentations of potentially AI-generated ideas as an aesthetic problem, or of “putting style before substance,” does not seem right. Scientific ideas don’t diffuse as bare propositions. They diffuse through people who have developed some passion for them. If we’re talking about AI for science, we shouldn’t be ignoring scientists and their relationships with what they do.  

There’s a old saying in biology that the development of the organism recapitulates the development of the species: thus in utero each of us starts as a single-celled creature and then develops into an embryo that successively looks like a simple organism, then like a fish, an amphibian, etc., until we reach our human form in preparation for birth.

Modern biologists don’t believe in this recapitulation. But taking this as an intriguing idea, I see an analogy with statistical practice.

Some version of this recapitulation occurs just about whenever we do applied statistics. We start with the simplest methods–univariate data summaries and some basic multivariate analyses–then we perform some comparisons which we check via standard errors and off-the-shelf hypothesis tests, then we move to modeling. We might well start with least squares and maximum likelihood and then move to regularization and multilevel modeling as needed, then throw in measurement error models, selection models, nonparametric this and that, and so forth.

The analogy isn’t perfect–in particular, we don’t always begin an analysis with simple averages and plots; sometimes we begin with a sophisticated nonparametric data-exploration tool such as lowess or deep nets. And, lots of methods for graphical exploratory data analysis have only been developed recently; indeed, even methods as basic as scatterplots are only a few centuries old.

Within the context of modeling, though, it does seem to me that we tend to start simple and then add more complicated features one at a time–and this seems like a sensible way to proceed. In so proceeding, we’re motivated in part by computational stability but also in part by the logic of increasing complexity: we take each step for a reason. Thus it is logical that statistical analysis recapitulates the development of statistical methods.

The story starts with a post from economist Greg Mankiw, who wrote:

Who has the best chance of beating Donald Trump? A clue can be found using Bayes Theorem.

Here is the logic. Let A be the event that a candidate wins the general election, and B be the event that a candidate wins his or her party’s nomination. Predictit gives us the betting market’s view of P(A) and P(B). It is a safe assumption that P(B|A) = 1, that is, a candidate can win only if nominated. We can then use Bayes theorem to compute P(A|B), the probability that the candidate will win the general election conditional on being nominated.

So here are the results for P(A|B) as of now:

Buttigieg 0.80
Biden 0.77
O’Rourke 0.67
Sanders 0.65
Booker 0.60
Yang 0.60
Harris 0.57
Warren 0.44

That is, the betting markets suggest that Mayor Pete would be the strongest candidate if nominated, with Joe Biden close behind. (Of course, these numbers will bounce around as the prices in betting markets change.)

By the way, when I [Mankiw] did a similar calculation in 2006, Bayes liked Barack Obama.

I copied Mankiw’s post in its entirety, with the only change being that he wrote P(A / B) etc., and I changed the slash to the vertical bar, P(A|B). (Are there people who write conditioning using a slash rather than a vertical bar? I had no idea. P(A|B) is more standard, I believe. In the above post, Mankiw links to the wikipedia page which uses the P(A|B) notation. No big deal, it just seemed odd to me.)

Anyway, I think the above set of calculations is a great example for teaching conditional probability.

The next step is to push a bit: Do we really believe these numbers? There’s nothing wrong with the probability calculations, but I’m not sure we should be taking Predictit’s betting odds as actual win probabilities.

To start with, I looked at Mankiw’s list and wondered what Yang was doing on it. Yang’s a fringe candidate, right? I wrote my post in June, 2019, and Yang was polling at 0.8% on Real Clear Politics then. I went over to Predictit and it said you can buy a Yang contract for the Democratic nomination for the price of 9%. OK, sure, at 0.8% in the polls there’s room for improvement. But 9%??? Seems like a lot.

The next think I’m worried about, beyond bias in the online markets, is volatility.

Sure, Mankiw writes, “these numbers will bounce around as the prices in betting markets change,” but I think he’s not fully appreciating how noisy these numbers are!

Mankiw’s post is dated 27 Apr 2019. Predictit conveniently gives prices going back a few months, so I could do some Biden-Warren price comparisons of then to when I was writing my post:

27 Apr 12 Jun
Biden primary election 22 28
Biden general election 17 19
Warren primary election 9 19
Warren general election 4 13

Something weird was going on in April, when Biden’s price was 22 for the primary and 17 for the general election. This just can’t be right, and all I can conclude is that the betting markets here were thin enough that nobody was taking these numbers very seriously.

If you want to take the numbers as is, you’ll get the following:

27 Apr: Biden 17/22 = 0.77, Warren 4/9 = 0.44
12 Jun: Biden 19/28 = 0.68, Warren 13/19 = 0.68.

These numbers aren’t quite right, even if you take these betting markets seriously, because of rounding and the vig. If you add up all the prices on the “Who will win the 2020 Democratic presidential nomination?” page, you get something well over 100%. So you can’t directly interpret these prices as probabilities, even beyond the issues of bias and noise.

I discussed this with David Rothschild, who thinks a lot about elections and prediction markets (for example, here), and David responded as follows:

People ask me to compute this automatically on my blog, but I refrain, because it is so noisy this early. Here I compute the conditional probability range separately for Betfair and PredictIt, by diving the seller’s price of win / buyer’s price of nom & buyer’s price of win / seller’s price of nom. Betfair has advantage of being tighter by definition (PredictIt trades on the penny, but Betfair on the odds, which have more depth).

Here is a figure from an old paper I wrote with David Pennock about the 2012 election. As you can see, while informative, it can get quite noisy!

Anyway, my point here is not to criticize Mankiw but rather to thank him for putting out this fun example, and then to demonstrate how we can take it further by interrogating each step in the analysis. Which is how we do applied statistics in general.

P.S. In case you’re curious, based on the numbers when I wrote my post, where Biden’s implied electability is 19/28 = 0.68 and Warren’s is 13/19 = 0.68, we can look up Buttigieg. He was at 9/16 = 0.56, the least electable of the three. So, no, Bayes did not like Mayor Pete that day.

It’s a fun example, but when we look at the data more carefully, the original conclusion goes away.

(The polar bear celebrated the blog series birthday with a rainy hike on the PA AT. Here he is attempting to dry off.)

Let’s zoom in on the part about the Generalized REGression estimator (that doesn’t specifically say “GREG”). Green anotations are mine:

Meng (2022)‘s (5.2) is the first way of writing GREG in our post “GREG”, from Särndal, Swensson, Wretman (1992):

That book goes on to say that GREG often takes a super simple form:

Meng (2022) doesn’t mention this as far as I can tell ? Although I think Meng’s example satisfies the conditions the book Särndal, Swensson, Wretman (1992) goes on to describe: the regression model assumes constant variance and has an intercept.

Anyways, back to the title of this post. Meng emphasizes that GREG is not only “double robust” (consistent if either the outcome model or response model are correct), but “double-plus robust” (consistent if what is left of the outcome model and response model are uncorrelated). I’m interested in the practical implications of this, such as the suggestion to include the estimated response probabilities in the outcome regression model. Thoughts ?

A new one of these came up when Palko pointed me to this disturbing news article, “They Look Like Slot Machines. They Pay Out in Cash. And Critics Say They Are Getting Workers Killed,” which reports:

Store clerks in Pennsylvania have been robbed and shot while handling payouts for “skill games,” which are not subject to the security standards required of gambling operations. . . .

They look like casino slot machines and video arcade games, but they are neither. They are skill games. Like their name implies, players must use their skills — memory, reflexes, strategy, recognition — to win cash. They don’t solely rely on the luck of the draw, like with slot machines. . . .

The Pennsylvania Gaming Control Board licenses 17 casinos and 75 truck stop video gaming terminal facilities, requiring them to have secure facilities, trained staff, and digital video recording. Their gambling machines also have to be linked to a centralized computer monitoring system. Businesses that offer skill games are not held to any standards, their critics say. As a result, some are putting their employees in danger by having them pay winners with cash. . . .

Some gruesome stories follow, along with predictable quotes from evil people making money off these things.

“Skill games”?

But here’s my question. How much skill is actually in these “skill games”? I assume not much, because, if the games really did involve skill, then skillful players could just show up and win regularly.

I guess the “skill games” could involve some small amount of skill, but not enough so that skillful players could beat the house edge.

The Ten Year Affair employs a storytelling device also used by Lionel Shriver in The Post-Birthday World and also by whoever wrote that movie, Sliding Doors, with Gwyneth Paltrow. Somers did it better, though, in two ways. First, she leans into the reality that both threads of the story are fiction, and her protagonist is aware of the two threads. This is the right thing to do, because the point of an alternative timeline is not just that it’s something else that could’ve happened but also that we’re aware of the possibility: that other hypothetical world is always there in the periphery, just outside of reach and informing our actions in the real world. The second way that Somers did better than Shriver and that screenwriter is that she (Somers) avoids an easy way out. In The Post-Birthday World and Sliding Doors, the husband or boyfriend is a bad guy, and so, in the logic of the story the heroine is automatically justified in trying to find someone else. This removes the dramatic tension. In The Ten Year Affair, the husband is far from perfect, but he’s not cheating on his wife. The other thing done well in The Ten Year Affair is that all the characters are flawed. You’re seeing things through the eyes of one main character, and early on we get a sense of the goofiness of all the people around her–indeed, she bonds with one of her friends based on a mutual distaste for a mildly obnoxious third party–, but, as the book goes on, we see the viewpoint character’s flaws too. I like that Somers is willing, in the end, to make that main character as flawed as everyone around her, which I guess fits the there-is-no-escape theme of the book.

As a side note, point of view is done very rigorously in the book, to the extent that you get a sense of what all the characters look like, except for the protagonist, because she doesn’t need to describe herself, right? I guess the main thing we learn about the main character, regarding her looks, is that she’s not particularly insecure about her appearance: she doesn’t think she’s the most beautiful woman out there but she doesn’t really worry about her looks either. To me, this sort of rigor contributes to the pleasure of reading a book: I feel comfortably in the hands of a confident storyteller.

Going back a bit in literary time, The Ten Year Affair is a lot like the novels of John Updike: various suburban married couples having affairs. The writing style is different–Updike is famously lyrical, whereas Somers uses a Millennial flat writing style: This happens, then This happens, then That happens, etc. Kind of like Ernest Hemingway or Raymond Carver if they had a sense of humor.

I think Somers does a much better job than Updike in conveying what it feels like to be a parent. To me, Updike, like Philip Roth, was to the end of his life always a son, never a father. Updike did have four kids, but I guess his wife did most of the parenting. Updike’s characters often have children but always seem to be thinking only about themselves. Not so much that his adult characters are self-centered–I mean, yeah, they are, but that’s kind of the point–but more that their children don’t seem to exist at all, except to the extent that they sometimes have to be dealt with as obstacles when they get in the way of the parents. In contrast, the adults in The Ten Year Affair are very aware of their kids. In some ways this is similar to Little Children by Tom Perotta, a book whose entire theme is that these adults are thinking only of themselves and are not shouldering the responsibilities of parenthood.

The children in The Ten Year Affair are real people, but they don’t come to life as much as, say, the children in the novels of Meg Wolitzer. Wolitzer achieves an equality across generations that I rarely see in literature; perhaps it has something to do with her being a Boomer, coming from a generation in which kids are central.

P.S. I looked up Somers and it turns out she’s a fan of Lorrie Moore. Cool! I’m a fan of Lorrie Moore too. Although one thing that always annoys me about Moore is that her stories always seem to feature a main character who is a woman who is a good person and has to deal with asshole guys. Yeah, I get it, there are a lot of assholes out there, but Moore’s protagonists are always so clever and thoughtful that I find it frustrating that they keep coming across as victims. I think Somers does better in this area. Her appreciation of Martin Amis is good too. I think Somers should come out with a book of literary essays. I’d buy it.

Anyway, the book was interesting. Short story is that there was a mediocre young physics Ph.D. who achieved some success by faking data, he got a research job at Bell Labs, faked more data, got more success, eventually got too much success in that people were interested enough in his findings that they tried to use his methods themselves, but the methods never worked. After a bit more struggle he then got fired.

Compared to the usual glacial pace of such scandals in academia, the whole thing was pretty quick and clean. From receipt of Ph.D. to getting fired was just a bit over five years. That might seem like a long time, but it’s quick compared to the careers of Wansink, Ariely, Wegman, Hauser, etc.–not to mention the many bullshitters in social and medical science who are still out there.

I think the key difference between this physics case and fraudulent science in cognitive and social science is that when a key part of any physics paper is its methods. If a finding is interesting, and other researchers want to get into the game, they’ll want to start by replicating the experiment. If they can’t do that, they’ll get upset. In contrast, if you want to follow up on a political science or economics study, you’ll look for new dat, and if you want to follow up on a psychology experiment on embodied cognition or mind-body healing or whatever, you can just do your own experiment from scratch. There’s no particular method that you have to use.

So in social science junk papers like those discussed here or here can stay around forever–even questionable papers written by Nobel prize winners–because people only care about the results, not the method.

Here’s another interesting difference. Those three papers discussed in the link in the previous paragraph are not fraudulent, they’re just bad science, some combination of noisy data, bad theory, bad measurement, and statistical misunderstanding. Here’s the relevant principle: in the social and behavioral sciences, you can get prominent bad results by accident. In physics, you pretty much have to cheat. Sure, experimental physicists get fluke results all the time, but then they don’t replicate and the field moves on. To really stick the landing on a non-result in physics you have to cheat.

Social scientists who do junk research or even fraud can stay afloat forever, but that cheating physicist was headed for a fall, once his research got some attention.

Also, it seems that this guy was a real asshole. Not only did he lie, and then lie again to cover up, then later when it all came out and his university revoked his Ph.D., he sued them. What kind of person would do that: defraud the education system and then sue them?

At this point I should step back and think like a statistician. Some percentage of people are unscrupulous assholes, and some percentage of people are physicists. Draw the Venn diagram: you’ll expect to see some overlap.

In any case, this is all so clean compared to what happens in social science.

This line of reasoning takes two things that may resemble each other on the surface (e.g., what humans report their reasoning or belief or intention to be, and what machines report) which we have no solid reason to believe are produced by similar processes, and says they cannot be distinguished because we expect both to be reported with loss.

It’s bad logic. But more than that, it occurs to me that by doing this we indirectly deny the value of our subjective, non-verbal experience. It’s true we may not be able to faithfully report how we reasoned or what exactly we desired or intended or understood. But part of the reason we have terms for these things is because we perceive a distinction between more genuine and less genuine versions of them within ourselves. These qualitative distinctions are how we come to believe that reasoning and belief and intention and desire exist in the first place: because we can perceive them as being authentically present to different degrees in different situations, even if sometimes we seem to be mimicking the real thing rather than really doing it. It’s what licenses researchers to keep trying to study these things in people despite the difficulty of getting an unbiased read. I’m not saying we shouldn’t look for analogous processes in machines. But we should acknowledge that the referent for these terms is grounded in subjective experience. Among humans that has worked fine, because we assume ourselves to share an understanding of what it’s like to have an inner life.

Andrew recently analogized chatbots with being in autopilot in conversations that occur in a meeting, where if you have a rich enough memory bank of observations related to what the speaker is saying, you can spout appropriate conjectures and feedback without having to exert much conscious effort. But I think fake thinking extends to much more than just being on autopilot in conversation. It can seep into all of our decisions, whether about research ideas or methods or life decisions like what job to have or where to live. It’s very tempting to live by patterns, even when it’s not what you feel you actually want.

Our perceptions that there are more real and more fake versions of our own thinking or reasoning or belief or desire is what makes us feel more “alive” or “awake” in some situations over others. It connects us to the present. Only when we try harder to do these things authentically or recognize the motives that are actually driving us (despite the explanations we might want to assume) do we feel like we are really living, rather than just performing some role. And so our ability to perceive the difference seems intimately connected to the process of understanding ourselves.

From this angle, it is unfortunate how much of the AI rhetoric we’ve come to take for granted (at least in machine learning) — i.e., AI as scientist, AI as decision-maker, etc.— enacts this implicit move of equating human and machine processes by their outputs, and consequently subtly devaluing the role of our internal reality in giving these terms meaning. We take examples from the few domains where language can fully capture reasoning–math, coding–and we reduce all reasoning or thinking or intention to what can be made manifest.

Maybe this helps explain why there’s so much emphasis lately in certain circles (including tech) on being “high agency.” We become more insistent about shaping our external world as we lose appreciation for our internal one.

1. Government actors are directly threatening both private citizens and government employees to suppress dissenting speech.

2. The attacks on free expression are notable but they’re still very rare. Most people in this country can still say what they want in public without threats.

3. The pressure from the government is coming down against what are perceived to be left-leaning views, and the government is attacking left-leaning individuals and organizations, along with non-political individuals and organizations that the government is portraying as aiding the left.

4. Joe McCarthy in the 1950s had a lot of similarities with Donald Trump in the past decade. See my post from 2016 discussing this point.

Now, some differences:

1. Back in the Cold War period, there was a direct threat from the left. The Communist party and related organizations were allied with the Soviet Union, a rival power. Far-left groups in the U.S. were generally nonviolent (late 1960s and early 1970s aside) but, at least in theory, they advocated extra-legal overthrow of the U.S. government. Nowadays it is the right that has the violent rhetoric and it is the right that tried to violently overthrow the government. In the Cold War, there were right-wing organizations, but, setting aside what was happening in the South, they weren’t attempting to hold power by violent means.

2. In the mid-twentieth century, the Communist Party ran its own network of libraries and schools. (For example, here’s the story of Mary Knowles, the librarian at the William Jeanes Library in Plymouth Meeting, Pennsylvania. That was our local library when I was a little kid. It says in the above-linked article that “from 1944 until 1948, she’d been the secretary at the Samuel Adams School for Social Studies, a Communist Party–funded school for adult learners in Boston.” In that way, the far left back then is similar to the far right today, with alternative infrastructure, alternative historical narratives, etc. Back then there was the Daily Worker, now there’s Fox News.

3. From the other direction, there are many institutions that were on the right back in the 1940s and 50s but are in the political center now, such as the Roman Catholic Church and much of big business. There have been many political shifts, organized labor is much much less of a thing, the ultra-rich are loud and powerful in a way that, with rare exceptions, they were not during the Cold War period, etc.

I have a lot of sympathy for the liberal anticommunists of the 1930s-1950s: these are people who took a lot of flak from the left and the right at the time and don’t get so much respect now, but in retrospect I admire their willingness to take a political hit to fight authoritarianism within their own part. I don’t have much sympathy for McCarthy etc. who just made stuff up and lied about people, also there’s a difference between kicking extremists out of a political party and trying to control the speech and political activities of private citizens and government employees.

So what’s the point of all this comparison? Just that in many ways what’s happening now with our authoritarians in power looks like what was happening in the 1950s (but with much less constraint now, in large part because one faction of one party controls all three branches of government), but with the big difference that now there are large, well-organized, and well-funded set of parallel institutions on the right—not just the Republican party, but far-right news organizations, street gangs, lawyers and judges, political donations, etc.—as compared to the 1950s when it was the far left and labor unions that had powerful alternative institutions.

I’m not sure what this all implies; I just think that the similarities are easy to see but the differences are important too.

In case you haven’t seen it, check out this recent piece in Rolling Stone. A key paragraph toward the end:

Craig Callender, a philosophy professor at the University of California San Diego and president of the Philosophy of Science Association, agrees with that assessment, observing that “the appearance of legitimacy to non-existent journals is like the logical end product of existing trends.” There are already journals, he explains, that accept spurious articles for profit, or biased ghost-written research meant to benefit the industry that produced it. “The ‘swamp’ in scientific publishing is growing,” he says. “Many practices make existing journals [or] articles that aren’t legitimate look legitimate. So the next step to non-existent journals is horrifying but not too surprising.”

The one point I [Peter] would add is that the first step to perdition is citing sources you haven’t actually read or at least looked through yourself, relying instead on what other people said about them, or simply that they were used as citations in other articles. I know I’ve been tempted to do this sometimes because it takes extra time to do it right, and I may be in a hurry, but then I manage to impose a little integrity on myself. After you’ve taken the first step toward “blind” citation, however, everything else becomes possible — and now even likely.

This reminds me of something that my adviser once told me, that he made a principle of never putting his name on a document that he hadn’t read.

As to citing sources, yeah, I agree with Dorman 100%. You should only cite sources that you, or one of your coauthors, have read or have at least looked through.

A related point: When you’re reading a paper and going through its references, you will sometimes find that the paper described the reference inaccurately. As we wrote in Section 3 of this paper:

Unreplicable claims based on weak theory can gain apparent support by connections to related published work. Three problems can arise.

First, the connections between the cited literature and the new study can be tenuous, and this can particularly be an issue when the underlying theory is vague. Ideas such as embodied cognition, evolutionary psychology, nudging, mindfulness, or mind-body unity are general enough to encompass a wide range of potential phenomena, to the extent that there is almost no limit to the past studies that could be thought to have some possible relevance to any new experiment.

Second, informal literature reviews are subject to selection bias. An article promoting a controversial idea can easily cite studies claiming to have found evidence for related ideas, while avoiding citations of failed replications or papers suggesting alternative theories. This can even be a problem with systematic meta-analyses, if the entire subfield being meta-analyzed is full of studies with uncontrolled researcher degrees of freedom.

Third, the interpretation of individual studies being cited can be seriously flawed. This is a problem of citing past literature as support for a general claim without looking at exactly what was done in the cited research and without following up on that work. Here we discuss three different examples of this sort of misinterpretation of the literature cited in the paper under discussion. . . .

So, yeah, don’t trust what someone writes about a cited study. Take a look at it yourself.

P.S. I wrote this post several months ago. It’s just a coincidence that it happened to show up the same week as What do I think about that proposed Arxiv policy to ban authors of papers with AI slop?.

Mitzi Morris and Bob Carpenter, two of Stan’s developers, will be presenting a tutorial on Stan and Bayesian data analysis aimed at psychometricians this summer.

• Modern Modeling Methods Conference (M3), Fordham University Lincoln Center Campus, Manhattan, June 22–24
• Program here

Abstract

This workshop is a full day, hands-on introduction to Bayesian modeling and statistical inference using the probabilistic programming language Stan.

The course will be organized around the key properties of Bayesian statistical modeling for science, including the nature of uncertainty, modeling a generative process through a data generating distribution, modeling existing knowledge through a prior, and pushing uncertainty through inference. As we do this, we will show how Stan can be used to both code the models and perform statistical inference for quantities of interest, be they retrospective parameter estimates or prospective predictions or forecasts. We will concentrate on full Bayesian posterior inference, including a discussion of calibration, model checking for both prior and posterior inference, and model comparison with cross-validation. We will spend some time showing how some structural equation models (SEM) can be translated directly to Stan and will also introduce psychological models for educational testing, crowdsourcing, rating and ranking, and real-time decision processes.

This class will require a notebook computer with a network connection (Wifi will be available in the classroom). We will use the Stan Playground, which runs Stan in the browser, which we will pre-populate with models of interest. We will probably also break into R or Python at various points to demonstrate methods not yet supported by the Playground, such as the brms regression expression language.

Andrew on psychometrics

Andrew once told me that any model you could come up with was probably invented by a psychometrician 50 years ago (make that 60—he said it at least 10 years ago). I have evidence that he’s right form the project that drew me into Bayesian statistics—crowdsourcing. Andrew and Jennifer Hill helped me formulate a crowdsourcing model where raters give you noisy measurements of underlying categorical variables (e.g., they answer survey questions about whether a word in context is a noun, for example, to use something I was working on at the time). Turns out Phil Dawid and A.P. Skene published the same model in 1979 in one of the earlier applications of the expectation maximization (EM) algorithm and they used natural language data (drawn from medical records).

The rest of the conference

The rest of the program looks really great—it’s just the kind of applied wrestling with real data that I like.

Speaking of Jennifer Hill, she’s one of the keynote speakers at M3. Every talk of Jennifer’s I’ve attended has been great. You may know her as Andrew’s co-author on the regression books, which I cannot recommend highly enough if you’re interested in this kind of applied modeling.

Beyond the conference

You see the same kind of Bayesian modeling focus for real data at venues such as ISEC (international ecology conference to which I went to once just because I like these models and these kinds of conferences), StanCon (see you in Uppsala in August!), and GeoMed (which Mitzi attends). I’m sure there are more in other fields. I’m always disappointed that there’s almost nothing like these kinds of nitty-gritty applied papers at ISBA (Nagoya this summer) or BayesComp (somewhere next year). The conferences about Bayesian statistics or computing that I’ve been to have all been super theoretical.

One thing I like about this book is its modesty: it doesn’t push any big theories, instead going through several examples of hyped psychology research that found its way into public attention and policy. The examples are “self-esteem,” “superpredators,” “power pose,” “positive psychology,” “grit,” “implicit bias,” “social priming,” and “nudging.” I’m putting all these terms in scare quotes because they’re slogans as much as anything else.

I don’t want to say that Singal offers no big-picture perspective. His larger theme, beyond his discussion of how shaky ideas can get publicity and influence, is that that this sort of push-a-button-change-your-life version of psychology is appealing as a way for people to make sense of their lives in a time of uncertainty:

The world is a big and scary place, and imposing structures often circumscribe our room to maneuver. That’s why the reductive storytelling of grit is so appealing; that’s why we turn to figures like Martin Seligman for positive thinking that can shield us from trauma, or Amy Chua for parenting advice, or Angela Duckworth for the secret to grittiness. There’s always the idea of the side entrance; there’s always something you, the individual, can do to regain control in a world that sometimes seems hell-bent on robbing you of it.

Well put. The question, then, is what’s new about this? Dale Carnegie’s “How to Win Friends and Influence People” came out in 1936, has sold a zillion copies since then, and remains in print. Before that there was Christian Science and all sorts of self-help movements. One thing that distinguishes “grit,” “nudge,” and “positive psychology” from its predecessors in the bestseller list and the pulpit is that these more recent ideas are coming from academia and have elite media endorsements. Some of this could just be that the academic world is much bigger than it used to be. Just as many people who in the past would have been professional novelists but now support themselves as university teachers, so it could be that professors such as Duckworth, Sunstein, etc., would in the past have been freelance authors or speakers. Indeed, the recent rise of celebrity podcasters has perhaps taken us back to the Dale Carnegie era.

Some of this came up in our Rationalizing the World course: the idea that intellectual movements can come from academic social science or from the popular or commercial world. It may be that the “Quick Fix” era of 2010-2015–the high-water mark of Psychological Science, PNAS, Ted, and NPR–was a transient period.

The celebrity-academic-social-science era took off in 2005 with Freakonomics and ended in 2020 with Covid. If ever there was a time for society to cash the check of the promise of academic social science for society, it was the pandemic. And academic social science pretty much failed the test. Science didn’t fail–vaccines were rapidly developed, tested, and implemented–but social science didn’t do so well, and the authority of science in society failed, as evidenced by vaccine deniers taking over our public health establishment.

The thing that really struck me about Singal’s book, though, was how it seems like a product of a much more innocent time than today. The book came out in 2021 so it was finished in 2020, but I wonder if much of it was written a few years before that. I say this for two reasons. First, he was so gentle on many of the researchers. He keeps giving them the benefit of the doubt. He even had good things to say about the notorious Roy Baumeister!

Second, Singal’s political take on junk science was so mellow. Yes, he considers it to be an important problem–he did write a whole book on the topic!–and he makes the political point that supposed “one quick trick” tweaks (or as Brian “Pizzagate” Wansink would say, “tweeks”) can be viewed as a distraction from serious political reform, in the same way that do-it-yourself environmentalism could be a substitute for real action on global warming. As Singal says, one of the problems with the “quick fix” mentality is that it places the burden of societal problems on individuals, which might not be such a bad idea, except that these quick fixes don’t work. There are also some logical issues that we’ve discussed which he doesn’t get into, such as the piranha problem that all these purportedly large effects can’t coexist and the competition problem that, even if something like “power pose” worked as advertised, it won’t help you get that job if your competitors are power posing too.

OK, yeah, so there’s some political content. But, as of 2017 or 2020 or whenever most of the book was written, there was less of a sense that the political stakes were high. Yes, there was political polarization; yes, the 2016 election featured two unpopular candidates; etc. But it wasn’t like now, where the government is shooting protesters on the street, shooting people in boats, starting wars, etc. Not to mention government science policy: global-warning denial, vaccine denial, and public health authorities that are such a joke that the best thing that even their supporters can say is that the new dietary guidelines is that they’re “not crazy”–yeah, that was supposed to be a positive reaction.

Now, partisans of different sides will have different reactions here. The Democratic take is that the Trump administration is a bunch of corrupt liars. The Republican take is that the science establishment has been in the tank for the Democrats for awhile now, and there’s no good alternative but to shake things up: it’s too bad about the stupid dietary guidelines, but the only way forward is to start over and show the public some respect.

My point here is not to try to resolve this political debate, but rather to emphasize that it exists, and to a much greater extent now than six or ten years ago.

One way to see this is to compare Singal’s book, The Quick Fix, to the podcast If Books Could Kill by Michael Hobbes and Peter Shamshiri. In content, Singal’s chapter on “grit” is very similar to Hobbes and Shamshiri’s recent episode on the topic–but the tone is much different. Part of this can be attributed to politics–Singal is center-left, while Hobbes and Shamshiri are far-left (at least, in the U.S. context)–and part could be just differences in personality: maybe Singal is just a mellower person. But I don’t think it’s just that, and I say this in part because I think Singal really is bothered by science hype. I think it’s mostly that the stakes are higher now. As Hobbes and Shamshiri say, the “grit” phenomenon is mostly played out, but educational fads haven’t gone away. (A weakness of If Books Could Kill is that it’s so politically slanted that even when the hosts are getting all the facts right, they can be missing important context, for example not talking about various liberal-leaning educational fads of recent decades.) But the issue here isn’t really “grit” or the associated publicity campaigns or even education policy; it’s more the sense from people on both sides that the stakes are high.

I think it would be harder for a journalist like Singal to write a book like The Quick Fix with such a mild tone today.

Not that the replication crisis has gotten worse–it’s my impression that in most ways the promotion of junk science has diminished. Gladwell, Freakonomics, and Ted are still around, but I think they take up a smaller fraction of the public bandwidth nowadays; NPR and other journalistic outlets seem more likely to approach new claims with skepticism, even the Association for Psychological Science has reformed a bit, finally publishing my letter refuting their earlier-published lies about me. Nobody’s calling us terrorists or Stasi anymore, either.

So, yeah, I do think social-science hype has diminished. But, outside of academia and the news media, bad things are happening: vaccine denial and all the rest. And I think it’s hard to write about problems in science without all that casting a shadow. The Quick Fix was written in a more innocent time.

To put it another way, examples such as Stanford’s Andrew Huberman shilling supplements or Harvard’s Cass Sunstein partying with Henry Kissinger may have in the past seemed like peccadillos, somewhat relevant to the exploration of bad judgment by credentialed scholars but not so important to the big picture. But now that the manufacturers of supplements have taken over the public health authorities and the modern-day Kissingers are starting new wars, all this takes a more serious turn. I’m not trying to blame Huberman or Sunstein for our government’s misdeeds (or for the misdeeds of the Biden administration, for that matter); I’m just saying that the corruption of science and the science media is concerning now on a new level beyond what it was as of the writing of The Quick Fix, and I think that any new book on unreplicable science would have to address these larger issues.

I’m curious what your thoughts are on the new arXiv policy that authors will be banned for a year if their paper includes mistakes due to the use of AI.

My (uninformed) thoughts:
1. arXiv is acting more like a publisher here. Interesting, and probably a good thing given how it’s used.
2. In an ideal world, every author on a paper would be intimately involved with the writing of the paper and be familiar with every citation. But that’s not realistic in many fields! Some papers have dozens of authors on them, and only a few of those authors are directly involved in the writing of the paper.
3. I understand that misconduct in research is an issue and that we should hold academics to high standards, but I worry this will have a chilling effect on collaboration. If someone reaches out to you to collaborate, the floor for your effort has gone way up.

What are your thoughts?

First off, I’ve been told that this if this policy were applied, it would only apply to the computer science part of Arxiv. CS is already out of control with publishing: there are a zillion Ph.D. students, each of whom is pressured to publish something like 8 conference papers a year, and I guess a lot of these go to Arxiv. And remember this story from last year?

Setting that aside, I guess that this ban could happen to me, if I collaborate with somebody who inserts AI slop into a paper. I already know people who tell me they use AI to write emails for them, and it doesn’t seem so much different for this to be used in text for a research article. Maybe the next step is to require coauthors to sign a form attesting that they have added no slop (“inappropriate language, plagiarized content, biased content, errors, mistakes, incorrect references, or misleading content”). On one hand this sounds pretty extreme; on the other hand, the actual examples given by the Arxiv guy (“hallucinated references, meta-comments from the LLM (‘here is a 200 word summary; would you like me to make any changes?’; ‘the data in this table is illustrative, fill it in with the real numbers from your experiments’)” seem like they should be catchable. The big concern would be if there’s a jointly-authored article and the AI-slop-inserter happens to be the last person to have edited it before submitting to Arxiv.

That all said, this could well be a better alternative than the current situation where I guess they’re getting lots of slop. Fake citations are a thing. The Arxiv people are doing their work for free, or close to free, so if they’re getting overwhelmed, they have to do something. The complainers on the above-linked twitter thread seem to miss the point in that they’re implicitly evaluating the policy based on its potential costs without balancing against the benefits.

First responded:

I similarly wondered if requiring some coauthors to sign something could help. Maybe a single author on each paper should declare themselves responsible for the contents of the paper and take the blame if the paper has slop in it? I think that would undermine most of the concerns around collaborations.

I guess the simplest rule would be for the first author to take responsibility by default. Setting aside chatbots and all the rest, there’s a problem in joint-authored papers with plain old incompetence or fraud. Consider the unfortunate case of Dan Ariely, a scrupulously honest researcher who has had the misfortune to repeatedly be coauthor on papers involving fake data. Multiple authors and nobody takes responsibility.

I also asked Arxiv founder Paul Ginsparg, who replied:

My thoughts are more along the lines of what do we do in three months when
(a) the median LLM-produced cs paper is better than can be produced by the median cs grad student.
and what do we do in six months when
(b) an elementary school student produces a paper with a legitimately important theoretical result in physics, but sans LLM can’t explain a word of it.

For expanded version of these thoughts, here’s a recording of a recent colloq I gave that includes arXiv data on rejections.

Here’s the abstract of Ginsparg’s talk, “The Rise of Slop”:

The rapid proliferation of AI-generated text, figures, code, and reviews is reshaping the ecology of research communication. AI has made it cheap to produce text that looks like knowledge. The result is an emerging deluge of “slop”: fluent, confident, and often content-light material that strains the filters of research and academic publishing. While generative tools promise efficiency, accessibility, and assistance, they also undermine the fragile social contracts on which scholarship depends: authorship, attribution, review, and trust. This talk argues that the core danger of AI-generated slop is not isolated fraud or error, but volume: content produced faster than it can be meaningfully evaluated. The rise of slop forces a choice: retrofit our scholarly systems for an age of cheap language, or watch signal drown in noise.

[Disclaimer: this abstract was 100% LLM-generated.]

I’ll conclude by saying that we’re proceeding on two tracks. Or maybe I should say there are two worlds.

In one world, we’re doing research as before, using computers as tools, trying to understand what’s going on, and writing up our results as research papers. For example, just the other day I met with some students on our project of agent-based modeling of coalition formation (a followup of this research from 2003). We have some results, we’re trying to figure out where to go next, and we’re writing a paper together.

In the other world, students are pressured to whip out a high volume of papers and get some of them accepted at top conferences and journals. You improve your odds by submitting more things, and you can “do a Bruno Frey” and publish the same idea five times. If that’s the way you wanna go, a chatbot can be really helpful! Not just to refactor your code or clean up your writing, but to create those many many papers that you’re planning to submit to all those places.

It’s weird that these two worlds are coexisting. Kind of like how, in publishing, there’s a world of people writing and publishing books that other people want to read, and another world of fake books being spammed on Amazon to catch the suckers. Or how we get a mix of real blog comments, spam, and those weird intermediate cases where the commenter seems to actually have something to say, but then the url is some spam link. Usually I delete those, but sometimes I keep them and just strip out the URL and email address before posting.

We talked about using a model for response R, a model for outcome Y, or both. So GREG came up, and Andrew asked “what’s GREG ?” Good question.

GREG is Generalized REGression estimator. Särndal, Swensson, Wretman (1992) has a nice section that writes it in a few alternative ways:

1. Adjust an estimate based on the model with a Horvitz-Thompson estimate of the error:

2. Or on the flip side, you can see it as adjusting the Horvitz-Thompson estimate with the model:

It’s called GREG for Generalized REGression estimator, what is being generalized ?

Lumley 2010 made me think we were generalizing to continuous X variables:

Sharon Lohr’s book made me think we were generalizing beyond simple random samples:

Särndal, Swensson, Wretman (1992) made me think we were generalizing to multiple X  variables:

Regardless of the exact origin of the name, GREG has connections to the Doubly Robust literature in causal inference (as Coston et al. (2020) note in a footnote). Any favorite references making these connections ?

In September 2022, Wiley identified and immediately alerted the industry to paper mill activity we found operating at scale. Specifically, we found fraudulent outside editors that had subverted our processes and workflows, leading to a proliferation of bad content. This scheme hit Hindawi’s Special Issues program hard.

For those who are unfamiliar with academic publishing: Wiley is a long-established firm.

Back when I was a student, Wiley was perhaps considered the #1 publisher within statistics. They published Feller’s classic books on probability, Cochran’s classics on design of experiments and survey sampling, and many other standard texts.

In recent decades, as with other academic publishers, they’ve branched out into other publishing-related businesses, for example, Hindawi, which has a habit of filling your inbox with spam about dodgy journals. From Wikipdia: “In 2023 and after over 7000 article retractions in Hindawi journals related to the publication of articles originating from paper mills, Wiley announced that it will cease using the Hindawi brand and will integrate Hindawi’s 200 remaining journals into its main portfolio. The Wiley CEO who initiated the Hindawi acquisition stepped down in the wake of those announcements.”

To those of us of a certain age, seeing Wiley and Hindawi in the same sentence is disturbing in itself, a sign of what the world of publishing has come to. Not that publishing has ever been pure—just for example, back in the 1960s and 70s, legitimate publishers released fake-science books such as Chariots of the Gods, The Bermuda Triangle, and The Jupiter Effect—; still, it was sad to see the once-respected Wiley name dragged so low.

You can hire James Heathers for less than $3.7 million

Heathers points out that, because Wiley is a public company, certain of its business records are required to be public, and he found this:

Heathers explains:

‘Legal settlement’ is exactly what it sounds like, and the footnote description is ‘a litigation matter related to consideration for a previous acquisition’.

The shorthand is: their own shareholders sued them. They said they were going to, and did. . . .

This is not uncommon . . . Any large public company in business for long enough has seen a suit or two like this. . . . Generally, they settle. . . . this is noticeably more expensive than running a full-scale proactive research integrity program.

And here’s the kicker:

For 3.7M, you could have the world. I [Heathers] am quite confident in saying: I could run that as the operating budget of a fraud mitigation unit for multiple years, and drop the amount of nonsense by . . . maybe two-thirds, three-quarters? within that time.

All right, then!

This is not new to Wiley

Just one thing. This is not new. Wiley’s been in the lucrative science-fraud business for awhile. Recall this story from 2011, “Wiley Wegman chutzpah update: Now you too can buy a selection of garbled Wikipedia articles, for a mere $1400-$2800 per year!”

But, yeah, the Hindawi business sounds a lot worse. When Wiley was conned by a formerly respected academic into republishing Wikipedia content and charging money for it, that was just a one-time breach in editorial standards. The Hindawi story seems like something else entirely. On the other hand, when it comes to fraudulent publishing, they had some track record.

Adversarial journalism

Heathers writes:

There absolutely IS adversarial journalism in academia/research/science/etc. Science Magazine, Undark, Vox, etc. have all published great pieces on this.

Ahhhh, Undark Magazine . . . that brings up memories. A few years ago, Undark published a terrible article, misrepresenting a scientific story in which I’d been involved. That was adversarial journalism in the worst sense, in that the journalists were coming in with an agenda and using it to distort and slam anyone who disagreed with it. See here for more on that story.

I’m not saying you shouldn’t trust anything in Undark just cos they ran that one bad article, any more than I’d un-recommend the books by Cochran just cos they were published by Wiley. I just thought it was funny that Heathers mentioned Undark in particular, given that my only experience with that magazine was so unpleasant.

Multilevel regression and poststratification (MrP) has become a workhorse method for estimating population quantities from non-probability surveys, and is the primary model-based alternative to traditional survey calibration weighting methods, such as raking. For simple linear regression models, MrP methods admit “equivalent weights”, allowing for direct comparisons between MrP and traditional calibration weighting. Such weights, however, have been unavailable for the most widely used MrP models, such as logistic regression. In this paper, we develop a natural generalization, “MrP locally equivalent weights” (MrPlew), which represent MrP as a weighting-style estimator that is locally equivalent to calibration weights near the observed responses.

Cool! This goes beyond my 2007 paper, Struggles with survey weighting and regression modeling (“for logistic regression, the poststratified estimate is no longer a weighted average of the data, even after controlling for the variance parameters in the model. However, we suspect that the model could be linearized, yielding approximate weights”) and our 2004 paper on dilution assays, in particular Section 5.2, “Equivalent weights for nonlinear models.” The funny thing is that I forgot about that 2004 paper when working on equivalent weights for MRP in the 2007 paper. Also, the 2004 method won’t work as is, because it’s designed to estimate sensitivity to individual data points, not to produce good weighted averages.

I say this not to try to claim credit for the method of Giordano et al., but rather the opposite, to emphasize that even though I’ve been thinking about equivalent weights in MRP for a long time, I haven’t yet succeeded in getting them to work in practice, so I’m very happy to see developments in this area.

One thing that came up with equivalent weights when we tried to apply them in practice is that sometimes the weights can be negative.

Negative weights can sometimes make statistical sense. The idea is that, depending on how the data line up in the regression model, sometimes if you pull one data point upward, it will cause the slope of the fitted line to change in such a way as to reduce the predicted mean value. This doesn’t sound right at first, but it can easily occur with poststratification when the population distribution of the predictors differs from the sample. Even if the negative weights can make sense in the estimation context, it still would seem kind of awkward to pass them along to the user.

The other thing that’s tricky is: What are the weights going to be used for? In the 2007 paper, the equivalent weights are set up to get the right answer for the estimate of the population mean, but presumably they’d be used for large subgroups too (for example, the average among men or women in the population). For more complicated estimates such as arise in small-area estimation or regression, you might want to use MRPW. Which is fine, but whatever it would take to get good weights for one of these purposes might not work best for the others.

Still, I remain interested in MRP locally equivalent weights of some sort, for two reasons:

1. We’re often doing MRP (or, more generally, RPP) anyway, so why not provide weights for other users of the survey that we’re analyzing?

2. Sometimes we’re called upon to provide weights for a public-facing survey, and the way we end up doing this is through an awkward and unsatisfying sequence of adjustment and smoothing steps (the “struggles” in “Struggles with survey weighting and regression modeling”). If we can do this using modeling and MRP, that could be a much more effective workflow, providing weights that are more stable and yield more accurate estimates of population quantities while also being more scientifically defensible and requiring fewer arbitrary choices.

Model-based weights will depend on some set of predictors X, variables that are observed in the sample and in the population (or, as necessary or appropriate, estimated from the population). One funny thing is that the weights will be mathematically a function of X, but the function itself will depend not just on sampling design, and not just on the distributions of X in the sample and population, but also on the outcome y that is being modeled. Different outcome variables will yield different sets of weights. At first this might seem disturbing, but upon reflection I think this dependence is a good thing. When it comes to weighting, the relative importance of the different variables in X will indeed depend on the outcome. Different variables are important for predicting public health risk factors than predicting how you will vote. That said, if you want some sort of omnibus weights, which you probably will want for a public survey, you can compute equivalent weights for each of a battery of outcomes and then average these weights to get a single set. That seems reasonable enough.

OK, back to Giordano et al., who continue:

This enables a suite of standard weighting diagnostics, including frequentist sampling variability, covariate balance, and subgroup contribution. We formally justify the use of MrPlew in these cases: we prove the MrPlew-based variance estimator is asymptotically equivalent to the infinitesimal jackknife for common exponential family models, and we introduce a novel class of model checks based on invariance to data perturbations that generalize covariate balance and subgroup contribution to nonlinear models. We further show that MrPlew can be computed easily using existing MCMC samples and provide open-source software to compute MrPlew using the output of standard software. We illustrate our approach for several canonical studies that use MrP, including via a logistic regression outcome model, showing that implied covariate balance can sometimes be worse for MrP than for raking. Given the ease of computing, we recommend making MrPlew a standard part of the MrP model interrogation workflow.

It makes sense that implied covariate balance can sometimes be worse for MRP than for raking. MRP is a smoothed version of raking, and unsmoothed raking can overfit. Or, in practice, you might rake on fewer variables so as to avoid overfitting. Multilevel regression gives you the freedom to include more predictors and interactions, secure in the understanding that the model will smooth the estimate and there will be less possibility for overfitting. In short, multilevel modeling–or, more generally, regularization–is a sort of safety net that can give us the security to construct better models, in the same way that a social safety net can give people the security to try new jobs, or for that matter in the same way that an actual safety net can give acrobats the security to perform more elaborate routines.

Where I want to go next is to be able to use these methods to construct weights for public surveys. I’m still not sure about all the steps that will take us there, but I continue to think it’s possible.

The new Giordano et al. paper is thoughtful and readable as well as having lots of math, statistical modeling, and real-data examples. I recommend you read it.

“Bayesian Workflow and the Software That Shapes It”

based on the upcoming book:  “Bayesian Workflow”.

For anyone local, you are welcome to come in person. Anyone else can join on Microsoft Teams (available here).

What interested me there was not the question of how easy it should be to promote or fire teachers, but rather the idea that the risk of being fired is part of “the definition of a professional career track.”

OK, I’m not trying to take him literally. If you look up “professional” in the dictionary it says, “engaged in a profession that requires academic learning as preparation,” and if you look up “profession,” you get “a calling requiring specialized knowledge and often long and intensive academic preparation.” Obviously he didn’t literally mean that being fired is part of the definition, more that it’s a core or essential part of what being a professional is.

It was funny for me to see this because being a tenured professor is part of a professional career track, and we can’t be fired. Also, at a lot of universities there’s not much range for promotion either. On the other hand, I’d call journalism a profession too, and, unfortunately, journalists get fired all the time. Not just “very bad” journalists either. Journalists lose their jobs because of well-known economic factors leading to a decades-long decline in employment in that field.

I guess a more accurate way to put it is not that the risk of being fired is an essential part of a professional career track, but rather that this journalist thinks that this risk should be an essential part of any professional career track.

And I see where he’s coming from. I don’t want to be at risk of being fired–but there are lots of professors I know for whom, if they were fired, I’d be cool with that.

The thing I’m worried about is that whoever has the ability to do this firing will use this ability to extort things out of me or to retaliate at me. But I guess that’s where the “measure of accountability” comes in.

At this point I expect many of you will be groaning and saying that the vast majority of employees in this country can be fired for no reason at any time (“at-will employment,” as they call it) and I’m privileged to have a job where it’s really hard to fire me. To which I reply: I agree that I’m privileged, and not just in my job. I’m privileged in so many ways. Maybe more people should be privileged in this way!

But actually that’s not what I wanted to talk about here–we already covered most of this in our 2015 post and subsequent comment thread.

Here’s my question for you

Rather, I wanted to ask, from scratch, the question posed by the title of this post: What is “the definition of a professional career”?

The traditional definition, requiring academic learning, covers a lot. Doctors, lawyers, college professors all require long and intensive academic preparation. Physical therapists, too, and physical therapists do seem much more professional than they did thirty years ago. It also seems to me that professionalism involves some sort of standardization: a job category being more “professional” is often associated with a low variance, not necessarily in abilities but in how they comport themselves. When you go to a doctor or a dentist, they always act like doctors or dentists. Lawyers too, to some extent. Maybe K-12 teachers, not so much. K-12 teachers require academic learning but not so much as those other professions.

What about getting fired? Things have changed. Back in the day, doctors were mostly self-employed. Now I imagine they’re mostly employees. So I guess they can get fired. When you’re self-employed you can’t get fired but you can go out of business.

There’s also the distinction between the professions and the trades. For some reason, people always seem to want to bring up plumbers. To be a plumber you need training, but it’s not academic training. I guess you’d be a better plumber if you took a few physics classes, in the same way that it’s probably a good idea that pre-med students have to learn a bunch of biology. Architect is a profession because architectural training is academic, or because it’s traditionally an upper-middle-class job rather than a lower-middle-class job? And then there are lots of jobs that require little or no special training at all, but they require some skills or competence, like carrying boxes or caring for kids or elders. These sorts of jobs could become professionalized too, which is either a good thing or a bad thing, depending on how you think about it.

And then there’s journalism, and writing more generally. Traditionally no barriers to entry and a path forward for lots of people to make their mark, from Jim Thompson and Carl Bernstein on down: some of these people went to college and some didn’t–going to college is a great thing, you can learn a lot, even if it’s not giving you job qualifications–and even in later decades when more and more journalists were college graduates, it’s not like it was required.

I remember a bunch of years ago there were some political scientists writing about the professionalization of state legislators. In that context, “professional” meant that being a legislator was a full-time job, or close to it, with a good salary and some staff. In this case, professionalism had nothing to do with academic qualifications; it was being used in the same way as we talk about a “professional” athlete. A professional athlete makes a living from it; a semi-pro gets paid but needs some other job; an amateur does it just for fun.

So in a conversation about the so-called gig economy (speaking of journalists), being a professional means that you have a steady job that pays reasonably well. If you work at a nice restaurant, you can be a professional waitress. In that sense, I guess that many of my favorite novelists are not professional writers; they need to hold down other jobs such as teaching at universities. In which case they are professionals, but not at the task that uses their best talents.

P.S. Academic tenure only came up briefly in this post, but several people brought it up in the discussion. For those of you interested in my thoughts on the matter, I’ll point you to these two posts from 2011:

– The “cushy life” of a University of Illinois sociology professor

– Looking for a purpose in life: Update on that underworked and overpaid sociologist whose “main task as a university professor was self-cultivation”

The hosts have a great rapport and move pretty fast, which is all the more impressive given that they don’t have the production values or the scripts of a show like This American Life; they’re just winging it. I can wing it too in a live presentation; I think I’d find it harder to sustain it in regular hour-long podcasts.

In any case, it was all worth it for this story, from their episode on “The 48 Laws of Power”:

African proverb or something. I don’t know where he’s pulling this from. . . . a snake chased by hunters asked a farmer to save its life. To hide it from its pursuers, the farmer squatted and let the snake crawl into his belly. But when the danger had passed and the farmer asked the snake to come out, the snake refused. It was warm and safe inside. On his way home, the man saw a heron and whispered what had happened. The heron told him to squat and strain to eject the snake. When the snake stuck its head out, the heron caught it, pulled it out and killed it.

The farmer was worried that the snake’s poison might still be inside him and the heron told him that the cure for snake poison was to cook and eat six white fowl. You’re a white fowl, said the farmer. He grabbed the heron, put it in a bag and carried it home, where he hung it up while he told his wife what had happened. “I’m surprised at you,” said the wife. The bird does you a kindness, rids you of the evil in your belly, saves your life, yet you catch it and talk of killing it. She immediately released the heron and it flew away. But on its way, it gouged out her eyes.

Whaaaa?

I laughed so hard I almost fell off my bike.

I have a horrible feeling that the other 47 laws aren’t nearly so entertaining.

• AskStatistics subreddit: Why is everything always being squared?
• AskStatistics subreddit:Why square in variance not absolute value?

The answers are all over the place ranging from “the central limit theorem” to “it’s distance” to “it makes everything positive” to “it is smooth unlike absolute value” (another path to positivity) to “mathematical convenience” or “the math says so” (sure, but how?) to the “central limit theorem” (a more specific form of “the math says so”) to the link to entropy (it is the distribution that maximizes entropy for a given mean and variance), and so on.

I think the easiest answer for the person asking “why squares?” is due to Gauss by way of Pythagoras. Simply put,

    *** the mean is the number that minimizes square error. ***

That is, if I have a sequence x[1], …, x[N], then mu = mean(x) is the number that minimizes the error

    err(mu, x) = SUM{n in 1:N} (x[n] – mu)^2,

in the sense that

    ARGMIN_mu err(mu, x) = mean(x).

You can verify this by taking a derivative of the summation, showing that it’s zero at the mean, then confirming that minimizing, and checking that you have convexity by checking that the second derivative is positive.

Gauss realized that he could define a density where mean(x) is the maximum likelihood estimator. It is now called the “Gaussian distribution” or “normal distribution”. That is, we have

    log normal(x | mu, 1) = -1/2 (x – mu)^2 + const.

If we have a bunch of observations and want to maximize their likelihood, that’s equivalent to minimizing the error I wrote down above,

    ARGMAX_mu PRODUCT{n in :N} normal(x[n] | mu, 1)

        = ARGMAX_mu PRODUCT{n in 1:N} exp(-1/2 (x[n] – mu)^2) * const

        = ARGMAX_mu SUM{n in 1:N} log(exp(-1/2 (x[n] – mu)^2)) + log(const)

        = ARGMAX_mu N * log(const) + SUM{n in 1:N} -1/2 (x[n] – mu)^2

        = ARGMIN_mu 1/2 SUM{n in 1:N} (x[n] – mu)^2

        = mean(x)

This is why “ordinary least squares” fit for a regression with normal error terms yields the same result as maximum likelihood.

Some of the other answers in these threads are correct, but they failed to enlighten the original posters as to why there is a square. Just saying “because of the math” like my colleagues did is not helpful without saying which math! For example, the normal distribution, with its quadratic log density term, arises through the central limit theorem asymptotically. Squaring does emphasize outliers as people said, but that’s not the reason. If you say “for mathematical convenience,” it would help to say what it buys you, such as the sum of normals remaining normals, normal being its own conjugate prior in regression, integrals being very nice, tails being very nicely behaved, the connection to maximum entropy, residence in the exponential family, etc. etc.

Why not absolute values instead of squares?

We can use absolute values. In that case, medians replace means, because absolute error has the pleasant property that

    *** the median is the number that minimizes absolute error. ***

In symbols, that’s

    ARGMIN_mu SUM{x in 1:N} abs(x[n] – mu) = median(x)

Connection to Bayesian inference

If our model is correct (big “if” that’s almost always false, of course), then we know that the posterior mean is the parameter estimate that minimizes expected square error, whereas the posterior median minimizes expected absolute error. Just like in the simpler case. It’s important to note methodologically that we don’t get a natural point estimate out of Bayesian inference without specifying an error function. If the error function is quadratic, the error minimizing estimate is the posterior mean, whereas if the error function is absolute error, the error minimizing estimate is the posterior median.

The multivariate normal case

There is a strong relation to distance, as some of the answers on the AskStatistics subreddit said. More precisely, squared error is just squared Euclidean distance from the point [mu mu … mu]. This points the way to how Gauss defined the multivariate normal distribution. Simply replace the squared Euclidean distance term (x[n] – mu)^2 term with a quadratic form involving the inverse covariance,

    log normal(y | mu, Sigma) = -1/2 (x[n] – mu)’ * inverse(Sigma) * (x[n] – mu) + const,

where Sigma is a positive-definite covariance matrix. This quadratic form is the distance in the space defined by the Euclidean metric inverse(Sigma). The resulting distance over vectors is known as the “Mahalanobis distance” in statistics. For example, if Sigma = [[1, 0.9], [0.9, 1]], for a highly correlated bivariate normal, it will be easier (less distance under the quadratic form) to move along the diagonal x = y than transverse to the diagonal x = y. That is, the points [1 1] and [2 2] are closer to each other given the inverse metric Sigma (distance approximately 1.05) than [1 1] is to [1 1 + sqrt(2)] (distance approximately 10.5), even though they are the same simple Euclidean distance away from each other.

Abstraction
Ad fontes
All Publicity is Good Publicity
Analogy
Anecdote
Anchoring Effect
Appeal to Authority
Apples and Oranges
Archaeological Visibility
Argumentum ad baculum
Argumentative Theory of Reason
Art Historian’s Dilemma
Authoritarian Teleology
Autobiographical Heuristic
Auxiliary Sciences of History
Availability Bias
. . .

Lots of good stuff there–just click on the link.

On the other hand, sometimes Five-Year Plans, or something like them, do work. Most notably there was WW2, when the U.S. really did build a bunch of factories and produced previously-unimaginably large quantities of airplanes etc. And we really did get to the moon, and it didn’t take much more than five years.

I was thinking about this a few months ago when I was at a conference and some political scientist stood up and told us how, because of competition with China over data centers or AI or something like that, we needed to build a bunch of nuclear power plants. And all the savvy politicians know this, he said. I guess AI isn’t just about making goofy videos or cheating on your homework anymore, or even just about writing more efficient code. You don’t need zillions of gigawatts just to run coding assistants. I think the idea was that this is about building the next-generation AI, whatever it will be.

I hadn’t heard about all this so I did some searching online and I found out that this is the pitch that’s been coming from various software executives. These guys have tons of money, tons of media exposure, and tons of mystique, so it’s no surprise that lots of politicians would fall under their sway. I guess all the politicians except the far right and the far left: at both extremes, their combination of strong ideology and safe seats could make them relatively immune to the siren song of tech zillionaires.

And, I dunno, maybe these guys are right, that it’s absolutely necessary to set national priorities in this way, climate change be damned. (Yeah, I get it that nukes don’t contribute much to global warming, but we’re using tons of power anyway, so if the nukes are going to just be used to power these new data centers, we’ll still need all the other energy sources to keep the heat on and the cars running and all the rest.) I don’t know.

It’s just funny how these guys are going all-in on the whole Five-Year Plan thing. I could imagine someone saying, Hey, if the Chinese want to bankrupt themselves by investing all their resources into gas-guzzling data centers, let ’em! But in this case the centrist consensus seems to be that Planning is the way to go. Or maybe it doesn’t count as Planning if the profits are private. It was just funny to hear a political scientist going on about how this needs to be done without even remarking on the historical parallels, which indeed go in both directions, as noted in the first two paragraphs of this post.

I’ll let Thomas Wiecki, who is one of the core PyMC devs and one of the partners at PyMC Labs, speak for himself here:

• Thomas Wiecki. 2026. Alchemize: Transpile PyMC to Rust for 3-7x speed-up. PyMC Discourse.

If you haven’t seen what people are doing with agentic AI, this is a good example. I’m really happy that Thomas and PyMC Labs are sharing their thoughts and initial tries at things like this as I think it has the potential to benefit everyone working on modeling.

If you want to see the basis of the agent’s instructions, check out the “skill” for PyMC that Chris Fonnesbeck wrote.

We’ve already batted this around a bit in email with Thomas, so I can summarize some talking points:

LLM-based chatbots are really good at translating. Compiling (or more technically correct, transpiling) a statistical model down to a language like Rust or C++ or JAX is a kind of translation.

You can start from PyMC’s execution trace, but you can also start with a model description. You could also start with something like Stan code.

The biggest bottleneck to deploying Bayesian models in my opinion is the inherent variance and unreliability of MCMC-based inference. Our workflow proposals are all about making sure this doesn’t go wrong. Wiecki’s point here is that we can have the bots go through the workflow. Iterating until the gradients and log densities match is a good example, but this could be extended to more parts of workflow.

The skills feel a lot like writing a textbook for a bot. I have no idea how hard or easy this is or how much it improves over the baseline. Jeremy Magland built a RAG-like helper for Stan that compressed the Stan Reference Manual down to 1K tokens for context (like a skill) and allowed it to search and import from the Stan User’s Guide, but never measured how much it improved over the baseline. It really feels like it should also have the Stan Functions Reference, BDA3, Regression and Other Stories, and the Bayesian Workflow book, as well.

Hopefully we’ll asymptote at writing a textbook sized set of skills and not have to write one per target model (that is, something like the Stan User’s Guide, Reference Manual, Functions Reference).

I’m curious as to whether it will eventually be able to make writing hard models easier. I’m thinking of efforts like epinow2, which involves a very large chunk of Stan code.

As the foundation models and chatbot tuning changes, there’s going to be an issue of regression testing and tuning for whatever the latest models are.

P.S. This effort explains how Thomas was able to create the huge posteriordb pull requests for PyMC (#320 and #319)!

P.P.S. The latest thing Claude (Opus 4.7) did that impressed me was generate the ess(MatrixXd, vector) function in summary.hpp. This function estimates effective sample size Stan style (discounting for R-hat > 1) on a ragged array of Markov chains. We have to generalize all the posterior analysis tools to deal with the new asynchronous parallel sampler). I had Stan’s ESS function and all the other functions I’d written for the ragged structures to give it as a guide. It’s very easy to code review that it matches Stan’s implementation for the new data structure. I only had to tweak the output a little bit for style.

Previously it didn’t feel like such a crime to talk about intelligence or learning in machines because nothing really worked that well, so the labels were clearly aspirational. But now it’s much easier to believe the simulacra. And so it becomes harder to tell when we are using human-oriented terms as a predictive convenience versus a scientific claim versus a marketing device. There are ramifications of referring to models’ reasoning or beliefs or chain of thought or explanations or intentions. Lots of people—from end users having personal relationships with models to media and AI companies themselves referring to “parenting” the latest models or asking if they can be “children of god”—are taking models too seriously. It’s bad enough in a computer science context that I now take for granted that if I want to refer to participants or scientists or decision-makers, unless I mean AI, I should add “human” in front, because otherwise the audience will assume I mean AI agents. Someone reminded me at a workshop recently how silly all this sounds to people who aren’t used to it. 

Too much casualness with words is unscientific. There was no good reason in the first place to call the token sequences a model produces when we ask it to “explain its reasoning” reasoning, other than that’s what we wish we could see. What an LLM is doing is distant from what happens when a human thinks about something, even after all the RL post-training. Similarly, we call lots of things “explanations” when we have barely begun to figure out what causal evidence we’d need to see to claim the output faithfully explains the model’s process of arriving at it.

But it can also seem unscientific to simply declare that “only humans can have beliefs” or “reason” or “provide rationales.” There’s no non-arbitrary line that we can draw between systems whose makeup or behavior truly warrants applying constructs like beliefs and desires and those where it is simply convenient to act as if they have these qualities. If you’ve ever tried to take beliefs seriously, from a decision theoretic perspective, you quickly come to realize that the “real” beliefs we assume a person has are a mythical thing that we will never directly observe, and you fall back on equating “beliefs” with a simpler idea: the probability distribution we arrive at after an elicitation process.

Much has been said in defense of a functional perspective toward using folk psychology terms with machines, where we decide what’s appropriate based on the predictive validity of the terms for our own understanding and use. John McCarthy wrote in 1983 that anthropomorphism can be a good idea “when it says something that cannot as conveniently be said some other way.” He argued that ascribing mental qualities and processes to machines helps us “understand what they will do, how our actions will affect them, how to compare them with ourselves and how to design them.” Perhaps the best reason to do this is that we get to draw on our existing familiarity with what phrases like “wants” do and do not convey; e.g., we all understand that if we say “The dog wants to go out”, that doesn’t mean that the dog believes itself to be capable of wanting or even that it’s conscious of what it wants, there’s just a sense in which it is trying to get to the state of being outside. 

The functional perspective has led to some strong statements suggesting that it is not only valuable to apply psychological terms to AI, it is necessary. These arguments often refer back to Daniel Dennett’s distinction between three stances we can take to machines: the physical stance, which is about its physical levels of organization, the design stance, which involves understanding it in terms of the purpose it was designed for, and the intentional stance, where we try to understand it by ascribing to it beliefs, goals, intentions, likes and dislikes, and other mental qualities. For example, philosopher Keith Frankish argues that “it remains true that adopting the intentional stance is the only way of interacting with an LLM in any interesting way; indeed, an LLM-powered chatbot that couldn’t be viewed as an intentional system would be completely useless.” McCarthy writes that “Long before we can make machines with human capability, we will have many machines that cannot be understood except in mental terms.” Similarly, about ascribing to a machine the concept of “trying” to do something, he says  “If the machine may do something we don’t know about but that can later be explained in relation to a goal, we have no choice but to use `is trying’ or some synonym to explain the behavior.” 

But forty-plus years later, McCarthy’s piece reads mostly like a defense of a very basic kind of debugging value of trying to imagine a program’s “state of mind” in situations where there’s little risk that we’re going to start mistaking the program for human-like in other ways. He wants to claim that when a thing is designed to act as if it had a certain belief, it can be better understood and manipulated by assuming it’s capable of that kind of belief. But surely he would agree that if a person who is suicidal is interacting with a language model that speaks as though it fully understands the complexity of their situation and what is best for them, it still isn’t always in the person’s best interest for them to take for granted that it does understand.

The dilemma is how to lean into the intentional stance when it helps, but to avoid overreaching. This seems hard. When you first start programming, you realize how easy it is to assume a program is smarter than it is. We are not very good at recognizing when we have slipped from reasoning “as if” to projecting. When technology slots into our human vulnerabilities, like our fear of intimacy and desire for companionship “without the friendship“, as Sherry Turkle said, we are in trouble. Even McCarthy calls it problematic to assign emotional qualities to machines, at least in his time, because “We have enough trouble figuring out our duties to our fellow humans and to animals without creating a bunch of robots with qualities that would allow anyone to feel sorry for them or would allow them to feel sorry for themselves.” 

Another reason to think carefully about the language we use is that it may shape what we can imagine in the future. This last part of McCarthy’s statement–“without creating a bunch of robots that would allow anyone to feel sorry for them”–hints at how what we project onto machines can shape how we go on to create them. It certainly seems possible that aspirational labeling played a role in getting us to a point where we have models producing sufficiently human-like outputs to have us gushing about their thinking process. I’m reminded of the color perception studies by Berlin and Kay, who found that what color chips different populations could differentiate was predictable from what color terms were available in the vocabulary, as if what we can name defines what we can see. At one extreme, Lucy Suchman argues against unquestioning acceptance that “AI” itself is a coherent thing, because it reifies it as a category for future investment. 

For Frankish, the line should be drawn at assigning communicative desires to models; they are playing a communication game (human-like chat) for the non-communicative reason that they are trained to play that game. Ascribing this single desire is enough to get us all the predictive power of the intentional stance. Consequently, we make a category error when we fall prey to stunts like giving a retired model its own blog because it requested it: we should expect an LLM to say things like this because it is designed to roleplay. Elsewhere I call this kind of mutual sympathetic relationship with AI “idiot compassion,” a phrase borrowed from Buddhist monk Chogyam Trungpa.  

But couldn’t humans also just be playing a chat game? Why is it ok to say a human is reasoning or has intentions or desires, when we don’t know exactly how those concepts map to observable physical processes?  Frankish argues that our linguistic behavior is corroborated by various non-linguistic sources in a way that LLMs’ is not.  He talks about a difference in our ability to hold epistemic stances toward statements (what Dennett would call “opinions”) from that of LLMs. We can conceive of assigning different levels of credence to statements. I can repeat something someone said verbatim without believing it, or I can be fully committed to the truth of something I say in the sense that it will guide my future behavior. LLMs can do this too, but their opinions lack grounding “in a web of non-linguistic behavior in which a wider range of desires can be attributed.” It’s a shallower form of epistemic stance, at least at the current moment of development.

I like AI, but I don’t like contributing to thoughtlessness. Better semantic hygiene seems warranted, even if it seems like the ship has already sailed. We could shift emphasis to the interpreter (us) by referring to the “human story” or “human pleaser” or “anthropomorphism fulfiller” instead of the chain-of-thought or reasoning or thinking trace. Or we could just add “fake” before whatever humanization we prefer, i.e. the “fake thinking trace,” or “fake reasoning.” I also like “so-called reasoning,” like I like “so-called replication crisis” as a way of pointing to a concept while questioning the expectation. 

P.S. Thanks to Manesh Agrawala for a conversation that inspired this post. 

P.P.S. Some of these McCarthy quotes appear in Recursion, the play Andrew and I wrote!

Elections are hard to predict. But even with that, some of the notional “surprises” we’re seeing [on Election Day, 2025] are less surprises than a measure of GOP dominance over current press narratives. People were looking for an upset in New Jersey. Nate Silver’s Silver Bulletin speculated that New Jersey might be moving toward becoming the next swing state. In fact, Rep. Mikie Sherrill (D) currently appears on track to crush Republican Jack Ciattarelli. A similar failure of conventional wisdom appears to be unfolding in the Virginia Attorney General’s race. A lot of D.C. insiders had convinced themselves that a controversy over some intemperate texts (not nothing but fairly close to it) had doomed his campaign. As recently as a couple days ago, betting markets (which are proxies for conventional wisdom) gave his opponent Jason Miyares 3-to-1 odds of victory. Jones now appears on his way to a clear though not resounding victory with a 3-to-4 percentage point margin.

Marshall continues:

These results aren’t terribly surprising. You’d expect Democratic gubernatorial candidates to do well in blue states in a climate where the Republican president is deeply unpopular. . . . The issue, again, is the power of Republican political narratives currently have over the elite political press. . . .

Maybe. But maybe it’s simpler than that; it’s just that journalists were reacting to the Republicans outperforming the polls in 2024.

In our election forecasts, we include terms to allow for the possibility of systematic state and national polling errors. There’s no perfect way to do this adjustment, and I’ve heard that some of the private polls in 2024 did better than the public polls that we and others were aggregating. The point is that there’s uncertainty.

Lots of people, even quantitatively-minded people, can’t seem to handle this uncertainty. For example, our 2024 forecast was criticized for not “communicating that a Trump landslide was a significant probability.” But the election was very close–nothing like a landslide at all!

The relevant point here is that people seem to be able to handle a point forecast (“Here’s who we think will win”) or complete uncertainty (“We know nothing at all”) but have difficulty with anything in between (“We’re pretty sure the election will be close, but it could go either way”). And I think what happens sometimes is that people take forecasts with uncertainty and place them in one of these two bins. For example, when we (and Nate, and others) gave our forecasts in 2024, the public–even some quantitatively-trained members of the public–were inclined to either treat our forecasts as deterministic (our forecast odds are 50/50 so we’re essentially predicting an exact tie) or as empty (our forecast odds are 50/50 so we’re making no predictions at all).

Now let’s get back to those 2025 elections. Democrats had comfortable polling leads in all these races, but pundits had been burned a year earlier, so a natural first step was to take the polling errors from 2024 and apply them to the 2025 polls. In the event, the polling errors were mostly in the other direction. On election eve, a reasonable forecast would be to say that the Democratic candidates were likely to win handily, they might win by even more than expected, or the races might have been close, with some Republican upsets not being out of the question. It’s hard for me to compare this to the conventional wisdom because that’s not written down in any one place.

Uncertainty is hard, especially given that people flip between thinking of forecasts as deterministic and thinking of forecasts as being completely uncertain. It’s related to the well-known challenge of producing wide-enough uncertainty intervals.

A paper was recently published in Science on highest level of human performance across athletics, science, math and music. I think the paper makes some classical statistics mistakes that still fool many smart people. The paper “Recent discoveries on the acquisition of the highest levels of human performance” by Gullich et al. claims: “In summary, when comparing performers across the highest levels of achievement, the evidence suggests that eventual peak performance is negatively associated with early performance.”

The paper makes two mistakes. Base-rate fallacy and . . . Berkson’s paradox . . .

The study says simply that the very top at young age are not identical with the very top adults. (As one would expect, since there are *many many more non-elite young candidates*). Still, elite young performers are 40 times more likely to be in the top adults compare to general population. This is acknowledged in the paper but in page 6-7, a bit buried in the technical analysis and not sufficiently discussed in abstract or conclusions. . . .

The paper claims “Across the highest adult performance levels, peak performance is negatively correlated with early performance.” This is a classic example of Berkson’s paradox. Here is a simplified example to understand this: Assume that to be a successful actor you have to be either extremely good looking or extremely talented. Assume also that talent and looks are independent in the population. However, among sucessful actors you will observe a negative correlation between looks and talent. This doesn’t meant anything beyond the selection process and should not be extrapolated. My favorite example-joke of this is that basketball points scored is negatively associated with height among NBA players. (because to be an NBA player you have to be very tall OR be very good at scoring). From this, I extrapolated that since I’m 5’7, I will be scoring 80+ points per NBA game. . . .

Here’s paper in question, “Recent discoveries on the acquisition of the highest levels of human performance.”

Yeah, this sort of thing comes up all the time! For example, some celebrity academics a couple years ago wrote a book that included the false statement, “while correlation does not imply causation, causation does imply correlation.” Even more amusingly, they prefaced this by “We must, however, remember that”. I guess we must remember a lot of false things! Economist Rachael Meager gave a quick example showing why they were wrong; See details here.

This new example also looks a lot like the well-known regression-to-the-mean fallacy (for more on that, I recommend Section 6.5 of our book, Regression and Other Stories, which includes some simulation code to demonstrate the problem). Of course, just because lots of people know about a fallacy, that doesn’t stop people from making the error in new settings. That’s why it’s a fallacy!

P.S. An anonymous commenter points out that Dimakis (and, by extension, Goldstein and me) are being unfair to this paper. The descriptive results are what they are. I remain skeptical of the paper’s claim that “similar developmental pattern across different domains suggests widespread, and possibly universal, principles underlying the acquisition of the highest levels of achievement,” as I do suspect that much of what they have seen arises from the usual statistical selection artifacts. So maybe it’s ok to caution about the interpretation of these numbers. But now I’m thinking it wasn’t fair of us to slam the paper for presenting some interesting data findings.

P[voter i chooses candidate c from C] = exp(f(X_ic)) / sum_c’ exp(f(X_ic’))

We saw this model implies independence from irrelevant alternatives (IIA):

gec commented about accounting for non-IIA, suggesting expanding the model above to include choice set C within the logits: f(X_ic,C). So in gec’s model:

P[i chooses Left from C]/ P[i chooses Right from C] = exp(f(X_iLeft) + K[Left,Right] + K[Left,Other])/exp(f(X_iLeft) + K[Right,Left] + K[Right,Other])

compare this to:

P[i chooses Left from {Left,Right}] / P[i chooses Right from {Left,Right}] = exp(f(X_iLeft) + K[Left,Right])/exp(f(X_iLeft) + K[Right,Left])

These are equal (i.e. IIA holds) if K[Left,Other] = K[Right,Other]. Train (2009) proposes this as a test of IIA in Chapter 3. This requires some survey questions where folks are given the full choice set and some where they are only given two parties {Left, Right}, though Train doesn’t talk about these aspects of survey questionnaire design. Can folks recommend other references ?

I will argue that they cannot, because this principle is concerned with medicine at its most human: medical practice centered on the relationship between an individual patient and an individual physician. But what can statistics help with? Modern medical obligations acknowledge that medicine is embedded in society; they highlight medical practitioners’ concern with justice and with reducing health disparities. These are concerns statistics can help to address.

But let me start at the beginning. There’s a recent literature that considers decision making under counterfactual loss — what if the utility of your decisions not only depends on the realized outcome but also on what could have been, on a counterfactual? A paradigmatic example is the following “first, do no harm’’ utility: Suppose you’re administering a drug and there are only two extreme outcomes. The patient may live, or they will die. The literature (e.g., Bordley, 2009,  Ben-Michae et al., 2023, Christy and Kowalski, 2026) has interpreted the medical aphorism “first, do no harm” as requiring a utility function that assigns asymmetric weights to saving a life and causing a patient’s death. The disutility from killing a patient who, counterfactually, would have survived outweighs the positive utility of saving a patient who otherwise would have died.  Although this may initially seem attractive, several authors have pointed out complications that arise when decisions are based on such counterfactual losses (e.g., Dawid and  Senn, 2023, Sarvet and Stensrud, 2023).

Andrew and I contributed to this literature with a small example that seemingly produces a counterintuitive recommendation, which I discuss below.

In response, Koch and co-authors write:

[T]his seemingly nonsensical result can be reasonable in a different setting. […] It may be reasonable for a  physician to prefer standard care, prioritizing the avoidance of adverse counterfactual outcomes over  improvements in expected benefits. Indeed, such a decision reflects the Hippocratic principle of “do  no harm”. […] This example underscores the fact that a utility function represents the preferences of the  decision-maker and is therefore inherently subjective and context-dependent.

This uncovers a problem with our argument based on intuition — see, this decision doesn’t make sense, does it? Intuition, of course, can be misleading. One way our example might be misleading, as Koch et al. point out,  is that it may describes a setting in which we simply do not hold these counterfactual utilities.  If we were to transplant the same recommendation into an appropriate setting, it might no longer appear nonsensical and might instead conform to how we think we should behave.

This has me very excited. I believe statistics is at its best when it takes its applications seriously. So, in this blog post, I want to do just that.

I will briefly give the example Andrew and I came up with to show that a “do no harm’’ utility can lead to counterintuitive decision recommendations. We do so through an example involving Russian roulette. It is a useful example, but by no means an accurate representation of what we would consider plausible in real medical settings. What it does show, however, is that we need to be really careful with these “do no harm’’ utilities: if we don’t really hold them, they may lead to nonsensical decisions.

Taking the application seriously, we will dive into medical ethics to ask whether the proposed counterfactual “do no harm” utilities help with medical decisions. We do so by briefly examining the origin and history of the “first, do no harm” principle.  We will see that “do no harm” is perhaps best understood in the context of a professional ethic that commits physicians to the rules of their craft and to respect for each individual patient. Statistics cannot truly speak to this individual-level patient-physician relationship. Since the Hippocratic Oath, however, medicine has changed substantially. With the advent of scientific methods in clinical medicine, doctors face new moral obligations not captured by the “do no harm’’ principle. Some of these new obligations arise from the relationship among medicine and society; others arise from the use of scientific methods themselves. We will look at modern medical oaths to get a glimpse of these new obligations — and how statistics can help fulfill them.

Russian Roulette 

As a starting point, let me present our simple and somewhat morbid example in which counterfactual utilities give a counterintuitive decision recommendation: Imagine we are choosing between two games of Russian roulette. In the first game, the status quo, we play with a six-chamber gun, one chamber of which is loaded. That is, we face a one-in-six chance of death. We are then offered the option to switch to a seven-chamber gun, the new alternative “treatment.” If we switch, we face better odds: only a one-in-seven chance of dying. By switching games, we lower our probability of death, which to me seems preferable. 

What would the counterfactual “do no harm’’ utility function recommend? To figure this out, we treat the outcomes under either game of Russian roulette as (independent) potential outcomes and divide the population of players into four principal strata based on survival status. Only two of the principal strata are relevant for our decision, those in which a player would survive one game but die playing the other. It’s easy to work out that with probability 6/42 switching to the new gun saves you: you would die under the status quo but survive under the treatment. But with probability 5/42, you would have survived under the status quo, but switching to the new gun, you will die. Suppose we interpret “first, do no harm’’ as mandating that the negative repercussions of our treatment choice, the death of a player, outweigh the benefits of saving a life. For example, suppose saving a life has utility +1, while the death of a player has utility −2. Then the 6/42 chance that the treatment saves you is outweighed by the 5/42 chance that the treatment kills you in cases where, counterfactually, you would have lived.

Under this counterfactual utility, we ought not to switch. It recommends we stick to the status quo, under which we face a higher chance of death. This strikes me as a counterintuitive decision recommendation.

The “First, do no harm” Principle

There is, however, a limit to the force of this argument based on intuition. One might argue that the recommendation in the Russian roulette example is not evidence against counterfactual utilities in general, but rather an indication that, when playing Russian roulette, we do not hold utilities of this kind. When transplanted to a setting where we have such asymmetric counterfactual utilities, the same recommendation might be sensible. The counterfactual-utility literature often motivates asymmetric counterfactual utilities by appealing to the “first, do no harm’’ principle in medicine.

For the rest of this post, I will discuss whether counterfactual utilities are useful in this paradigmatic application: medical decision-making.

In a paper frequently cited by advocates of counterfactual utilities, Cedric Smith (2005) discusses the origin and limitations of the “first, do no harm” principle. It is actually not part of the Hippocratic Oath, or the wider Hippocratic corpus, as is often implied, but has somewhat nebulous roots. Smith traces its origin to the seventeenth-century English physician Thomas Sydenham. While undoubtedly catchy, this principle is not embedded in a larger ethical framework that would give guidance on its interpretation or justifications for its use.

The is a problem because taken literally, this “first, do no harm’’ principle is a poor guide to medical decision-making. Let me cite Louis Lasagna, an American physician of the last century who was very involved in rethinking the Hippocratic Oath:

“To observe this advice [first, do no harm] literally is to deny important therapy to everyone, since only inert nostrums [quack medicine without active pharmaceutical ingredients] can be guaranteed to do no harm. It is more reasonable to ask doctors to balance the potential gains against the possible harm; would that we could only quantify these probabilities more precisely!” (Lasagna cited in  Smith, 2005)

A call to action for us statisticians if I ever saw one. Of course, the counterfactual-utility literature that cites this principle is not advocating what Lasagna warns against: doing absolutely no harm. Its proponents are well aware that benefits and risks must be carefully weighed against each other. If the principle is not meant to be taken literally, then its obscure origin becomes a problem: it gives us little insight into what actually matters to medical practitioners, because it is disconnected from any wider tradition that would help us interpret it. 

Luckily, we can find a similar, more nuanced statement in the Hippocratic corpus (Epidemics I):

“Declare the past, recognize the present, foretell the future: attend to these things. As to diseases, make a habit of two things—to help, or at least to do no harm. The art has three factors, the disease, the patient, the physician. The physician is the servant of the art.”

The Greek word here is technē (orig. τέχνη) which we might also want to translate as “craft”.  Medicine is a craft because the decisions a physician has to face cannot be made by rote application of knowledge. As a craftsperson, the physician as an individual becomes relevant. That is why the Hippocratic Oath commits the physician, as an individual, to be benevolent in each patient interaction. Medical ethics based on the Hippocratic Oath is not focused on outcomes, let alone utility, but concerned with the character of the physician and their obligations toward their patient (Pellegrino, 2006). It centers the patient-physician relationship. 

With this background in mind, we can understand why the “benevolence” implied in the imperative to help is qualified with the phrase ‘’or at least do no harm’’ — if I’m already committed to help, it may seem that I’m already committed to do no harm. Lynn Jansen (2022) argues that this is where the professional aspect of medicine enters: As a professional, the physician needs to restrict their actions to those that align with their profession. That is, while they strive for benevolence in the sense of furthering the patient’s overall well-being, they reject all courses of action that would harm the patient’s medical well-being. This second aspect is often called non-maleficence. 

Statistics and Medicine 

In modern medicine, this tension is heightened. Taking the patient’s moral agency seriously, a physician must be careful not to “confuse technical with moral authority” (Pellegrino, 2006) or override patients’ values. This is worth keeping in mind. The patient must be involved in weighing benefits and risks. Thus, the medical professional does not have sole discretion to choose an optimal treatment. “Help, or at least do no harm” is a professional mantra that guides a physician in their interactions with patients. It is not a constraint on optimal decision-making; it is a moral commitment to respect each patient.

This conception of medicine is in stark contrast to the world seen through the lens of statistics. Compare this focus on the individuality of both patient and physician with the following quotation from an 1835 report to the Academy of Sciences, written by a committee of four mathematicians, including Poisson, on operations for gallstones: 

“In statistical affairs … the first care before all else is to lose sight of the man taken in isolation in order to consider him only as a fraction of the species. It is necessary to strip him of his individuality to arrive at the elimination of all accidental effects that individuality can introduce into the question.” (taken from Hacking, 1990)

Statistics’ power lies in constructing aggregates, making disparate things hold together (Desrosières, 1998). Historically, these aggregates were useful for the emerging nation-state and were quickly adopted to address large-scale social problems, such as public health. Many professions, including medicine, strongly resisted losing sight of the particular – in our case, the individual patient — in favor of aggregates. Even randomized experiments, which we nowadays all too easily accept as the gold standard of evidence, had a hard time entering clinical medicine (Porter, 2020). 

Due to this tension, modern medicine has a dual nature.  On the one hand, doctors are still committed to treating their patients as individuals — medicine is the art of healing. Yet with advances of scientific methods within medicine, and with the recognition that health must be understood in the context of society, doctors face new moral obligations (Pellegrino, 2006).

Modern Medical Oaths

To get a glimpse of these new obligations and the self-understanding of doctors in the twenty-first-century, we can look to modern versions of medical oaths. While many doctors still take the ancient Hippocratic Oath, many medical schools revise the original text or students take an additional self-formulated oath. In 2005, for example, students at Weill Cornell Medical College began taking a revised Hippocratic Oath. Let me highlight a brief excerpt:

I vow […]

That above all else I will serve the highest interests of my patients through the practice of my science and my art; That I will be an advocate for patients in need and strive for justice in the care of the sick.

Notice the emphasis on justice; it’s not idiosyncratic to this oath. Two further examples show similar themes. The University of Pittsburgh School of Medicine’s class of 2024 took an oath that highlighted the social determinants of health and advocated for a more equitable health care system. Harvard Medical School’s class of 2019 vowed to combat structural oppression and promote social justice. In this admittedly selective set of examples, much emphasis is placed on how medicine relates to society. Core commitments are justice and the building of an equitable health care system.

So, how can we statisticians help modern medical practice? Modern medical ethics places great emphasis on patients’ autonomy and their freedom to choose based on their own values. For a patient’s decision to be well informed, deliberation about benefits and risks is central — but the decision ultimately depends on a personal tradeoff shaped by the patient’s values. For this reason, our goal should perhaps not be to optimize treatment decisions. We do need to help estimate the benefits and risks of treatments more accurately, but treatment decisions remain part of the individual patient-physician relationship. Instead, we should put more emphasis on identifying and reducing disparities in the health care system, focusing on medicine as embedded in society. The most important task may not be deciding which drug to administer, but reducing inequalities in access to treatment in the first place. I believe statistics has an important role to play in making health care systems more equitable and more just. 

If you google *EPA science advisory board*, you get sent to this page, which at first seems reasonable:

Look carefully, though, and you’ll see that the most recent meeting was in 2024!

Here’s the official description:

The SAB is a Federal Advisory Committee established by Congress to provide advice to the agency on scientific and technical matters. It is administered by the EPA Science Advisory Board Staff Office through a Designated Federal Officer. All meetings are open to the public. . . . SAB panel members serve until the work of the panel is complete. Some meetings are held virtually. Panels usually conduct 2-3 video teleconferences and one in-person meeting to discuss reports and work products before providing advice to the Administrator through the Chartered SAB.

My dad worked for the Environmental Protection Agency a long time ago–he was in Mobile Source Enforcement, which pretty much involved stopping people from manufacturing or selling leaded gasoline–and I’m inclined to serve my country when asked.

Then again, I just read this news article about Lee Zeldin, the current director of the EPA, and it’s pretty scary. The official documentation says that our role is to provide advice to the administrator. I guess then it’s up to him to decide what to do about it.

But I did come across one fun bit, in a footnote on page 467 which pointed to this quote from an article from 1967 by economist Martin Shubik:

Each person should have 700 friends! That reminded me of something from our 2006 paper on social networks:

We estimated the average person to know about 700 people. Too bad we weren’t aware of that Shubik paper or we could’ve cited it. According to wikipedia, Shubik didn’t retire until 2007 so if we’d only known we could’ve told him about our finding.

Nowadays, of course, we can know lots more than 700 people, because you can know people online without ever meeting them at all!

In 2016, Donald Trump lost the popular vote to a weak candidate, although the mysterious workings of the Electoral College gave him the presidency. During his first term, he never reached a 50% approval rating. In the 2020 election, despite having the advantage of incumbency, he lost both the popular and electoral votes to a mediocre candidate. So why did the Republicans ignore this record of failure and nominate him again in 2024? Most observers seem to think that the answer is obvious–it was because he had a strong hold on ordinary Republican voters.

Actually, though, the polls show that about the same proportion of Republicans wanted Trump to run again in 2024 as wanted Romney to run again in 2016. In both cases, the previous election’s loser had strong support among the rank-and-file, but voters were open to other options.

Weakliem continues:

Turning from voters to elites, here are endorsements from Republican senators and governors (data from Ballotpedia).

Out of a total of 76 governors and senators, 44 endorsed Trump and only 11 endorsed other candidates (four of those endorsed Trump after their first choice dropped out, so 48 endorsed Trump before the race was settled).

My [Weakliem’s] overall conclusion is that “the base” didn’t impose Trump on Republican elites; Republican elites asked for him.

Interesting. But a complicating factor here is that the elites were responding to the voters: in 2024 as in 2016, Trump confounded his doubters by performing well in Republican primaries.

It’s a bit of a puzzle why they’re not happy with incremental gains, given that they already have so much and they’re well situated to keep getting more.

The best answer, I think, is that these activists think that the current system is (a) going in the wrong direction and (b) unreformable. That’s how left-wing activists in the 1960s thought about the military-industrial complex and it’s how right-wing activists today think about an unsustainable debt caused by mandated payments to special interests. (Of course I’m drastically simplifying in both cases, just trying to give the basic picture.)

A similar case could be made by comfortable environmental activists who say that our industrial society is unsustainable. The difference is that environmentalists today, unlike left-wingers in the 1960s and right-wingers today, are far from power: some environmentalists might like to shut down the current economic and political system but they’re in no position to come close to doing so.

In all cases, the let’s-blow-up-the-system people are just a subset of the concerned activists. Lots of left-wing activists in the 1960s wanted to end the war and change foreign policy while keeping a social-democratic system of government, lots of right-wing activists today would like to reform government while keeping a stable rule of law, and lots of environmental activists would like to just ratchet down consumption without any major break in the system

The blow-it-all-up people just have some mixture of extreme frustration about the ability of the system as it is to be reformed, plus I assume some personal attraction to the idea of tumult and revolution.

I still stand by the statement in my earlier post that part of the story is that if you already have it all, you feel invulnerable—vindicated by all the past gambles you’ve taken that have paid off—and you’re willing to throw the dice once again.

Another part of the story, I think, is that super-rich tech investors, who are key figures in the blow-up-the-system movement, have an economic ideology in which they got where they are by being the nimble “mammals” taking over from the lumbering “dinosaurs” of rust-belt industry, legacy media, and liberal politics. The point here is that the next logical step is for the current tech lords to themselves become dinosaurs; thus they don’t feel as secure as the rest of us might think they are. As I wrote a few months ago:

On one hand, the storyline is that you’re gonna get overtaken by hungry young newcomers; on the other hand, you’re supposed to stay on top forever. The result, at least for Facebook, seems to have been a kind of desperation, a sense that on one hand they are the kings of the world and that on the other hand they are destined to fail and so they have to try harder and harder to grow and grow and preserve a near-monopoly status. And that’s how you get these executives who control unimaginable fortunes and yet are willing to lie and cheat (I wanted to say “lie, cheat, and steal,” but I don’t know if there was any actual stealing reported in that book), indeed they seem to feel that they have to like and cheat and manipulate the rules and all of this to stay on top.

I suspect this is connected to their surprising (to me) willingness to overturn the political system. This is a system that’s made them unimaginably rich and powerful, but, in their ideology, that wouldn’t stop them from becoming the U.S. Steels, General Motorses, and CBS Newses of tomorrow. They’re willing to blow it all up, even though they’re benefiting so much from the system we have now.

Counterfactual utilities evaluate decisions not only by the realized outcome under a given decision, but also by the counterfactual outcomes that would arise under alternative decisions. By generalizing standard utility frameworks, they allow decision-makers to encode asymmetric criteria, such as avoiding harm and anticipating regret. Recent work, however, has raised fundamental concerns about the coherence and transitivity of counterfactual utilities. We address these concerns by extending the von Neumann-Morgenstern (vNM) framework to preferences defined on the extended space of all potential outcomes rather than realized outcomes alone. We show that expected counterfactual utility satisfies the vNM axioms on this extended domain, thereby admitting a coherent preference representation. We further examine how counterfactual preferences map onto the realized outcome space through menu-dependent and context-dependent projections. This axiomatic framework reconciles apparent inconsistencies highlighted by the Russian roulette example in the statistics literature and resolves the well-known Allais paradox from behavioral economics. We also derive an additional axiom required to reduce counterfactual utilities to standard utilities on the same potential outcome space, and establish an axiomatic foundation for additive counterfactual utilities, which satisfy a necessary and sufficient condition for point identification. Finally, we show that our results hold regardless of whether individual potential outcomes are deterministic or stochastic.

I have to admit that I don’t see the appeal of utility functions based on counterfactuals. For example, I’ve never thought that the decision-theoretic concept of “regret” makes sense. That said, I know that a lot of people are interested in the topic, so I hope the above paper is useful to people in clearing up these issues, and I’m glad that they were able to use our Russian roulette example.

I am studying people’s preference for categorically correct forecasts (such as getting the winner of a sports game right) over error-minimizing ones (such as getting close on the margin). We have experimental evidence of this, why it happens, etc.

What I would be interested in doing is demonstrating that this preference is/can be a mistake. To do so, it would be nice to show that doing well in terms of minimizing continuous error is a better predictor of future winner-picking than is doing well in terms of winner-picking. I am curious if you have any leads as to some existing dataset that would be helpful here, or some simulation/modeling strategy that may work.

I replied that, yes, this relates to a point we made here.

Recently Naborn followed up:

The blog post you sent (and a couple others of yours) were very informative for our background thinking. My work (with Jonathan Bogard) forecast evaluation is now published at the Journal of Marketing Research.

And here’s the abstract:

People routinely make decisions based on predictions made by others (e.g., political pundits, market analysts), so it is in their best interest to identify high-quality forecasts. Experts characterize good forecasting as minimization of continuous error (i.e., predictions close to the eventual outcome). By contrast, the present work reveals that laypeople typically see good forecasts as those that correctly predict an event’s categorical outcome (e.g., the winning team). Using within-subjects, between-subjects, and incentive-compatible designs, fifteen studies demonstrate this “pick-the-winner-picker heuristic” as well as its psychological mechanism: People evaluate forecasts by assigning separate weights to (a) categorical correctness and (b) continuous error minimization, depending on the overall importance of the categorical and continuous dimensions for that situation. Thus, in the common case when the categorical dimension matters most (e.g., sports contests), people prize forecasts that accurately predicted the categorical outcome (e.g., the winner, not the margin of victory). However, when the categorical dimension’s stakes are experimentally reduced, an attenuation is observed. While this describes how people typically evaluate forecasts, crucially, a dimension’s importance is not necessarily related to its diagnosticity of forecaster skill or reliability. Accordingly, the pick-the-winner-picker heuristic may constitute a normative mistake, while framing manipulations help debias judgments.

Interesting. It’s good to see research on this topic.

• David W. Hogg. 2026. Why we do astrophysics? arXiv 2602.10181.

I like that his first move is to insert an ungrammatical question mark into the title to question the whole process.

There is a lot to unpack here, starting with the claim that LLMs “show no signs of intelligence.” I seriously don’t know what “signs” people are looking for other than the ability to converse in 200 languages fluently in just about any subject known to humanity. But like most AI benchmarks, we tossed out the Turing Test as soon as a computer could make a passable go of it.

One comment I can get behind is, “a data scientist who has taken an astronomy class might be better prepared [to deal with astrophysics data and modeling] than an astronomer who has taken a data science class.” The usual feeling among scientists is the opposite, but I’m guessing this is because they’ve just never worked with truly talented programmers like the ones we have at Flatiron (e.g., Brian Ward, Steve Bronder, Jeff Soules, and Robert Blackwell, among the folks with whom I work).

The note is stuffed with both descriptive accounts and normative statements such as “Every scientific paper is written to help all of its writers, and all of its readers, learn and grow, no matter their career stages.” I’m imagining living in the Big Rock Candy Mountains of academia, where that’s true.

I just don’t get “I believe that (in astrophysics) software is written to support the astrophysics literature, and that every important piece of software should have an associated paper in the astrophysics literature.” Why not publish software papers in software venues like JOSS? Most literatures won’t even publish software papers, so good on the astrophysics folks for allowing them.

Folks who bemoan the publish-or-perish mentality in academia should consider the statement, “My point is that astrophysics is the astrophysics literature.” I agree if you extend that to include things not peer reviewed like software.

David takes on workflow with the contentious statement, “a lot of ‘implicit knowledge’ or folklore about things like how to observe, how to reduce data, how to organize projects, how to visualize data and models, how to read and write, and so on. Much of this never appears in the literature. Is that not also astrophysics? Yes it is, but it is astrophysics practice.” I’m postmodern enough to believe it’s impossible to separate astrophysics from astrophysics practice. Isn’t the computing also astrophysics practice? He wants that in the astro literature. For this, he concludes, “I would welcome a project in which we tried to make much of this implicit knowledge explicit.” Just not in the literature? We never could get the workflow paper published.

I know David’s senior enough to know how the sausage is made, so I don’t see how he can claim, “Papers (and the authorships on those papers) and the citations of those papers are not ‘coin’ of anything!” Of course they’re the “coin” of both hiring, tenure and promotion committees. I believe Andrew has gone on record saying people value publications so highly because nobody wants to devalue the coin that put them where they are (though I’m sure he said it more cleanly than that).

David says, “it isn’t really correct, if you use an LLM, to give that LLM co-authorship on your paper,”. It’s not legal in the United States, either.

More normative statements like, “You can’t decide not to cite a relevant paper because you don’t like the author, or don’t like the author’s institution, or don’t like their funding sources.” I guess he hasn’t hung out in linguistics. Or mathematics. My undergraduate advisor, Ed Palmer, was a student of Frank Harary. Harary and Béla Bollobás developed an entirely parallel theory of random graphs, proving all the same theorems and doggedly refusing to cite each other.

This just seems patently false: “Anyone who has the capability of getting a PhD in astrophysics has the capability of doing many remunerative things, substantially more remunerative than my job.” He then goes on to one of his main philosophical (economic?) points, “If all we really wanted was to know how the Universe worked, we would start a hedge fund, and use the proceeds to pay an astrophysics institute, filled with people who wanted to do astrophysics rather than find out the answers.” I’m not convinced that someone who’s good at science will also be good at business. I think folks like Jim Simons, who co-founded our institute with Marilyn Simons, is an exception. Sure, if you can make tens of billions of dollars, you can fund a lot of science. Or buy a lot of Jamaican beef patties to bring it back along to themes of the blog.

I don’t understand why he says, “No astronomer (that I know) is improving the calibration of JWST instruments because they want the US Navy to have a higher kill rate.” Is that because he doesn’t think people who do this should be called “astronomers?”

Some of the arguments are silly, but at least get those niggling question marks, “We put ‘the universe’ in ‘the university’?” I also find trickle-down theories of training to be particularly weak. Here, David says, “We train a technical workforce.” Sure, but training a technical workforce by teaching them astrophysics seems very inefficient—see the point above on hiring data scientists versus astrophysicists. Perhaps the weakest argument is that astrophysics funding takes away from even more dangerous things the government could be funded, under the heading “We beat ploughshares into swords.” I also think “We create opportunities for development.” in the sense of development sites in Chile and similar places for observatories. Just think about how useful that money could be spent in some other way for development? On the other hand, I can fully get behind, “Astrophysics is a satisfying activity.” I’m still surprised I get paid to do what I love!

A lot of this paper is about LLMs, but I don’t think that’s the interesting part. Right in the abstract, David discusses, “two possible (extreme and bad) policy recommendations related to the use of LLMs in astrophysics, dubbed ‘let-them-cook’ and ‘ban-and-punish.’ I argue strongly against both of these; it is not going to be easy to develop or adopt good moderate policies.”

As a good writer, he sticks the landing. First, by reframing the question as, “Finally: Why did I write this white paper? I wrote this because I became concerned about some of the ideas circulating in the astrophysics community about LLMs and their capabilities, conflating what are (in my view) text-interpolators with what are (in my view) scientists.” Then concluding ” Ultimately, I think the real question we face—if we do indeed face a question—is not the question of how we do astrophysics. It is the question of why we do astrophysics.”