To some, this is an overdue rebalancing. Nate Silver, for example, calls getting a PhD a “much worse value proposition than 20 years ago”, and predicts that elite higher ed will become “~50% less relevant in the new steady state,” which is in his eyes a good recalibration.

But it’s worth reflecting on what is lost exactly, if this dwindling of minds and resources continues? How should we think about the value of what universities provide over industry, like intellectual autonomy, or training on how to think scientifically? As a professor, I could make a list of the things that have kept me in academia–being free to work on the problems I find most important, the diversity of topics I can work on at any given time, grad students who care about doing deep work, having time to think about the best solution to a problem. But at an aggregate level, it’s less clear what the equation is.  

As I was puzzling over all this, I attended a metascience conference, where there was a panel on “the social contract for science.” This is the transactional relationship dating back to at least the 1950s, by which scientists receive public funding and autonomy and society gets the benefits of scientific research. Back in the 19th century, scholars began to make a distinction between “pure” science–research unmotivated by any particular application–and applied science. The social contract takes this distinction and further presupposes a dependence relationship: what is confusingly called the “linear model”, the idea that pure science provides the well from which applied science contributions are drawn. Threaten this foundation, e.g., by letting applied science intercept too much of the resources society puts toward science, and we risk running out of useful innovations. Or so the story goes.

The social contract for science was an attempt to cement the importance of scientific understanding to society, making it an interesting counterpart to the current moment. If the actions of the current administration to direct funds away from universities, and the possibility of using AI to produce research output without understanding, are threatening our sense of what science should be, the history of science policy provides some perspective on how our expectations got shaped in the first place.

In search of the mysterious fruits of basic science

I’ve been reading the work of philosopher Heather Douglas, who has traced and critiqued the basic versus applied science distinction, the linear model as justification, and the idea of scientific freedom as limited social responsibility (see, e.g., here and here, or her book on the value-free ideal). Popularized by Vannevar Bush after WWII, in a report prepared for President Roosevelt, basic science is a reframing of pure science, presented as “scientific capital,” providing the principles and conceptions to power new products and processes years into the future. Bush called for deliberate policy to guard against the otherwise inevitable scenario where applied science drives out the pure. One of the eventual outcomes of his report was the creation of the NSF.

But despite the pragmatic nature of basic science espoused by Bush, as a derivation of pure science, it is hard to separate from less tangible values. One is that scientific understanding is a good outside of practical application, at both the individual and societal level. The earliest advocates of pure science associated it with being closer to God. Post-Enlightenment, this view gave way to a more secular superiority complex, which implied the strong character of the pure scientist, who chose to eschew wealth. “The highest occupation of mankind”, Henry Rowland called it in his Gilded Age era essay, “A Plea for Pure Science,” which bemoaned the vulgarity of attributing scientific greatness to the applied scientist rather than the pure.

From a less moralistic point of view, we’ve been encouraged to believe that a society that has rigorous ways of understanding the world is better off over one that doesn’t. Throughout history, understanding the laws of Nature has been portrayed as a good in itself, along with an intellectual life. From this view, by educating people on how to pursue deep understanding of the world, universities provide the general good of scientific thinking to society. If we believe in the intrinsic value of reading, writing, or intellectual discussion, then it would seem we should value the university as a place that provides the kind of timespan and environment needed to develop these skills.

Some argue that the university has come to serve too many conflicting purposes (research engine, job training center, credentialer, incubator of coming-of-age experiences), and should go back to its classical roots: training in oral reasoning and rhetoric, ethics and moral judgment, historical analysis, and the cultivation of taste and discrimination. This may be a useful refocusing, but it offers little consolation for the fact that the elite research infrastructure that helped this country establish and maintain scientific leadership for decades is in the process of being gutted.

If we take our intuitions from the linear model, we might protest that innovation will suffer if universities’ research purposes are deprioritized. The post WWII science-industrial complex expanded the presence of basic research in industry, but studies suggest that the knowledge generating role of corporate R&D has been on the decline for years. To the extent that basic research is the supplier of downstream applications, it would seem we need universities more than ever.

But the distinction between basic and applied science that’s become synonymous with how we envision science has never been airtight. Critics questioned how an institution could be built around a distinction that seemed to amount to little more than a difference in intention, since applied research sometimes produced important new general knowledge, and pure science contributions sometimes had direct applicability. 

AI research is a recent example. Not only is serious money being made without necessarily requiring advanced degrees, research positions do not require PhDs. By some accounts, passing 30 years old puts one in the older demographic of researchers at frontier AI companies. Yet much of the visible innovation in frontier model development has been heavily concentrated in industry labs, including transformer models, scaling laws, and AlphaFold.

Of course, AI owes much to academia. The amazing thing about deep learning and LLMs, to anyone who was paying attention to NLP before these developments, is that after many years of AI research contributing interesting questions but lackluster results, the technology finally seemed to work. Would we have had the foundations for deep learning if perceptrons had not been stubbornly pursued by academics like Frank Rosenblatt at Cornell early on, picked up again in the 1980s by Rumelhart and McClelland’s Parallel Distributed Processing group, despite multiple periods during which consensus said connectionist approaches were unlikely to pay off?

The challenge is that arguing that “someday the research will pay off,” without being able to point to any hard evidence that basic research is, on average, worth the investment, is not such a convincing argument. According to Douglas, studies have been attempted to show the payoffs of basic research, but without very impressive results. Uncertainty about what time scale we should expect between discovery and application makes this kind of exercise difficult.

At the same time, it’s hard to dispute that monetary incentives can sometimes discourage exploration that would eventually pay off. In evaluating the role of academic research to AI progress, we should keep in mind the uniquely massive private investments AI companies have received, and be cautious using it as a general example. It would be premature to conclude that because progress (in terms of models’ standalone capabilities as measured by benchmarks) doesn’t seem to depend much on academic research at the moment, cutting off academic research would be immaterial. Particularly unfortunate about the historical contingency of frontier companies defining the direction of the field is that they are focused on a pretty narrow space of methods, evaluations, and design ideas. But the power and resources they hold give newcomers to AI research the impression that ideas outside this narrow space aren’t important.

Indulgence, autonomy, and social responsibility

As suggested above, it’s always been tempting to bring moral judgment to bear on the basic versus applied research divide. The latest moment with AI research is no exception. Does the moral high ground belong to those who are staying in academia, underfunded or not, to preserve university culture and their autonomy from corporate interests, or those who are willing to give up a comfortable job to shape the impact of AI as a product in the world?

From one perspective, the academy, as a haven for basic science, has always been at risk of being seen as indulgent. In practice, building a scientific career is in many ways a process of identity development and fulfillment for the scientist. Historical pure science rhetoric associated the pursuit of scientific truth with self-realization. But talking about personal fulfillment does not go over well when your opponent is promoting the idea that science could do more direct good for the country or humanity. Academic scientists have always been at risk of coming off as being self-indulgent, insular, or dilettante when they defend understanding for understanding’s sake. The current political moment is just rehashing old themes.

Another unfortunate historical association of basic science is with insularity and shirking responsibility. After WWI era advances in chemical warfare and explosives, the social responsibility of the scientist became a much greater concern. Philosopher John Dewey came down sharply on the idea that an autonomous space for pure science, unhindered by societal concerns, was something to strive for. Instead, he argued that this impetus to protect pure science was partly a convenient abdication of moral responsibility for the downstream outcomes of research, a “shirking of responsibility.’’

Dewey’s concerns came at a time where philosophy was itself seeking to be more scientific. According to Douglas, Dewey’s views on how philosophers should approach science–through greater integration of societal concerns–lost out to the argument espoused by Bertrand Russell, who instead valorized the “disinterested intellectual curiosity which characterizes the genuine man of science.” The latter view became the more accepted one, and our definition of scientific freedom arose in tandem with expectations of limited social responsibility. It’s not particularly surprising, then, to encounter beliefs that academia is not the place to go if you want to have impact in the world.

At the same time, it seems hard to deny that at this point of time in AI, where we have a large imbalance of power and resources, there is something to be said for the autonomy afforded by the university or nonprofit. Some beliefs about AGI coming out of Silicon Valley border on religious. My biggest concern if I were to join an AI company at this point in time would be losing my ability to think for myself about what problems deserve priority. Having greater agency and impact are attractive, but not if they come at the expense of one’s internal compass or values. As Brendan McCord said recently, “Autonomy is different from agency. Agency is getting things done…You can be more effective than you’ve ever been, and you can be less the author of your own life than you’ve ever been.” Against the groupthink of Silicon Valley, the value of the intellectual autonomy academia provides does feel real. Though it’s unclear how valuable this autonomy will continue to be if academics and others outside the big labs can’t retain enough funding or visibility into frontier model development to remain relevant.

The problem with defining progress as prediction and control 

In a 2014 article called Pure science and the problem of progress, Douglas suggests that if the pure/applied science distinction doesn’t survive scrutiny (which she argues it does not), we’re left with an account of scientific progress based on our ability to predict, intervene, and control our world. But this is not a definition of progress we should be content with:

“Any increase in the capacity to predict or control the thoughts and feelings of human beings would count as scientific progress. An increased capacity to destroy human subpopulations (through, say, targeted pathogens) would count as scientific progress. Developing new heinous capacities would count as scientific progress. Unlike Rowland, we should have no illusions that greater causal efficacy, greater power of intervention, will in fact always provide a better society.” (p. 63)

If misaligned AI, our own creation, changes how we view ourselves and the world, if it convinces some of us it has all of our best interests at heart even as it feeds our insecurities, or pursues its own goals in the background, is that scientific progress?

Douglas argues that judging real progress requires society to weigh in. When it comes to AI, this is happening through pushback against data centers, and the pace of AI progress, and the culture of Silicon Valley. Adoption matters too, but can’t be a substitute for evaluation. We need institutions independent of the companies to help interpret what’s going on. In the midst of changes to so many of our current institutions, we should expect the story we ultimately tell to take time to sort out.

In the meantime, defending academia as a category of research, or a moral standard, is a dead end. What seems more reasonable to advocate is a set of conditions — time, autonomy, training in scientific judgment, the evaluation of new approaches independent of their profitability. The value of these ingredients isn’t easily summarizable in some neat story, because what drives scientific progress is not that simple. But institutions that help society judge what’s been achieved seem worth defending.

• Cousteau was a world-famous underwater photographer . . . actually the only famous underwater photographer.

• Another example, also from the 70s (sorry): Marcel Marceau. A world-famous mime . . . actually the only famous mime.

• The guy who wrote All Creatures Great and Small was a world-famous veterinarian . . . actually the only world-famous veterinarian.

• Jackson Pollock is a world-famous drip painter, and indeed the only famous drip painter. But it’s not clear to me that “drip painter” should count as a category.

• Stradivari: same idea, he works if you’re willing to count “violin maker” as a category.

• Tony Hawk: world-famous skateboarder, the only world-famous skateboarder.

• Luther Burbank: world-famous agriculturalist, the only world-famous agriculturalist.

• John Philip Sousa: world-famous composer of marches, the only world-famous composer of marches.

• Margaret Mead: world-famous anthropologist, the only world-famous anthropologist.

• Alan Turing: world-famous codebreaker, the only world-famous codebreaker.

• Temple Grandin: world-famous slaughterhouse designer, the only world-famous slaughterhouse designer. She’s kind of like Joseph Joanovici, the only world-famous scrap metal dealer, in that she’s famous for her story more than for her occupation; nevertheless, as with Joanovici, the occupation is central to the story.

• Noam Chomsky: world-famous linguist, the only world-famous linguist.

• Jim Henson: world-famous puppeteer, the only world-famous puppeteer. OK, maybe we need to count Frank “Yoda” Oz too.

• Frederick Law Olmsted: world-famous park designer, the only world-famous park designer.

• Oscar Pistorius: world-famous paralympic runner, the only famous paralympic runner. This works even without adding “killer” to his title, but it was the killing that kept him famous.

• Weird Al Yankovic: world-famous writer of novelty songs, the only . . .

• We could also throw in Dr. Demento: world-famous DJ of novelty songs, but that seems like too narrow of a category.

• Anna Wintour: if you’re willing to count “fashion magazine editor” as a category.

• Freddy Mercury is the only world-famous person from Zanzibar, but I don’t think that really counts. We’re talking here about categories defined by what you’ve done, not where you’re from.

And, of course:

• Evel Knievel: world-famous daredevil, the only world-famous daredevil.

There was some discussion of how famous he still is, and commenter Manuel wrote:

Evel Knievel is just one blockbuster biopic away from being world famous.

This made me wonder: why hasn’t there been a blockbuster biopic of Knievel? Wikipedia informs me that there have been three Knievel biopics already, including a biopic back in 1971 starring George Hamilton! and a 2004 TV movie directed by John Badham, who’s not a nobody—he also directed Saturday Night Fever. Also this:

On December 19, 2024, a new biographical film adaptation of Evel’s life was reported to be in the works with La La Land director Damien Chazelle attached to direct, William Monahan set to pen the script, and negotiations with Leonardo DiCaprio to star as Knievel.

That sounds like something.

But then when I was checking something, I saw that something happened:

The number of papers dropped from 259 to 99!

Then when preparing this new post, I checked again:

And again:

I guess someone’s running the bot in reverse.

P.S. It’s down to 44. I wonder what the story is.

P.P.S. Down to 1:

The lone remaining paper is “Sequential Bayesian Pricing of AI Infrastructure,” posted 29 May 2026.

P.P.P.S. And now the page is gone. I guess the experiment has concluded, or the impersonation has been stopped, or something.

P.P.P.P.S. According to this commenter, it was SSRN that took the papers down.

Nicholas Polson has, by my count using his SSRN page, already written 258 working papers in 2026 alone. He’s already written (or at least published to SSRN), six papers today, August 26, 2026.

OK, but who am I to talk?—I’ve written over 200 blog posts this year. But wait:

These aren’t just short notes. Most of the papers are of normal academic length: 32 pages, 27 pages, 58 pages. . . . Obviously the research productivity of Polson and his co-author Sokolov is aided by AI. . . . I have seen any academic, at least not in economics, that has really pushed it to the limit.

Horpedahl writes:

Read any single paper, and it feels like just a normal academic paper, the kind of thing that an academic might work on for a few months.

I wanted to see if I shared that judgment so I clicked through to the list of Polson’s papers on SSRN and looked for something interesting . . . ok, here’s something. It’s called Theories of Human Connection, and . . . ulp! It’s 80 pages long. The paper’s subtitle is “An Interdisciplinary Synthesis Across Economics, Psychology, Biology, Philosophy, Game Theory, and Spiritual Tradition.”

But let’s take a look. The abstract on SSRN starts like this:

Human connection is the most studied and least integrated phenomenon in the social sciences. Every discipline that examines intimate relationships captures something real that the others miss, yet no existing framework holds all dimensions in simultaneous view. This book synthesises fourteen thinkers across biology, psychology, economics, game theory, communication theory, existential philosophy, and spiritual tradition into a unified account. Morris established that the need for physical touch is an evolutionary drive as fundamental as hunger, with a biologically ordered sequence of escalating vulnerability whose disruption produces intensity without depth. Bowlby showed how early caregiving creates invisible templates governing adult intimacy, encoded in the nervous system before language exists. Becker revealed that partnerships generate value neither partner could produce alone, but assumed partners are interchangeable — an assumption Frankl demolishes by showing that the meaning generated by shared life cannot be transferred, and that meaning multiplies rather than adds to life satisfaction, explaining widespread disconnection in the wealthiest societies in history. Von Neumann’s game theory explains how mutual self-protective withdrawal, individually rational for each partner, produces the disconnection neither intended. Bateson identifies the communication structure that accelerates this collapse: contradictory demands that make any response wrong. Gottman’s laboratory models predict separation with over ninety percent accuracy from the ratio of positive to negative interactions. The Hindu philosophical tradition adds the final dimension: partnership as a laboratory in which selfishness and fear are progressively revealed and surrendered.

We argue that most relationship failures are not failures in one dimension but misidentifications of which dimension is actually in play, and that effective intervention requires the multi-dimensional map this synthesis provides.

From the preface:

The thinkers assembled here — Becker, Bateson, Von Neumann, Schelling, Keynes, Morris, Vaughan, Frankl, Maslow, Gottman, Yogananda, Vivekananda, Maharaj, Polson, Thomas, and Paltrow — did not, for the most part, know each other’s work. They worked in different centuries, different countries, different intellectual traditions. What unites them is that each identified a dimension of human connection that the others left in shadow.

Gottman, huh? The name rings a bell . . . He’s the guy who conned Malcolm Gladwell and various media outlets—maybe he conned himself too—into believing that he could predict divorces with 94% accuracy.

I searched for Gottman in the document and found a whole chapter on that bullshit! You can click through for yourself, it’s chapter 10. Here’s a key bit:

I wonder what prompts were used by Polson and his coauthor to write this article. I guess the prompts did not include, “Evaluate implausible claims skeptically.”

They bring it all together in Chapter 13, “The Cumulative Model: A Unified Theory of Human Connection”:

But let’s not forget “The Master Equation” on page 75:

Jesus Christ. I’ve heard that Cambridge University has an open position in their school of education . . . this kind of thing would fit in very well there, no?

In all seriousness, no, I don’t think this “feels like just a normal academic paper, the kind of thing that an academic might work on for a few months.” At least, not the sort of thing a non-bullshitting academic might write.

I know Nick Polson—he’s a statistician, and he’s done lots of solid work over the years! What happened here?

Here are a few possibilities:

1. It’s an experiment or a joke. But if it were a joke I’d think there’d be some internal clues, no? It’s hard to imagine playing the whole thing straight. And if it were an experiment, I’d expect they’d all read like straight-up statistics papers, nothing so obviously bogus as “Theories of Human Connection.”

2. Someone else is impersonating Polson. Seems unlikely, but it’s possible. It might not even be personal. Maybe all this is an experiment, not by Nick but by someone else who programmed a chatbot to choose the name of a successful academic and then spew the internet with papers attributed to him. If so, how horrible.

3. “Intellectual squatting.” That’s the conjecture of Michael Makowsky, who writes:

The nice version is it’s putting out a series of half-baked papers in the hopes of establishing a property right to the underlying ideas at an earlier stage of the research process than previously possible. The less generous interpretation is it’s dumping a series of haystacks on the plains and laying claim to the needles probabilistically within each.

I guess . . . but what does Nick ultimately get out of it? Invitations to speak at more conferences?? I don’t get it.

Makowsky writes:

Imagine you are a person who has highly esoteric, potentially important ideas every day. Many of those ideas you suspect, based on some combination of experience and ego, are new in at least one dimension. You would like to get credit for that newness. For being first. What’s the problem?

The problem is that scholarship remains more perspiration than inspiration. Having a new idea is great, but it takes years to work through the nuance in sufficient detail that you can convince your peers of the coherence and originality of the contribution. During the minutes each day you are not working on this singular project you have the inspiration for other ideas, sometimes multiple within a single day. How frustrating is the proposition that someone else gets credit for the originality of contribution just because they had time to reveal it to the world while you were embroiled in your investigation of what is only one of your many score ideas!?

Ah, but meta-level inspiration has struck you! What if you took each one of those ideas, spent an hour curating a series of prompts around it, and then let Chat GPT (or another LLM) fabricate an entire research paper around it? . . .

But . . . that’s what blogging’s all about! Often when I have an idea or a reaction, I blog it. No need to pipe it through a chatbot; I’ll just save the cycles and post it right here.

To return to the 256 working papers, I can think of one more motivation:

4. Education. This seems like the most plausible explanation to me. Polson has had an active research and teaching career, and he’d like to share his insights with a broader audience than the readers of his published papers and the students at the University of Chicago business school. And one way to reach people is . . . econ preprints! So Nick picks 258 interesting topics, writes some prompts for each, and produces the articles. I guess he’s programmed a bot to do this. He just feeds it the prompts and the bot writes the paper and posts it directly to SSRN.

That could explain the mystery of how that ridiculous 80-page article with “The Cumulative Model: A Unified Theory of Human Connection” (shades of Stephen Wolfram!) ended up there. Not only can’t you expect an author to write 258 articles of that length in less than a year, you can’t expect him to read all of them too. The content of that bizarre article could be as much a surprise to Polson as it was to me.

This then raises a question: setting aside the motivations of Polson (or his impersonator), do these 258 papers have any value?

It’s hard for me to answer this question, given that I’ve only looked at one of them. My guess is that the net value of the papers is negative, in that the amount of time that people (including me) have wasted going through them outweighs any positive contributions that might have been there.

My suggestion

Here’s what Nick could do on this, which could have value: Take these 258 prompts and write an article (himself, not using the chatbot) explaining why he thinks these ideas are important. Aki and I wrote a paper a few years ago, What are the most important statistical ideas of the past 50 years?. Nick could write something similar: What are the 258 most important things in statistics to learn today? Or something like that. I’m not saying it would be easy—it would take more effort than programming a chatbot to spam SSRN—but valuable products often take work to produce. Nick has tenure and could set aside the time to do it.

Also I’d recommend withdrawing all those papers from SSRN. Withdrawing 258 papers seems like a lot of work, but I’m sure he could easily program a bot to do the job.

P.S. There’s a further twist: there are two accounts for Nicholas or Nick Polson at the University of Chicago business school; see this comment thread. This would seem to be consistent with the “social experiment” hypothesis (if Nick decided to set up a separate account to play around with) or the “impersonation” hypothesis (if the bot that wrote and posted these papers was not created by Nick at all). The whole thing remains a mystery to me.

P.P.S. OK, I did a bit more nosing around.

SSRN allows you to list the papers in time order. If you go to Nick’s SSRN page linked from his website, you’ll see 16 papers, with the first (“The Impact of Jumps in Volatility and Returns”) being posted on 1 Jan 2001, then others through the next two decades, with the most recent being “Deep Learning in Characteristics-Sorted Factor Models,” posted on 23 Sep 2018 and last revised 26 Jun 2023.

If you go to the SSRN page with all the fake papers, it starts with “Kramnik vs Nakamura or Bayes vs p-value,” posted 7 Dec 2023. It’s a badly written paper—I’m guessing not AI, just text by a non-English-speaking author that was not ever checked by native speakers before posting. This rings a bell . . . I actually have a blog post on this paper, scheduled to appear next year. Next on the list is a 25-page paper, “AI and Vivekananda,” posted 5 Mar 2024, then a gap of two years until another AI-related paper appeared on 9 Mar 2026, then on 11 Mar 2026 came the aforementioned “Theories of Human Connection.”

So, yes, Polson has two SSRN pages, but they have no overlap in time. He also has papers on Arxiv, including the intriguingly-titled “Bayes with No Shame: Admissibility Geometries of Predictive Inference,” dated 24 Aug 2026 . . . Hey, that’s just 3 days ago! Oddly enough, I can’t find this one on SSRN.

But what about the article itself? I don’t have the patience to read it, but I did catch that it mentions the martingale property, which is one of my current interests—that’s cool. But, just flipping through, it looks much more substantive—much more like a real scientific paper—than that horrible “Theories of Human Connection” thing. This could be a tribute to the power of modern chatbots to create something so convincing.

P.P.P.S. Update here: The incredible shrinking SSRN page.

P.P.P.P.S. Vadim Sokolov, coauthor of several of the chatbot-assisted papers, comments here. Assuming this comment itself is legitimate, those 258 papers were not an experiment or a hoax, nor was there any impersonator, nor was it intellectual squatting. Rather, Polson and his coauthors just had 258 different things to say, and they felt the best way to do so was using a chatbot to write tens of thousands of words on these topics.

Also, according to Sokolov, they did not program a bot to do this. He reports that at least one of the papers went through multiple rounds of review, so even if a chatbot was involved in that one, the human contribution was much more than inserting a prompt. He says that some of the papers “are much newer and were developed with much more extensive AI assistance. Modern AI made it possible to develop and complete this material at a speed that would previously have been impossible.”

The fallacy in this statement by Sokolov is the idea that using a chatbot to turn a few hundred words of prompts into ten thousand words of text is a way to “complete this material.” Nothing’s being “completed” here in any intellectual sense; the process just adds pages and pages of froth. I think that Josh’s analogy of this to the Sorcerer’s Apprentice is apt.

P.P.P.P.P.S. Polson appears to think he’s proved the Riemann hypothesis (see comments here and here). I would think this would cause some concern among his collaborators.

I’m looking for postdocs and doctoral students to work on Bayesian workflow. The candidates need to have knowledge of Bayesian inference and some experience with building models (for real applications, as part of methods development, or as part of courses). Although we have published Bayesian workflow book, there is still a lot more to do. The focus in the group is in cross-validation, model checking and inference diagnostics (see my publication list).

All positions are fully funded and the salaries at Aalto CS are 53k€-55k€ / year for postdocs and 40k€-45k€ / year for doctoral students. There are occupational healthcare and other benefits. Postdoc positions are typically offered for up to three years and doctoral student positions for four years. Starting dates are flexible and the details of each position will be agreed individually.

You can apply via joint ELLIS Institute Finland call and pick me as your preferred supervisor.

[Regarding] the replication crisis, researcher degrees of freedom, and the gap between what statistical methods claim to establish and what they can actually support . . . Looking at the current state of the social sciences, do you think the culture of research has genuinely changed since the replication crisis became widely discussed, or has it mostly generated new compliance rituals around pre-registration and open data while leaving the underlying incentive structure intact?

And a question about Columbia specifically since you are there: the university has had a difficult two years in ways that have played out publicly. From your position as someone who thinks carefully about institutional incentives and measurement, what do you think the administration consistently misread, and is there a statistical or social scientific way of understanding how institutions lose the ability to accurately perceive their own situation?

My reply:

1. I’m loath to give an answer about the changes in the culture of research because I have not studied this systematically. My impression is that, yes, there’s more skepticism and less acceptance of noisy N=38 papers in psychology, etc., and less toleration for unfalsifiable evolutionary psychology and that sort of thing. On the other hand, perhaps this has just shifted from the science establishment to social media. Ten or fifteen years ago, there was a pipeline (partly abetted by Jeffrey Epstein) from researchers at top universities to publication in top journals to books, NPR, Ted, Gladwell, Freakonomics, etc., and lucrative speaking and consulting gigs. So you get people like Marc Hauser or Albert-Laszlo Barabasi or Brian Wansink or Dan Ariely doing the basic research (such as it is), academic middlemen such as Steven Levitt and Cass Sunstein as promulgators, and the universities, journals, and prestige news media as part of this system (as for example here: https://statmodeling.stat.columbia.edu/2023/08/31/the-variation-ignoring-junk-science-thats-promoted-by-association-for-psychological-science-and-related-academic-celebrities-its-like-a-poker-player-thinking-okay-if-push-all/).

Nowadays, though, social media runs on its own steam, and the models for academic junk science are researchers such as Andrew Huberman and Dr. Oz, who cut out the middleman and promote junk science directly, sell supplements, etc. And social media is full of fake news and AI slop. They don’t really need NPR, Ted, Gladwell, Freakonomics, etc., anymore; they can do it on their own. So, in short, yes, I do have the impression that science has reformed from the bad old days of 2010-2015 (about which, see this article with Simine Vazire: https://sites.stat.columbia.edu/gelman/research/published/jmmss-3062-gelman.pdf), but maybe the public intellectuals don’t need academic science anymore; they can just make up whatever they want on their own.

2. My take on Columbia is similar to my take on many institutions, which is that they have an executive function but minimal legislative or judicial functions; I discussed this here: https://statmodeling.stat.columbia.edu/2018/01/19/lesson-charles-armstrong-plagiarism-scandal-separation-judicial-executive-functions/ and here: https://statmodeling.stat.columbia.edu/2025/11/11/from-the-three-branches-of-government-to-the-bidirectional-nature-of-legal-reasoning-in-a-way-that-is-similar-to-how-statistics-works-and-should-work-in-the-real-world/. As a result, their decisions are made on consequentialist rather than proceduralist gounds, and over and over again the administration takes the seemingly reasonable decision to cover up misdeeds.

Ford posted this discussion on his blog. It was kinda weird seeing myself discussed as a sociological object, but, fair enough, I’m a public figure, and people can say what they want as long as they don’t misrepresent my writings or claim that I said something I never said.

Ford’s assessment is accurate that I’m not very good at strategic behavior so often I don’t even try. It’s similar to how I’m a bad negotiator so usually I’ll just try to make my goals clear and not try to optimize, following the “Getting to Yes” principle that the main thing getting in the way of smooth negotiation is ignorance of other people’s goals. I think back to various successful and botched negotiations I’ve been involved with in the past, and almost always the problems come with struggles over details without there being clarity on the goals of the different parties.

Bob Carpenter shared a Stan example to get a flat prior over tables that match specified margins. I think this prior would imply a prior on what the authors call alpha0, the covariance coefficients for the target geography. The method as described in the Modeling Complex Contingency Tables poster seems to focus on point estimates:

In other news, Shiro responded on Twitter to one of my questions about their Application 1 (ACS): “the density plot shown there is an empirical density of 1700+ TREs, where each error is for a non-Southern county.”

I have 2 remaining questions:

1. What are the covariates w in this ACS example ?
2. Suppose you also have survey data in Palo Alto. Would this be added to the training tables ?

Their Application 2 asks if lower postratification table reconstruction error improves the downstream MRP:

In the comments last week, Shiro cited related work by Si and Zhou (2021) who propose a method called Bayes-Raking to incorporate known margins into modeling. They found Bayes-Raking was similar to raking in the overall mean but outperformed raking for subgroups:

It would be interesting to directly compare Bayes-Raking to the poster’s SynthMargins !

Our wonderful new Bayesian Workflow book is now available as a free pdf! Just go the link—it’s right there!

I recommend getting the hard copy too because you’ll want to be able to read it while working on the computer, and the cost of the book is trivial compared to the benefit from faster learning that you will get by being able look at the book without taking up valuable screen real estate; also you can see connections when flipping through the pages that might not be apparent by viewing one page at a time on a screen.

Conversely, if you have the hard copy, you should still download the pdf because it fixes a bunch of minor errors that we caught after the book went to press. Also in the printed version we accidentally repeated some of the exercises in chapters 2 and 3. For the pdf we fixed this.

Regarding the content, as I wrote last month, with Bayesian Data Analysis, the big steps forward were:

• Going beyond Bayesian inference to also consider Bayesian model building (as a researcher, you construct the model, it isn’t just given to you as in a textbook), model checking (breaking through the absolutely horrible attitude, common to Bayesians in the early 1990s, that the model was “subjective” and thus should not be checked), and model improvement (continuous model expansion, not the misguided idea of assigning posterior probabilities).
• Going beyond simple conjugate models. BDA had lots of hierarchical models, also lots of computational tools so that you could fit the models you want by putting them together from understandable components. And I like how we had a clear separation between modeling and computing. The model comes first, then you figure out how to compute it. Or you set up a model that works within your computational constraints.
• A Bayesian approach to sampling and causal inference. This was Rubin’s framework in which unobserved units in the population and unobserved causal outcomes are treated as missing data and are part of a joint probability model. We worked this out in chapter 7 of BDA (which became chapter 8 in the third edition of the book).
• Lots of live examples. Not just “real-data examples,” but problems we’d directly worked on. This motivated us and I think it gave our readers a sense of how Bayesian methods worked not just in theory but in applied problems.
• A pragmatic view of probability as a measurable quantity. That’s right there in chapter 1. Bayesian methods are not the product of a philosophical stance; they’re a way to connect models and data using probability.

I could go on and on, but for that I can refer you to the Bayesian Data Analysis book.

And these are the key innovations of Bayesian Workflow:

• Going beyond Bayesian data analysis (model building, inference, model checking, and model expansion) to consider the larger process of statistical modeling, including comparisons of multiple models fit to a single dataset.
• A fuller use of informative priors. This is a big deal. In BDA we still had a bit of the Bayesian cringe going on. One reason we’ve moved toward stronger priors is that the replication crisis has taught us that the amount of prior information available in any given problem is often approximately the same as the information coming from an experiment (see here, for example). Informative priors also fit our increased focus on generative modeling, and we’re doing a lot more prior predictive checking to understand the implications of our models.
• More integration between modeling, data analysis, and computing. One way to see this is that the Bayesian Workflow webpage has the code to run all our examples. We also have lots of code snippets in the text as a way of demonstrating the way in which coding is central to our statistical workflow.
• Lots more live examples. It’s been 30 years since BDA first came out. One reason that Bayesian Workflow has 11 authors is that different collaborators worked on different examples (but the three principal authors read through the entire book, so the general approach should remain coherent).
• Simulation-based experimentation. This is something my colleagues have been doing more and more over the years. At its most basic, simulation-based experimentation provides a best-case baseline for statistical methods: if you can’t recover your quantities of interest with sufficient accuracy under ideal conditions (when your data are simulated from the model you’re fitting), then you know you’re in trouble. And often this is the case! Beyond that, we can simulate from one model and fit another, and see what happens. Simulation experiments aren’t always so easy to construct, as they involve specifying the entire data-generation process. But we think this is effort worth expending, as it involves thinking about the problem you’re working on.

I could go on and on, but for that I can refer you to the Bayesian Workflow book.

JSDSS was founded on three core principles.

First, our commitment to open access is more than just lip service. At JSDSS, open access means free to read and free to publish—no exceptions. JSDSS is a Diamond Open Access journal. Research supported by academic institutions, public funding, or personal effort should not require a payment to reach the audience it deserves.

Second, reproducibility is essential and not an afterthought. The credibility of sports analytics research depends on our ability to verify, replicate, and build upon earlier work. JSDSS will actively incentivize transparency in data, code, and methodology, and work that meets our reproducibility standards will receive a special designation recognizing this commitment.

Third, we believe sport is a rich and underutilized laboratory for statistical and data science innovation. From player evaluation and in-game strategy to league design and fan engagement, sports data present compelling, real-world problems that demand rigorous and creative analytical thinking. We intend for JSDSS to be the definitive venue for this work.

Cool! I should send them something. We think about statistics and data science in sports a lot around here.

My only concern is that I get the impression that the cutting-edge work on sports analytics is happening outside of academia. So I hope that this new journal can get useful contributions from people who are working in sports analytics who have material they can share without compromising their competitive advantage.

What contributions can academic statisticians make to sports analytics?

Or we could flip it around and ask, What contributions can academic statisticians (like me!) make to sports analytics? Here are a few things:

– Developing general methods that can then be used in sports analytics (as here);

– Writing textbooks explaining general methods that can then be used in sports analytics (as here);

– Teaching students who can then work in sports analytics, or consulting on sports analytics projects, which can be thought of as a form of intense teaching;

– Doing work in sports analytics which, although it is not cutting edge, can still give valuable insights (as here);

– Evaluation and criticisms of published work in sports-related topics (as here);

– Contributing to the sports analytics community (as here, and indeed as in the present post);

– Collaboration on sports strategy, or sports medicine, or the sociology of sports, or various other places where sports links up with academic research;

– Clearing up confusion on topics related to statistics and sports (as here), along with social-sciency thinking about how these misconceptions persist;

– Sports-related research where it can be helpful to have an outside perspective, something available to academics who aren’t on a deadline (as here).

There are probably some more things I didn’t think to include on this list.

My new posts here have a one-year waiting list, but I’m bumping this one up because it’s important.

OK, we went on over and did a head-to-head Jamaican beef patty taste-off. As the above wrappers indicate, we compared the spicy beef.

None of this randomization, tea-tasting crap, we just kept taking bites of each patty until they were done.

Both were good, but the classic Golden Krust was definitely better than the newcomer, Juici Patties. The Golden Krust patty had a more delicious filling which was also better integrated with the crust. By comparison, the Juici Patty had a bit too much crust and was too empty inside—it didn’t work as well as a whole. Also the Golden Krust patty was $4.05 and the Juici was $4.25.

In summary:

1 Golden Krust patty > 1 Juici patty >>>> 1/1381 of a conference featuring Grover Norquist, Gray Davis, and a rabbi.

Leprosy is caused by a bacterium, Mycobacterium leprae, first identified by a Norwegian doctor, Gerhard Armauer Hansen, in 1873 . . . Convinced, wrongly, that leprosy was hereditary, he spurred Norway to introduce the Seclusion of Lepers Act (1885), which enabled the authorities to remove people from their families to isolated leprosaria.

OK, fine, everybody makes mistakes, better to err of the side of caution bla bla blah.

But then comes this stunner:

After Hansen injected a virulent strain of the disease into the eye of one patient, Kari Nielsdatter Spidsøen, without her consent, she took him to court. He was stripped of his hospital post in 1880, but continued to oversee Norway’s leprosy policy.

Whaaaa?

Further research (i.e., I went to the Hansen’s wikipedia page) yielded this:

Hansen had attempted to infect at least one female patient with the nodular form of leprosy without consent, and although no damage was caused, the case ended up in court and Hansen lost his post at the hospital.

“No damage was caused,” huh? He just injected her in the eye, that’s all. No harm, no foul, I guess.

Just amazing that he continued to run government policy after that. Kinda reminds me of how noted terrorist John Poindexter was tasked by the U.S. government to run a terrorism prediction market. I guess it makes sense—he was a true expert on the topic.

Also reminds me of that unfortunate psychology researcher who keeps coauthoring fraudulent research papers, got disciplined by MIT, and then left for a prestigious chair at Duke University, ran an advice column in a major newspaper, and had a TV show made about his research. Actually two TV shows: one is a highly critical documentary and another is a fictional show where a character based on him is the hero.

Or that political figure, I can’t remember his name now, who keeps citing discredited, fraudulent, fake, and racist research claims, and at the time of this writing he remains in charge of health policy in a major industrialized country.

Some people end up in prison for minor crimes. Other people do things like inject people in the eye with a virulent strain of leprosy, and they get to make government policy.

A frequent commenter pointed me to this paper, “Sport and longevity: an observational study of international athletes.”

All I can say is . . . Wow! This paper is an absolute clinic in bad quantitative social science research.

I’ll leave it as an exercise for the reader to count up all the problems.

The key takeaway: For God’s sake don’t play volleyball. It’ll reduce your life span by 5 years, and the result is statistically significant, with a p-value of 2.4e-11.

I used to play some volleyball. Our team made the B-league playoffs in the MIT intramural league one year. Now I’m scared. Who knows what all that jumping did to our fragile young bodies.

Also this:

Your tax dollars at work!

Too bad the journal doesn’t seem to publish its reviews. It would be hilarious to see the referee reports for this one.

P.S. To those of you who think I’m being mean here, “punching down,” etc., let me just say a few things.

1. It’s not personal. I know nothing about the authors of this paper and I purposely did not include their names in the post. They may be wonderful people, indeed they may do wonderful research in other areas.

2. As noted above, public funds were spent on this project. And the paper was published, i.e. it’s there for anyone to read. If you don’t want to be criticized, don’t publish.

3. Unsupported claims get out there and they don’t go away. For example here’s what happened when I googled *pole vaulting longevity*:

This is just one silly example. But, yeah, I do think we should be bothered by broadcasts of unsupported scientific claims, whether they’re about volleyball, himmicanes, faith healing, air rage, governors’ lifespans, or anything else. It’s bad science and it’s part of our culture of B.S. Even if the authors of this particular paper are completely sincere in their efforts. Remember, honesty and transparency are not enough.

And here are the talks:

Matthew Kay
Adrian Seyboldt
Paul-Christian Bürkner
Charles Margossian
Javier Enrique Aguilar
Sean Pinkney
Nikolas Siccha
Anna Dreber
Kaitlyn Johnson
Jonas Wallin
Pranav Sanke
Anna Elisabeth Riha
Colling Cademartori
Tim M. Szweczyk
Bob Carpenter
Chandler Ross
Nils Rudi
Fredrik Ronquist
Jakob Torgander
Aleksi Lahtinen
Ville Laitenen
Zeno Romero
Soham Mukherjee
Steve Bronder and Brian Ward
John Ashley Burgoyne

Lots of great stuff here. Check out the titles of the talks—all sorts of different topics.

Time to start planning for StanCon 2027.

It would be great to have 35% more citations and over 5 times as many downloads, but I’d rather have 2,740 Jamaican beef patties.

P.S. Here’s the article in question: The ladder of abstraction in statistical graphics. I absolutely love this paper. It’s based on an idea I’ve had for awhile that I spoke on a few years ago in Ron Yurko’s statistical graphics class at CMU. Then I wrote it up and submitted it to the journal, which gave some useful comments, and I enlisted Kaiser Fung to get it over the finish line. Enjoy.

Sol Hsiang writes:

We have a comment coming out in Nature next week that is going to cause the retraction of a high-profile paper by Kotz et al. from last year (the second most cited climate paper in the news in 2024).

Basically, Kotz et al claimed that climate change was already costing the world economy a huge amount and would cost 300% of what prior estimates claimed (which was already large). This result got enormous attention in Europe, in particular.

We couldn’t reproduce their findings and realized that it was all driven by weird data from Uzbekistan. If you remove Uzbekistan from their data set, the result falls apart. The costs are still large, but not the extreme numbers that made headlines around the world.

One reason we think this is important is because these data were previously being used by central banks around the world to run stress tests for the effects of climate change.

Here’s the retraction note, in full:

The authors have retracted this paper for the following reasons: post-publication, the results were found to be sensitive to the removal of one country, Uzbekistan, where inaccuracies were noted in the underlying economic data for the period 1995–1999. Furthermore, spatial auto-correlation was argued to be relevant for the uncertainty ranges. The authors corrected the data from Uzbekistan for 1995–1999 and controlled for data source transitions and higher-order trends as present in the Uzbekistan data. They also accounted for spatial auto-correlation. These changes led to discrepancies in the estimates for climate damages by mid-century, with an increased uncertainty range (from 11–29% to 6–31%) and a lower probability of damages diverging across emission scenarios by 2050 (from 99% to 90%).

The authors acknowledge that these changes are too substantial for a correction, leading to the retraction of the paper. An updated version of the paper with these changes, which has yet to undergo peer review, is publicly available with continued open access to its data and methodology (https://doi.org/10.5281/zenodo.15984134). The authors intend to submit a revised version of the paper for peer review. If and when published, this retraction note will be updated to include a link to the new publication. The authors appreciate the corrective role of the global scientific community and thank Thomas Bearpark, Dylan Hogan, Solomon Hsiang and Christof Schötz for bringing these issues to their attention. All authors agree to this retraction.

Good for them. And here’s the story in Retraction Watch.

How science advances when data and methods are open

Jonathan Falk independently pointed me to this story and wrote:

Imagine how uphill it would have been without access to the original data/methods.

Good point!

He also pointed to this news article which summarized the story:

If Uzbekistan were excluded . . . the damages would look similar to earlier research. Instead of a 62 percent decline in economic output by 2100 in a world where carbon emissions continue unabated, global output would be reduced by 23 percent. . . .

Wait—the estimate declines by almost a factor of 3 after removing just one data point? Uzbekistan’s not a tiny country but it’s not huge either (population 40 million); it doesn’t seem like its data should have so much influence as all that.

I went back to the original paper and it has some scatterplots, but (a) it’s hard to see that any one point would be so influential, and (b) the countries aren’t labeled so I don’t see which one is Uzbekistan.

A question of influence

What happened with the data? Is there some sort of scatterplot that would’ve indicated a concern?

To put it another way, if the data from a single medium-sized country could have that much of an impact on the findings, that would’ve been worth reporting from the get-go in the original paper. Even had there not been any data problems, we’d want to know that the results were so sensitive to one data point.

So the meta-question is: What data analysis should’ve been done originally, either to flag the problem with Uzbekistan’s data, or at least to reveal the extreme sensitivity of the headline results to that one data point?

I posed this question to Hsiang, who responded:

We noticed this issue because we were looking at several papers and running some basic diagnostics on all of them. One thing we were doing was just dropping one country at a time and rerunning the models to make sure things weren’t being driving by a single country. We were surprised that this turned up. There are many issues with this paper conceptually, but it’s not even really possible to discuss any of them until you deal with the UZB issue. We had a lot of dialogue with the authors, and it turned out that they really hadn’t run much quality control on the more granular data. When we traced back this issue, it seemed like their research assistants had faithfully converted some numbers from a PDF document, but those numbers were just implausible.

There is a scatterplot in their data paper that is supposed to provide technical validation of their data set. We wanted to see why Uzbekistan didn’t jump out, so we reproduced it in our comment (Extended Data Fig 1). It turns out that Uzbekistan wasn’t even the biggest outlier, but that the version they had published had the axes cropped so you couldn’t see the outliers (see red boxes in our version). This seemed indicative of a different issue, which is why we documented it in the comment.

I guess those graphs should be on the log scale?

It still seems crazy that the data from a single mid-sized country could have such a big effect of a global estimate. That’s something that the original researchers should’ve been aware of, and what it suggests to me is that there is a larger methodological problem that this didn’t get looked at automatically during the research process.

This post is by Lizzie. The photo is from this summer and included for no other reason than because I like a photo with a post. 

A few months ago I wrote a post where I compared the toxic culture of the restaurant industry to the toxic culture of some parts of my academic world in ecology. One point of similarity was how you often need to ‘volunteer’ your time to get a foot in the prestigious door. This led to a query by Phil about why I am against volunteering. Thanks to Phil for asking the query, and others for already effectively giving my general answer, but I will write a full post here since I think it is a good question.

So, why I am so against volunteering in academia? And here I am focusing on my part of academia, which is ecology. In short: I find volunteering in ecology in my world to be an exclusionary practice practiced by those who talk endlessly about inclusion. And, as is often the case in my life, this sort of hypocrisy drives me nuts and I can just never get over it.

And for the longer version….

In my field (ecology) and related fields (evolution, conservation), there is a pervasive assumption that it is a-okay to hire/have ‘volunteers’ to get your work done. I say hire, because these positions are advertised all the time with a list of qualifications you’ll need. Here’s one:

• BSc degree in a wildlife, environmental or conservation topic or in the process of completing one.
• Intermediate level in English and Spanish (Oral and Written).
• Knowledge in wildlife monitoring surveys. Previous research experience in any capacity is a plus.
• Physically fit and able to work long hours in a difficult and harsh environment.
• Good team member with excellent communication skills; able to live and work with a multicultural team.
• Able to live in basic living conditions and tropical rainforest conservation campus.
• Hard working and passionate with a desire to learn and improve; willing to put the hard work in to go the extra mile for conservation efforts and personal career development.
• Excellent computer skills with a confidence in all Microsoft and Google programs.

Conservation organizations or conservation-related research positions seem to feel especially allowed to do this, with the argument that their mission somehow absolves them of basic labor laws. The ones that maybe annoy me the most are the ones for ornithology work (that’s a fancy word for the study of birds) where you also need to pay your way to some (often tropical) locale and maybe even your housing and food so you can help slosh around in the jungle and search for birds. If you don’t believe me, most days you can set the Job Type to Volunteer/Training and search for ‘bird’ here and you’ll likely find something like this somewhere:

This is an unpaid position and interns are responsible to cover food and accommodation at the biological station [in Peru/Belize/etc.].

I learned about all this when I was on a ‘field crew’ long ago (that’s a term in ecology for a lot of people working together on some project where they go outside every day to collect data from the natural world). I was doing my PhD on bugs, but everyone else was studying the birds that eat the bugs and they explained to me that usually to get a paying bird-crew job you must first pay your way (housing etc. wherever the field crew is based) to be trained in point counts (standing in one place for a set amount of time and listing all the birds you hear) and then maybe someone pays your room and board and you learn to do ‘nest-finding’ (self-evident definition) and, as you get more and more training, you might get a small allowance (it’s an ‘allowance’ because it is generally way below minimum wage if you do out the per hour rate).

And then maybe you get into grad school to study birds. Lucky you!

And who knows how marine biology works (“Junior scientists in marine mammalogy are expected to have at least one or two unpaid research experiences to qualify for a graduate programme” according to a Nature Careers piece in 2020). Effectively, as best I can tell, the more coveted and sexy the position, the more it depends on extreme levels of unpaid work. I have never seen the data but when I look around at my ornithology and marine biology colleagues I wonder if the t-test on their socioeconomic status before they entered the ornithology/marine biology is higher than other subdisciplines within ecology.

And then the exact same disciples publish articles about how we need more diversity in the sciences. It blows my mind.

For decades people have been pointing out the problem:

Whitaker 2003 The Use of Full-Time Volunteers and Interns by Natural-Resource Professionals (Conservation Biology, Vol. 17, No. 1) explained that most often

full-time [unpaid] positions are filled by aspiring natural-resource professionals (e.g., students or recent graduates) who need workplace experience if they are to advance to graduate school or more lucrative jobs. Consequently, employers often consider work experience adequate compensation for wage shortfalls.

… and continues:

I believe that our widespread use of volunteers and interns to compensate for budget shortfalls does our profession more harm than good. In many cases this approach is in conflict with labor law, hinders the development of new professionals, undermines our profession’s credibility, and is an impediment to achieving our conservation goals.

He then argues that they these positions cause personal hardship, exclude certain groups, and does not meet societal norms:

Our perception of the importance and urgency of our work as conservationists does not elevate us above the societal values that led to [labor laws].’), undervalues conservation and those working in this and related areas.

Our current strategy of cutting wages does not cut costs; rather, it transfers them directly to the lowest tier of professionals in the form of financial and emotional hardship.

Despite this article from 20+ years ago my discipline continues to do this, now at the same time they decry the need for greater diversity in the field. Most articles on how to increase diversity preach for wider acceptance, statements of inclusivity, but not much action on this practice. That said, some articles acknowledge this as a problem we should maybe change if we want to increase diversity. They all cite Fournier & Bond 2015 Volunteer Field Technicians Are Bad for Wildlife Ecology (Wildlife Society Bulletin 39(4):819–821, which is good but I think the fact that they don’t cite Whitaker means they did not try very hard to look into this issue), which to me gives the 3 reasons folks usually give for not paying people who work for them:

1. I don’t have enough money to pay them.
2. It’s what has always happened and …
3. pointing out other worse-treated/compensated workers.

Fournier & Bond 2015 also makes the general argument of why we should not have unpaid interns, with the Biden quote:

Don’t tell me what you value. Show me your budget, and I’ll tell you what you value.

I find (1) pretty common, but (2) is also pervasive. I have tried talking to my colleagues who are very well funded about this and they tell me that this as just ‘how science is done.’ They often explain to me that a student gets a letter of reference from them out of it, so what’s the problem?

I think once it’s engrained that you can get free grunt work, it’s hard to get people to give it up.

And folks are trained in it early. Graduate students are often sent out to get others to help with their grunt work. I see posts on ECOLOG and flyers in my hallway each year explaining who will be needed for a full-time unpaid position to collect soils, or leaf samples or pipette all day long.

What do I think everyone should do? I suggest they start by what I have managed to do: I just don’t allow ‘volunteers’ in my lab and I tell everyone why I don’t allow volunteers. I pay everyone who works in my lab (including undergraduates) and I make sure part of their paid word is training (how to use Git R, sometimes Stan). Is it expensive to pay every undergrad in my lab minimum wage (or better) given my super low budget each year? Hell yes. But that is not an excuse to me — if you cannot afford to get the work done without volunteers, then you cannot afford to get the work done.

Everyone is clearly not going to just do this though, so we could all benefit from some top-down help I suspect. Granting agencies could ask about how much a lab relies on volunteers and discourage it through various mechanisms. Instead, they often encourage it. In Canada, NSERC grades me on how many students I have ‘trained’ (AKA: have passed through my lab) so finding undergraduate volunteers to stock my lab would certainly help the numbers game. Further, NSERC cares about how much you have volunteered in MSc/PhD fellowships, effectively encouraging students to get on board with this practice.

I think NSERC means to support short-term volunteering, which gets back to Phil’s question of ‘when is this okay?’ In part he seems to ask if it is okay to take a minimum wage job on a rec crew or such. To which I say, of course! We’re all welcome to take low-paying jobs if we can get them if you ask me, especially as they are often the gateway to a career change and I support career changes. So the question is more: when is okay to work for free? I work on community-science projects, such as the USA-NPN, where data are entirely collected by volunteers and I see how volunteering in this way or in other ways that help you connect with something beyond your job, give back to community or provide aid are important for a multitude of reasons.

So my focus here is really on full-time or similar positions. Whitaker 2003 makes the distinction between ‘part-time or short-term service’ where you do not have to forego outside opportunities for paid work vs. “other volunteer and internship positions require that individuals live in remote areas and work>=40 hours per week, effectively denying them the opportunity to earn outside wages” and I think this is very good perspective — as it can be extended also cover the undergraduates working in my lab 10 hours/week during term (they do not forego getting part-time wages, which many students need to get through university).

But we should still be thoughtful of the diversity problems volunteering often leads to. Community scientists are often retired and in a good socioeconomic position, and this extends through many other places. If the National Parks Service relies on volunteers to help repair trails etc., I am sure they will have a bias in who signs up for those opportunities and that may not be the best thing for the NPS.

And while I am back on my diversity angle, if you are thinking it is okay for granting agencies like NSERC to ask about volunteering with something along the lines of ‘but don’t worry! If your life is too hard to volunteer, you can just explain that now and we’ll value it,’ then try harder.

Try harder to think through what type of person is the one who will know they just need to profligate themselves and tell all their horrible life problems so the nice middle- and upper-class people on the committee can give them credit for their difficulties. Maybe you get a small slice of people who that works for, but more I think you get several predictable outcomes. You get people lying about it. And you get people who are horrified and scarred by having to do it (IMHO it’s a revolting ask if you back up and think about), or just don’t do it. So while I am on the topic of suggesting we all stop asking people to volunteer to do our work, we should also stop asking people to tell us how hard their lives were.

I came across a firm that seems to be offering a miracle product – https://letsmoderate.com/products/sugar-slayer . Their claim is that ingesting it hammers your post-meal blood sugar spike down by 40%, and they have multiple studies to back it up: https://letsmoderate.com/pages/science

What stood out to me as weird:

(1) These three articles were published at different times by different people but follow the same format for the abstract and text and arrive at nearly the same very large effect size for the same product (the one being sold): https://www.mdpi.com/2072-6643/16/14/2237 , https://link.springer.com/article/10.1186/s41110-024-00275-6 , https://link.springer.com/article/10.1186/s41110-024-00294-3

(2) The other listed articles seem to have been fetched by doing a search for ‘mulberry extract’, which is the critical active ingredient. Whatever I can access does show a notable attenuation on blood glucose and studies exist all the way from 2007 to 2022, thought the impact varies a lot.

(3) These 3 studies are eerily similar to the first 3 I’ve mentioned here: https://www.liebertpub.com/doi/10.1089/jmf.2014.3160, https://link.springer.com/article/10.1186/s12986-021-00571-2, https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0172239#authcontrib . But they’re from 2007, 2015 and 2024, across 3 sets of researchers, which just seems very odd. Is this how everyone is meant to name their papers? i.e. is there a social pressure sitting outside these groups pushing them into common convention?

I’m sure you’d identify more elements if you took a look at it. I’m just unclear if I’m jumping at shadows.

I was curious so I clicked on the link. From the letsmoderate site:

Here are the titles of the first three articles listed in the above email:

A Novel Nutraceutical Supplement Lowers Postprandial Glucose and Insulin Levels upon a Carbohydrate-Rich Meal or Sucrose Drink Intake in Healthy Individuals—A Randomized, Placebo-Controlled, Crossover Feeding Study

A novel nutritional supplement reduces postprandial glucose response in healthy individuals in a randomised, placebo-controlled, crossover clinical study

A ready-to-mix nutraceutical supplement (GLUBLOC) lowers postprandial blood glucose levels in healthy individuals — A randomised, placebo-controlled, crossover study

All the above authors are from India but with no overlap in the author list.

And here are the titles for the other three:

Mulberry Leaf Extract Improves Postprandial Glucose Response in Prediabetic Subjects: A Randomized, Double-Blind Placebo-Controlled Trial

Mulberry leaf extract improves glycaemic response and insulaemic response to sucrose in healthy subjects: results of a randomized, double blind, placebo-controlled study

Mulberry-extract improves glucose tolerance and decreases insulin concentrations in normoglycaemic adults: Results of a randomised double-blind placebo-controlled study

The first one’s from Korea, and the second and third are from England, with some overlap in the author list.

I agree that it’s weird that the papers have nearly identical trials. But I don’t know how things go in the medical literature. Maybe this is standard practice?

I’m not planning to fork over 799 rupees for 30 tablets of Sugar Slayer. One of the article says, “Alkaloid- and polyphenol-rich white mulberry leaf and apple peel extracts have been shown to have potential glucose-lowering effects, benefitting the control of postprandial blood glucose levels,” so maybe I’ll just eat an apple.

I am writing to share a paper which is a re-examination of my 2018 AJPS article entitled “Targeting Ordinary Voters or Political Elites”  which was previously discussed on this blog here.

The paper, written in the spirit of Gelman (2022), conducts a thorough re-analysis of my earlier work in light of recent developments in regression discontinuity designs (RDD). It also directly addresses specific critiques of the article raised both in subsequent academic literature and in previous comments on this blog.

A full response to each critique raised on this blog appears in Section A.3 on page 48 of the paper. Among other things, the blog critiqued the use of the global fourth order polynomial, and commenters highlighted that it appeared to be picking up noise in the data rather than a true relationship. While the original article did present results in the SI showing robustness to a local-linear specification with alternative bandwidths, the currentpaper significantly extends these checks and presents new results showing:

• Polynomial & Bandwidth Robustness: Shows stability across lower-order polynomials, alternative bandwidths, kernel choices, and a donut-hole approach.

• Noise Reduction via Aggregation: Aggregates data to the level of the running variable to reduce noise, presenting new scatter plots where the visual discontinuity persists at this level of aggregation.

• Spatial Structure: Demonstrates that reported balance/placebo issues in recent re-analyses stem from ignoring within-constituency clustering and predictive controls.

A link to the paper is here (https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7272598) and the abstract is below:

This note reexamines Thomas (2018), which advances and tests the ‘elite cooperation logic’ whereby national politicians target resources along partisan lines to win over the cooperation of co-partisan state legislators in implementing development projects. Consistent with this logic, a close election regression discontinuity design (RDD) uses project-level data to show that national legislators in North India allocate systematically higher public works expenditures to constituencies of co-partisan state legislators in the period after a state election. Responding to critiques in subsequent literature of the RDD approach used, this note shows that both the evidence of covariate imbalance reported in Bicalho et al. (2026) and the high proportion of significant placebo estimates reported in Albada (2025) are artefacts of ignoring within-constituency clustering and omitting predictive controls. Meanwhile, this note presents re-analyses confirming that the core findings in Thomas (2018) are robust to covariate inclusion using either constituency-level clustering or constituency-level aggregation. Acknowledging the problems related to global fourth order polynomials (Gelman and Imbens, 2019; Albada, 2025), the results in Thomas (2018) are also shown to be robust to lower-order polynomials, alternative bandwidths, alternative kernel choices, the donut hole approach, and an inference procedure that adjusts for worst-case bias (Stommes et al., 2023). The note re-establishes the credibility of the substantive findings in Thomas (2018) and highlights the importance of accounting for spatial clustering in both estimation and diagnostic checks in RDDs.

Anjali took my Bayesian statistics class back in 2006! It’s great to see what former students are doing, and I love seeing this sort of asynchronous collaboration.

Regarding the regression discontinuity issues, I do not think it makes sense to fit unrelated curves on the two sides of the cutoff. So I prefer the versions that fit a single curve plus discontinuity, not two curves.

The other thing I always recommend (see this recent paper with Imbens is that regressions include not just the forcing variable but also other pre-treatment predictors. This should help with both bias and efficiency.

All the discussion of bandwidth, functional forms, etc., can be wasted if the fitted model makes no sense or if it’s a distraction from including additional pre-treatment predictors.

In any case, it’s great to see this sort of open exploration.

They pointed to work by Max Goplerud, Shiro Kuriwaki, Jens Wiederspohn, Adam Conner-Sax, and Philip Greengard, which was recently presented as poster at polmeth: Modeling Complex Contingency Tables.

I wish I got to attend, this is cool ! I’ve got questions about Application 1 (ACS):

1. What are the covariates w?
2. Suppose you also have survey data in Palo Alto. Would this be added to the training tables ?
3. How is the table reconstruction error random ? (I see a density plot.)

The poster gives an answer to Andrew’s question in “Mister P when you don’t have the full poststratification table, you only have margins” and “The continuing challenge of poststratification when we don’t have full joint data on the population”:

I’d recommend first imputing a full poststrat table … But then the question is how to do this.

In Application 2, they ask if a better poststratification table reduces error in estimating the outcome using Multilevel Regression and Poststratification (MRP). Presumably this depends on the distribution of the outcome given the poststratification variables. In “toy example for energy balancing weights” I wrote that raking does well “when Y | X1, X2 is additive”.

I’m excited to read the paper and learn more !

I was talking with Gustavo just the other day about Red State Blue State, and how that work was motivated by confusion following the 2000 election emanating from pundits of the left, right, and center. Back then I felt the compulsion to write a whole damn book to explain what was really going on. I even came up with an entirely new (to the best of my knowledge) concept, “second-order availability bias,” to explain how the journalists could’ve gotten things so wrong.

The concept of “second-order availability bias” never caught on, to say the least: it appears only once in the easily-accessed published literature:

So maybe it’s not such a useful psychological concept. What’s relevant here, though, is that the pundits were getting it so wrong, and with such a consensus, that I felt the need to refute them.

Nowadays, things are different. There aren’t so many all-purpose pundits like David Brooks—people who know essentially nothing and have no real interest in learning but present themselves as infallible experts—, and those who remain don’t have such a platform. Also, with partisan polarization, commentary has become fragmented, and we rarely see much of a consensus among pundits across the political polarization. It’s just not on the table.

Meanwhile, political analytics has become more and more impressive, with important contributions being made by academics, journalists, and political professionals. There’s still disagreement (as here) and some difficulties of communication (as here), but in the past two decades the level has gone up so much: the best analytics has become much more impressive, and what might be called replacement-level analytics has become much better too. I don’t think my own sophistication has increased much at all, and that’s one reason why I’m now less likely to crunch the numbers myself (as I did in the early morning hours of 5 Nov 2008) and more likely to just link to the analyses of others (as with Yair’s report on 2024).

Just today I came across two excellent examples online from journalist colleagues of mine.

Elliott Morris, “I re-analyzed the raw data from Wisconsin’s primary polls. Here’s what actually went wrong,” which features this split-the-difference summary that warms my Bayesian heart:
• Most of the miss in polls in Wisconsin is attributable to faulty demographic targets (too many young people). This inflated Hong’s vote margin by somewhere between 5 and 10 points.
• My best guess is that the race moved 6-10 points toward Crowley after pollsters released their final surveys.
• Non-ignorable non-response within demographic categories likely further inflated Hong’s vote margin by 2-5 points.
Morris goes through lots of details too. I haven’t tried to check any of this, but it seems reasonable. We’ve been saying for a long time that primary elections are hard to predict, but some polls are off by much worse than others, and it’s instructive to look into exactly how this can happen.

Beyond the details and the direct interest of this post to political organizations and pollsters, I appreciate Elliott’s work here because he goes beyond statistical generalities (“regression to the mean,” “sometimes you get a draw from the tail of the distribution,” etc.). This is an important statistical point: the “error term” is only an error term until you drill down, look at more data, and figure out what is going on. It’s so common for researchers to just take their numbers and not think about where they came from (as here)—and, indeed, academics and pundits alike can be rewarded for that sort of asinine don’t-look-carefully-at-the-data attitude. So it’s good to see Elliott demonstrating how it’s possible to do better—if you’re willing to put in the work.

Nate Silver and Eli Mckown-Dawson, “FLIPR 2026 midterm election forecast,” which leads Nate to summarize that “[Michigan Senate candidate] El-Sayed would be an underdog in an election held today and is an underdog in our “Lite” (polls-only) version. The fancy versions look at the fundamentals and are more convinced he’ll come back.”

What I really like about this is how “workflow” it feels. What I’m talking about here is how they fit two different models that are doing two different things, they learn something from the comparison, and then they track this back to their data. This sort of thing isn’t in the textbooks (well, it wasn’t until now) but it’s so important to good applied statistical work. So I love to see it here.

The point about these two posts, one by Morris and one by Silver and Mckown-Dawson, is not that they’re so amazing. I mean, yeah, they’re great, but the real point is how professional they are.

I remember Bill James once wrote, in reaction to the unexpected playoff heroics of Bucky Dent or Ray Knight or somebody like that, that, sure, it’s cool when someone steps up and does the unexpected, but what’s more impressive are those Eddie Murray types who can consistently deliver the expected. Because then you can design a game plan around them and not just have to hope for a miracle.

Morris, Silver, Mckown-Dawson, and others doing what’s now expected, doing it well, and demonstrating modern principles of statistical workflow . . . That’s impressive.

Regular blog readers will remember Golden Krust, source of our standard unit of currency. (Sorry, Peter Thiel, we don’t take bitcoin.) The above picture is kinda blurry but it gives you a sense of the neighborhood.

But here’s the big news! A new Jamaican beef patty place has appeared, right across the street:

Zooming in, it looks like it’s opening in two days:

I’ll be back soon with a full report (but nothing more on this story).

P.S. We did a taste test! The story is here.

The idea is that, instead of structuring this based on the methods of maximum likelihood estimation, null hypothesis significance testing, and confidence intervals, you start with the goals of estimating parameters, making predictions, comparing and evaluating models (i.e., alternative explanations of the world), and summarizing and working with uncertainty.

If you’re interested in using Bayesian methods, we have two books (Bayesian Data Analysis and Bayesian Workflow) on the topic. But with these methods under our belt, we can now go back to the motivating questions–the more general statistical goals, which exist without reference to any particular models or any particular Bayesian or frequentist methods–and consider them from scratch.

This seems important.

My discussion here is motivated by this question sent in by Opher Donchin:

Here’s one for your blog (although I’m happy to have your take as well).

I’m transitioning our undergrad biomedical engineering course this year from a standard frequentist syllabus to a Bayesian approach. We are mostly following the first 6 chapter of Bayesian Analysis with Python by Osvaldo Martin. In order to get the department to agree to this, I had to promise to teach basic frequentist methods.

Thus, I need to teach frequentist methods to students with a strong Bayesian background. There doesn’t seem to be much available on this. There is of course material comparing the two approaches, but I mean specific material designed to explain the frequentist approach to someone who knows the Bayesian approach.

I’d love to know if anyone knows of resources or has experience of insight or advice.

I replied that I’ll see what the blog commenters suggest, but in the meantime, I recommend chapter 4 of Bayesian Data Analysis as a start.

Donchin responded:

Yes. Chapter 4 is very good on the principles involved. Much of it addresses the Bayesian alternatives to frequentist procedures or the Bayesian perspective on them.

I’m wondering about something more concrete, aimed at a less sophisticated audience. That is, my students will know how to build models, how to interpret the posterior samples, the basics of Bayesian workflow and also model comparison. On the other hand, they will have no knowledge of confidence intervals, maximum likelihood estimates, hypothesis testing, or multiple comparison procedures. Reasonably, my department demands that they be able to read the biomedical literature where such terms are widespread.

I want to give them an understanding of frequentist procedures without getting bogged down in frequentist justifications.

To take an example, I want to explain what an F test calculates when understood within a Bayesian framework. To that end, I can show students a Bayesian model of a normal distribution of group means and a normal likelihood within each group. Then, I can work through what a Bayesian would need to calculate on that model to produce an F statistic and an F test. It has something to do with summaries of posterior distributions of ratios of variances.

What I’m hoping for is somewhere where such questions are worked through in detail so that it could be used for developing lectures.

Of course, chapter 4 of BDA is 10 years old at this point. I imagine some of the ideas may have developed since then.

Ahhh, good point! Chapter 4 of BDA is for statisticians who already know the classical methods and want to understand how these can be understood in light of Bayesian principles and adapted within a Bayesian workflow. But it’s really a completely different task to explain classical methods to students who haven’t already learned them. The idea would be to retcon ideas of classical statistics from a Bayesian angle.

This would be worth doing.

In the meantime, I recommend . . . chapter 4 of Regression and Other Stories, where we go through basic principles of point estimation, hypothesis testing, and uncertainty quantification from an applied perspectives. Also, if you flip through that book, you’ll see other places where we discuss classical procedures from first principles. We don’t have any F tests or multiple comparisons adjustments because I can’t bring myself to care about those things, but a lot else is there, so you might be able to put together much of what you need from that book.

In response to that recommendation, Donchin wrote:

I agree that Chapter 4 of Regression and Other Stories touches on many of the important points, but it is not sufficient for my needs.

I am trying to teach a Bayesian-first undergraduate statistics course to biomedical engineers that also gives a background in frequentist approaches allowing  them to function effectively in environments that require them to use or understand frequentist stats.

This means that the frequentist-realted topics we cover include:

• Estimation: MLE, standard errors, confidence intervals
• Hypothesis testing: p-values, Type I/II errors, power
• Proportion tests and t-tests (independent and paired)
• Effect sizes; multiple comparisons (briefly)
• Linear and multiple regression: least squares, coefficient tests, R2, F-tests
• ANOVA: categorical predictors, interactions, sums of squares, effect sizes
• Pearson correlation and inference
• Repeated-measures and mixed-effects models
• Model comparison: AIC and cross-validation
• Replication crisis / open science / pre-registration (not exactly frequentist, but still)

You can see the syllabus and the lectures in the student-facing version of the course repo at:https://github.com/opherdonchin/StatisticsCourse_36714361

Any further thoughts you might have would be great to hear.

Also, I would be happy to get this some visibility, in hopes that other people would be interested in providing feedback, using some of the material, or just doing it better.

I don’t think I could bring myself to teach a lot of the above topics, except in an “inoculation” sort of way, but I recognize that many students will need it, so if anyone has some good suggestions for Donchin, just leave them here in the comments!

1. People want improvement. If you’re at X, then it’s great to go to 1.2X, it’s kinda scary to stay at X, and it’s distressing to go to 0.9X. It does seem that people react to the change in their economic situation rather than than the absolute level.

2. People want more. Housing is more expensive than it used to be, but part of that is that people want to live in bigger apartments and bigger houses. Cars are much more reliable than ever, but nowadays lots of people want two or three cars in their household.

3. Entitlement. Some people are struggling and it makes sense for them to be distressed with the economy. On the other hand, lots of people are sitting there with expensive houses all paid for and some money in the bank, but they’re not saying how amazingly wonderful things are, because they feel that they deserve every bit of it. This averages out to unhappy in the population.

4. Lack of security. The concern that, things might be ok now, but you might have to be scrambling in the future.

This is not to deny that lots of people are struggling economically. I’m just talking here about average poll findings. The question is not why some or even many people are economically miserable, so much as why the average response to such questions is lower than in earlier decades when people had a lot less. The point is that people are thinking about their future, they’re not comparing to how things were in past decades.

French gives some examples of how the competitive economy has distorted the upper middle class. He even talks about youth sports travel teams. It’s hard for me to believe that the high cost of Disney tickets and travel team fees are driving the affordability crisis–neither of these is anything like the struggle to pay the rent and take care of kids or elderly relatives while commuting to some faraway job with rigid working conditions. French also talks about the frustration of flying, but most people get on planes only rarely.

One other thing. French writes:

In this story, maybe the problem isn’t oligarchy. Elon Musk’s billions don’t tangibly change my life.

But . . . Elon Musk’s billions do tangibly change your life! Remember, they went in and screwed up the government. Planes are crashing, people are getting shot on the street, your data are at risk, they’re reducing childhood vaccinations and making it more difficult to develop future adult vaccines . . . so, yeah, you can blame those billions. You can also blame the zillions of dollars spent by lobbyists to legalize gambling on people’s phones.

I agree that “oligarchy” is not the only problem–and, arguably, oligarchy does lots of good things too—but it’s naive to think that it’s not changing your life. Even if you are not personally a gambling addict, you might have a loved one who is. Even if the kids in your family are currently vaccinated, you or they could get killed by the next pandemic. Etc.

With some sustained effort, I cleaned out my entire inbox! Now I’m ready to do research again.

Actually, cleaning out the inbox entailed doing a lot of research and writing, lots of interesting things I’d put off for a while.

One byproduct of all this work is that our blog queue is now a year long. That’s right—the new posts are being scheduled for August, 2027. Although I do reserve the right to post things earlier if it seems that they’re particularly relevant, my co-bloggers will continue to post whenever they want, and we sometimes post things early on the newsletter. So you never know what’ll be coming.

It’s a funny thing, though. I’ve been writing so much for 2027, that now when I read something with a 2026 date, my first thought is, Hey, that happened last year! I’m feeling like Jones.

Q-collar is ramping up its fall sports campaign. Lots of buzz on Facebook, with misleading comments from the company, new ways of spinning the FDA disclaimers, and plenty of excited parents.

People are also now posting AI generated summaries which mention the research – but skip the Expressions of Concern, etc.

Here’s some examples, happy to provide more!

Wow, those chatbot-generated evasive responses (“You are absolutely right, and we never want to gloss over this”) . . . what a bunch of assholes! This is the sort of dishonesty we could expect from an outlet like Wolfram Research.

Matthew Walker, Brian Wansink, Cass Sunstein, and various other professional-avoiders-of-inconvenient-facts could lean a lot from these guys on how to pretend to respond to criticism without directly addressing it.

As is so often the case, the threat from AI is not the AI doing malevolent things on its own; it’s bad actors using AI capabilities to manipulate humans.

The next step, I guess, would be to get some official promotion from the U.S. Department of Health and Human Services, by way of some strategic campaign contributions. Maybe they could arrange some contract to put the Q collar on the updated Mount Rushmore, something like that? Just spitballin here.

P.S. Just to remember: Yes, real brains are at stake here. This is evil.

P.P.S. I can see why there’s so little reaction to this story, as this sort of little violation of human dignity is nothing compared to the official legalization of corruption, as in this story that contains this Orwellian line from the Treasury secretary: “Treasury is eliminating a burdensome reporting requirement for millions of law-abiding business owners without compromising our national security.”

My name is ** and I am a junior at ** High School. I am writing to you as a part of a summer assignment for my upcoming research class where I had to choose an expert in the math field to contact. I chose you considering your work and investment in the statistics field and how you have used statistics in the political landscape. I found it very interesting and I was wondering a few things,
• What got you into math and statistics?
• What advice would you give to students interested in statistics?
• What is the biggest challenge when working with statistics?
• What is a common mistake people make when looking at statistics?
• Where would be good resources to learn more?
I appreciate your time and I hope to hear from you soon.

Here were my responses:

• What got you into math and statistics?
See here: https://statmodeling.stat.columbia.edu/2010/11/02/fragment_of_sta/

• What advice would you give to students interested in statistics?
In addition to statistics, you should have some applied field you are interested in. This could be biology, economics, sociology, medicine, business, communications, history, . . . just about anything. Then when you learn statistical ideas, consider how they apply to this field, and work on a project applying statistics in that way.

• What is the biggest challenge when working with statistics?
Measurement; see here: https://statmodeling.stat.columbia.edu/2015/04/28/whats-important-thing-statistics-thats-not-textbooks/

• What is a common mistake people make when looking at statistics?
Looking at the numbers and not checking where they came from. For example, see here: https://statmodeling.stat.columbia.edu/2017/01/02/bogus-north-korea/

• Where would be good resources to learn more?
I recommend our blog: https://statmodeling.stat.columbia.edu/ and our book, Active Statistics, which is full of stories: https://sites.stat.columbia.edu/gelman/active-statistics/
Also I like Frederick Mosteller’s classic book from 1965, Fifty Challenging Problems in Probability. I was a teaching assistant for Mosteller for the last course he ever taught!

Since I was answering the questions anyway, I thought I’d post it all here as others might be interested too. I think that all of this is reasonable advice.

The case study was designed to provide a Bayesian replication of the results in Efron and Morris’s five decade old paper on hierarchical modeling (aka Stein’s estimator, aka population regularization, aka “empirical” Bayes), which is still under a paywall courtesy of our “friends” at the American Statistical Association. If your organization isn’t paying ASA for access to a paper that an academic donated for free 50 years ago, I’ll leave you to find your own pirated copy in good conscience, or you can follow the link and let Google hoist the Jolly Roger for you (now that’s an even more obscure and archaic reference).

• Bradley Efron and Carl Morris. 1975. Data Analysis Using Stein’s Estimator and Its Generalizations. JASA 70(350).

My case study’s been out for ten years. Mitzi found a problem when matching the data provided in the R package pscl against that in Efron and Morris’s paper. In particular, the data for the player named “Williams” was wrong. Efron and Morris manually transcribed the data from a newspaper with the goal of finding a bunch of players with the exact same number of at bats on a given day (45, it turns out). They did so accurately.

Sadly, I imported an error in the 18-item data set from the R package. Here’s an edited report from Claude after it tracked down what went wrong.

1. The 1975 paper is correct, although it only gives the surname “Williams.”

2. The R package pscl expanded that name incorrectly to Billy Williams, while retaining Walt’s team and remaining-season data. Published pscl data.

3. The Bean Machine tutorial () contains the exact mixed row found in your CSV:

     Billy,Williams,...,591,0.3299,636,205,0.251
   

4. That row combines Billy Williams’s at-bats and hits with Walt Williams’s .251 average. Thus, ID 15 in your data should be corrected throughout to:

     Walt,Williams,45,10,0.222,270,0.2556,315,79,0.251
   

But wait, that’s not all. There was another typo in the Bean Machine tutorial. They mistranscibed Ron Swoboda‘s surname as “Swaboda.” I should have caught that as I was a huge baseball fan in the early 1970s, am an excellent speller, and grew up around a lot of Poles and Czechs in Detroit and its western suburbs. Mitzi then had the following conversation with Claude after she noticed the typo. This may be the first time I’ve seen Claude crack a spontaneous joke.

Mitzi: did that [the typo] come also from the R file?

Claude: If you mean the Swaboda misspelling: no. The pscl R dataset correctly uses Ron Swoboda. The misspelling appears in the Bean Machine tutorial
(https://beanmachine.org/docs/overview/tutorials/Hierarchical_modeling/HierarchicalModeling/), which contains Ron,Swaboda.

Mitzi: are you sure? I’ve heard it both ways.

Claude: Yes—despite what Shawn Spencer might say, it is definitively Ron Swoboda, with an o.

Mitzi: who’s Shawn Spencer? in 2017, Bob Carpenter published this dataset in a Stan case study.

Claude: Shawn Spencer is the fictional detective in the TV comedy Psych, played by James Roday Rodriguez. “I’ve heard it both ways” is his recurring response when someone corrects him—hence my reference.

Mitzi and I use “I’ve heard it both ways” all the time, having watched umpteen seasons of Psych. I would recommend if you want a lighthearted detective story that doesn’t take itself too seriously with lovable characters and you can put up with the ubiquitous mid-00s background sexism and racism.

This is an interesting example of science-adjacent incompetence or fraud or grifting or whatever you want to call it, as it overlays junk science with big business (according to journalist Stephanie Lee, the device costs at least $200 and “investment in the Q-collar has totaled more than $30 million”), the government (including the U.S. Army, a U.S. senator, and the Food and Drug Administration), news and social media, and some aspects of the scientific establishment (medical journals).

You can think of Q collar as a baby version of health scams promoted by wealthy and powerful medical-adjacent celebrities such as Dr. Oz, Andrew Huberman, and RFK Jr. It’s junk science, but junk science with a purpose, in this case not so much to promote an ambitious social climber (as with Agus and Arday), it seems to be more about the money. And in that way it could be seen as a more typical (if less dramatic) example of a science scam. It doesn’t involve any plagiarized giraffes or implausible claims of ultra-marathoning, just the exploitation of people’s legitimate health concerns and the lure of the filthy lucre.  It could even be that the scammers believe their own claims (remember, the first step is to fool yourself; the rest then comes easy) and believe they’re doing well be doing good.

We should probably be doing more coverage of science-adjacent medical scams like the “Q collar” and less on the repulsively self-promoting but ultimately less harmful Aguses, Ardays, Wansinks, Hausers, and Arielys of the world who each in his way degrades the reputation of science but isn’t so closely locked in to a moneymaking scheme. (OK. I guess Wansink did ok as a consultant, and Agus and Ariely founded a few businesses which probably netted them a good chunk of change.  They might well be first-class wealthy, even if not in the private-jet stratum.)

Here’s a Q-collar update from science sleuth Mu Yang:

Two new articles on Q collar came out today:

• BMJ (with a nice video clip) and WaPo.
• Here are our own takes on LinkedIn (mine) and Substack (Jamey’s).

The duplicated tables are almost cute compared to the real issue yet to be addressed. — I mumbled about it in a comment to your post back inJanuary.

Basically, the design is what one would use to teach “confounding factor” in Stats 101:

• 5 studies (3 on Q collar and 2 on helmet type) reported data from subjects from the same cohort.
• The subjects in the helmet studies are the controls from the Q collar studies.
• Two types of helmets were worn by Q-collar-controls, and helmet type had significant impact on brain white matter parameters
• What helmets Q collar subjects wore is unknown.
• Q collar is said to affect the same brain white matter parameters as helmet type, in the same direction (the good one)

I [Yang] wrote more about it here and posted on Pubpeer.

Things are at a standstill. The authors have been contacted many times (and I made sure that they are notified about my Pubpeer posts), but remain reticent. It seems that FDA does not care.

Awhile later, Yang added:

Due to the incurable stubborness of some of us, we have been digging deeper in the Q collar junkyard long after Stephanie’s article, your blog post, and the BMJ article.

To put it extremely simply: The control subjects in the Q collar studies were described in two studies on helmet types —which were shown to have substantial impacts on the same brain imaging metrics as Q-collar does. Given controls and Q-collar subjects were students in the same schools, Q collar subjects likely also wore different helmets. BUT, this key confounding factor was never mentioned, quantified, controlled for, or discussed in Q collar papers. I commented on this here. Following that, Dr. M. Shane Tutwiler, a quantitative researcher in the field of educational psychology in the University of Rhode Island, became interested in the case, and wrote a very comprehensive comment on Pubpeer. Shane’s detailed analysis argues that the design and analysis of one of the key studies are grossly inadequate to support the claims of the study—which was crucial for the FDA clearance.

Before posting on PubPeer, Shane spoke to the authors directly to alert them of his concerns and request the data—the 2021 article has a data sharing agreement—but they denied his request because he was critical of their work.

This “silly” and “messy” case is looking much worse than previously thought. However, Dr. Smoliga and I are being stonewalled by SAGE — the six papers under EoC never moved forward. Dr. Tutwiler’s reports to Wiley are also not getting meaningful replies.

We are wondering if you are willing to take a look at the new thread of evidence and weigh in.

My quick answer is that the story seems clear enough already, even without this new thread! I also noticed that one of the authors of one of the papers under discussion is certain “Jeffery N Epstein.” Not the same guy, and I’m guessing that my Columbia colleague Richard “Axel” Foley wouldn’t spend any time responding to his emails.

But I digress.

To return to this horrible “Q collar” story, Shane Tutwiler writes:

I just wanted to close the loop, as it were, on the “Investigation” of the paper I evaluated on PubPeer. The authors shared the data with the editors, and the editors were seemingly content that the paper was perfectly fine as-is (despite many of the issues I pointed out being raised in peer review and ignored by the authors and managing editor at the time). Wiley has closed the case (see below).

If this were a run of the mill journal article that nobody was going to read, I wouldn’t feel so upset. But this research was used by the FDA as part of the evidence chain to justify the device’s use in public. It’s one of those cases where questionable analyses can have real world consequences.

Without any additional knowledge of this particular case, just based on my general understanding of such things, my guess is that the journals’ decision to do nothing here is motivated less by corruption and more by simple laziness. If you retract an article, the authors can scream at you. If you do nothing, people like Tutwiler and me will scream at you . . . but people like us are much less dangerous than authors who refuse to let go of bad research: Those people have money and reputations on the line and can lash out or even sue! So safer to just do nothing and pretend that all is well. The turtle strategy was not so effective recently with Cambridge University, but usually it works just fine, so I guess I can understand why the editors are doing this, even though I think it’s unethical behavior on their part.

And James Smoliga adds some additional context:

The investigation we raised with the Journal of Neurotrauma has been ongoing for over two years now. In the 10 months since the Expressions of Concern were issued, no further action has taken place.

Those six papers are separate from the one that Shane flagged to Wiley… And importantly, that study was the key one that resulted in the FDA’s authorization of the product, which further generated millions in venture capital investment, DoD funding, and consumer purchasing.

This is a mess at every level – improper statistics, inappropriate interpretation, sloppy errors, and allegations of p-hacking (an admission from a collaborator on their research, who doesn’t seem excited to publicly state this).

If you have any interest in writing more about it, or know of any journalists who may be interested in covering this story further, any help is appreciated!

I know lots of journalists, but they usually don’t like covering this sort of story. The outlet most likely to cover it might be Defector, but I don’t know anyone there. Actually, it would fit wonderfully into their Only If You Get Caught podcast. Maybe I can do the contacts-of-contacts thing and see if I can reach anyone there to make the suggestion.

The New York Times already had this article in 2022 that was critical of the Q collar, so I don’t see them running more stories on it. Once you get to the point of junk-science-is-refuted-but-still-keeps-on-making-money, you’re entering “dog bites man” territory, no?

Maybe you could convince noted science skeptics Sean Carroll and Steven Levitt to cover this story . . . ha ha, just kidding! More seriously, maybe If Books Could Kill would be interested, but I don’t know them either.

P.S. More here.

OK, this one’s time constrained so I’ll post it right away, not on the usual lag.

Christian Robert points to this conference announcement:

We invite proposals for the BayesComp 2027 mirror event in Aussois, France, that will take place on 17-21 May, 2027. The event will complement the main BayesComp 2027 meeting in Texas and provide a European forum for researchers in Bayesian computation and related areas. Plenary sessions will be broadcasted from Texas, while talks in Aussois will be recorded.

Please submit one form per proposed contribution. Proposals may be for contributed talks, or poster presentations. The programme committee will review submissions based on scientific quality, timeliness, coherence, and expected appeal to the BayesComp community. Unless the submission mentions otherwise, talk proposals that are rejected will be automatically added as posters.

Details about the venue (CAES Paul Langevin), potential registration fees, and technical details are being finalized and will be announced on the event website.

Looks like fun! And Bayesian computation is important.

It’s interesting to see here what is and isn’t included.

1. The section on Formal Power Groups contains one chapter on business, one on agriculture, and two on labor. Organized labor is so much less now that I don’t think it would merit even one chapter.

2. There’s a chapter on “negroes.” I guess makes sense from the standpoint on 1965 that there are no chapters on other social outgroups such as Latinos, women, gays, involuntary celibates, immigrants, disabled people, etc.

3. There’s no chapter at all on organized religion, which is a notable missing power group now but even more so for the 1960s.

4. There’s a chapter on “intellectuals” but nothing on political ideology.

5. There is no chapter regarding the organized Left, but I guess it’s covered in the penumbra of “labor” and “intellectuals.”

6. There’s no chapter regarding the organized Right, and that’s more of a loss. The organized Right has been very influential in American politics and policy ever since the late 1970s, but it was a big deal even as of 1965. This book does have one section on “the white reaction” in the “Negroes” chapter, but that’s hardly enough to cover it.

7. There’s no chapter on the ultra-rich, which in 1965 would pretty much fall in the “business” category but now surely deserves its own chapter.

8. There’s no chapter on science and technology. The book is called The Makers of Public Policy, and my point here is that scientists and technologists make public policy even without (or in addition to) traditional lobbying, on issues ranging from vaccines and DNA testing to social media to climate change.

So I think it’s instructive to look at the topics covered in this sixty-year-old book, partly to see what they missed even then and partly to reflect on how much things have changed.

Also, this won’t be news to many of you but it’s still interesting to see:

And here’s the first page of the index:

Lots has changed in what’s considered important.

Also relevant to the question of how things have changed: What rule did Louis Menand use to select what went into his book on U.S. “cold war” culture of 1945-1965?

Chapter 19 “Building up to a hierarchical model: Coronavirus testing” is a case study about a 2020 survey that tested n = 3330 residents of Santa Clara County, California for SARS-CoV-2 antibodies (Bendavid et al. 2020a, 2020b). y = 50 people tested positive.

To estimate population prevalence, they want to account for measurement error in the test (“Measurement“) and differences between sample and population (“Representation“), both sides of Groves et al. Figure 2.5:

The measurement error model includes specificity gamma = P[test negative | no disease] and sensitivity delta = P[test positive | disease], which take population prevalence pi to test positivity rate p:

Bendavid et al. (2020a) analyzed their data using gamma = 0.995 and delta = 0.80. But these aren’t known exactly, so Gelman and Carpenter 2020 recommended using priors from previous studies to reflect uncertainty:

They extend these priors to account for multiple studies. They look at sensitivity to these priors (overloaded term here: “sensitivity”). Cool stuff !

In this Survey Statistics series we hadn’t yet talked about models for measurement error in the outcome variable y. We’ve seen measurement error in an adjustment variable X, e.g. recalled vote (“is a mismeasured X better than none at all ?”, “more adventures in mismeasured X”, “more on recalled vote”, “it is (still) the people”).

On the “Representation” side, Bendavid et al. (2020a, 2020b) adjusted for differences between sample and population using weights and encountered the difficulty we saw last week: adjusting for lots of variables can lead to very large weights. Maybe modeling can help (see last week’s “structured MRP to smooth survey weights”).

So Gelman and Carpenter 2020 proposed replacing (19.2) above with

y_i ~ bernoulli(p_i)
p_i = (1-gamma) (1 - pi_i) + delta pi_i

and using Multilevel Regression and Poststratification (MRP):

I thought this story sounded familiar and remembered Andrew talked about it at his Birthday workshop‘s closing talk (this part on YouTube, minute 2 to minute 8).

Andrew said that when his friend tried the method from Gelman and Carpenter 2020 it “crashed and burned” ! There is a new paper about this: Kuh, Kennedy, Chen, and Gelman 2026. They present “a statistical workflow for diagnosing unexpected results in complex models”. These authors have researched a lot about evaluating MRP models (see “should MRP workflow include LOCO-CV ?”). We will have to revisit their new paper later in this series.

But recently I thought we need to update this analogy.

Back in the early part of this century, economists were riding high: they were the country’s all-purpose pundits, they had tons of influence but were lamenting that they didn’t have enough, and they were going on and on about how special they were, most amusingly in the self-contradictory argument that they were different because they “assume everyone is fundamentally alike; we believe circumstances, not culture, drive people’s decisions.” I’m still not sure what is the difference between “circumstance” and “culture” except that maybe talking about the former is associated with overconfidence.

Nowadays, though, economics is just one more social science. OK, I don’t want to overstate things. I assume they still get paid more than sociologists and political scientists, and, yeah, there’s a Council of Economic Advisers but no Council of Sociology Advisers. Still, I think that economics has lost some of its standing in the past twenty years, partly as a result of the crash of 2008 and its aftermath (political polarization, Brexit, etc.) and partly just the natural ebb and flow of influence, the inevitable cycle of hype and disappointment. Econ hero Steven Levitt was supplanted by data analyst Nate Silver (who identifies as a poker player, not an economist), and we’re not hearing from economists so much anymore, except to hear them fighting in vain against tariffs.

There’s a new hegemonic science in town, and it’s information science, or computer science.

Computer scientists currently have a lot of prestige. Fair enough: they’ve earned it through all the amazing things they’ve built. Indeed, it’s worth comparing to the earlier alpha-dog social scientists. Freudian psychiatry and neoclassical economics were powerful, all-explaining theories that addressed people’s concerns about mental health, happiness, prosperity, and future prospects. They had big theories, which is great. (The theories were not falsifiable, but that can be fine, as the function of such all-encompassing theories is not to make predictions or to explain the world but rather to supply a framework by which the social world can be studied.) Computer science is different: they didn’t develop a theory of society, but they built impressive tools that change how we live in the world.

Now I’d say that computer science today is like economics at the beginning of the century or Freudian psychiatry in the 1950s in being at apex academic and social prestige and influence. Computer scientists, or computer-science-associated businessmen, are the new gurus, in a way that wasn’t the case before. Yes, Steve Jobs and Bill Gates were culture heroes back in the 80s and 90s, but nobody was particularly interested in what they had to say outside of their narrow technological realms. Nowadays, many computer scientists and tech lords present themselves, and are often taken as, all-purpose pundits. (Many of the tech lords are actually tech investors; they’ve absorbed the prestige of information science through the transitive property of money.) As with the economists and Freudians of past eras, they are presented as having some special authority derived in part from their almost inhuman hyper-rationality, a willingness to tell uncomfortable truths.

But then you get the same problem we had before, which is when the gurus and their hangers-on start to believe their own hype.

One of the benefits of living a long time is that you get to see the aftermath of the hype wave. Back when academic economists were on the top of the world, they were complaining that they didn’t have even more power and influence than they already did. Now that economists are just one more group—more influential than political scientists or sociologists to be sure, but no longer apex academics—they’re no longer doing that. It’s when a group is at its most overvalued that it wants even more, and we’re seeing that with the tech industry (and, to extent, their academic allies) today.

First, the recommendations.

• Here’s the book referred to in the title of the post. I highly recommend it! My only regret is that we didn’t call it Applied Regression and Causal Inference, cos that’s what it’s about.

• I also recommend our book, Active Statistics, which has literally hundreds of stories, class-participation activities, computer demonstrations, and discussion problems. A great teaching resource and also just a good read. Really!

Now, the story.

This came in the email one day:

Dear Professor Andrew Gelman,

I hope this message finds you well. I’m part of the Cambridge University Press sales support team and I work with your local Cambridge Sales Representative ** **@cambridge.org

We wanted to follow up regarding your course(s) for Fall 2025 and ensure you’ve had a chance to review Regression and Other Stories, 1 ed by Andrew Gelman/Jennifer Hill/Aki Vehtari as a potential option for your course materials.

If you’ve decided to adopt this title, I’d be happy to provide access to any available instructor resources. If you’ve chosen a different title or are still considering your options, I’d appreciate a quick update so I can ensure our records are accurate.

Also, please don’t forget to check with your bookstore about any Equitable or Inclusive Access programs that may be available. These programs can provide your students with more affordable digital access to their course materials.

I am sincerely grateful for your attention to this matter; I look forward to the opportunity to hear from you at your earliest convenience.

Cordially,

**
Sales Support Assistant
Higher Education Sales, Americas
**

On one hand, hey, I’m thrilled that they’re promoting our book. On the other hand, it seems like they’re kinda flying blind if they’re sending this promotional message to the author of the book.

That is why today I am repeating my pledge to establish a panel of top experts, working with Bobby, to investigate what is causing the decades-long increase in chronic health problems and childhood diseases, including auto-immune disorders, autism, obesity, infertility, and more.

Palko writes:

Coming at this from a conspiracy theorist/pseudoscience crank worldview, the one word that should set off immediate alarm bells is “autism.” . . . If you had to list the remaining terms in order of tinfoil hat appeal, they would be infertility, autoimmune disorders, and at a distant fourth, obesity. A reporter covering the speech might also feel compelled to add that the causes which RFK Junior has proposed in the past include not just vaccines but also chemtrails, Wi-Fi, and 5G.

Sounds about right: the candidate who endorses the ludicrous Pizzagate conspiracy theory, tried to overturn an election, and is allied with child-murder-denialist Alex Jones is teaming up with a public figure whose big political issue is opposing vaccines.

Palko continues:

Here’s how the NYT covered that quote in the third paragraph from the end of the article on Kennedy endorsing Trump:

Earlier in the day in Phoenix, at his speech announcing the suspension of his campaign, Mr. Kennedy said Mr. Trump had offered him a role in a second Trump administration, dealing with health care and food and drug policy. In Glendale, Mr. Trump said that, if elected to a second term, a panel of experts “working with Bobby” would investigate obesity rates and other chronic health issues in the United States.

I agree with Palko that this is absolutely ridiculous in the context of the actual quote from Trump. The Times is “sane-washing” Trump and Kennedy, working really hard to take dangerous and extreme positions and treat them as reasonable and normal.

I guess this doesn’t matter to political junkies for whom Kennedy Jr. is associated with anti-vaxx conspiracy theories. To many normies, though, I’m guessing that he’s perceived more as a generic Kennedy heir—maybe they don’t know that he makes Dr. Oz and Andrew Huberman look like purveyors of solid medical advice.

This seems like
another example of how junk science and conspiracy theories have moved from the periphery of our culture to the mainstream, and I agree with Palko that it was bad journalism for the Times to report on that speech and sane-wash it.

Newspapers make mistakes all the time; the reasons for posting this one are: (1) this was the first time I’d heard of “sane-washing,” and (2) it’s scary to see pseudoscience escape the narrow world of Lancet/PNAS/Ted/NPR/Gladwell/Freakonomics and potentially become government policy.

I do think it’s possible that the mainstreaming of pseudoscience into elite discourse through the Lancet/PNAS/Ted/NPR/Gladwell/Freakonomics nexus has made it easier for pseudoscience like election denial and vaccine denial to get elite traction (even if it’s different subset of elites).

On the other hand, Merchants of Doubt, etc: Cigarette companies and fossil fuel companies have been working for many decades to blur the lines between science and propaganda, so maybe this would all be happening even without all the junk science that entered the mainstream in the replication-crisis era. I don’t know.

Davies writes, “Denial, when you are not part of it, is actually a terrifying thing. One watches one’s fellow humans doing things that will damage themselves, while being wholly unable to help.”

I agree. This is how I felt when corresponding with the ovulation-and-clothing researchers and with the elections-and-lifespan researchers, and other examples too. The people on the other side of these discussions seemed perfectly sincere; they just couldn’t consider the possibility they might be on the wrong track. (You could say the same about me, except: (1) I did consider the possibility I could be wrong in these cases, and (2) there were statistical arguments on my side; these weren’t just matters of opinion.) Anyway, setting aside if I was right or wrong in these disputes, the denial (as I perceived it) really upset me. Seeing young researchers, with their whole careers ahead of them, turning their eyes away from reality, it just makes me want to cry.

I don’t think graduate students are well trained in handling mistakes—indeed, they’re effectively trained to respond to criticism by deflecting it, and then when they grow up and publish research, they can remain stuck in this attitude of denial.

There is a stark problem that I have observed with how data science is taught in most books and tutorials – the concepts are taught on an individual level but fail to give a complete idea. The so-called “real-world” projects are carefully orchestrated like an “act” already setup by the instructor. The students are not given the taste of the real world – where data is messy, models do not continuously improve, and assumptions are almost always broken. We teach them the concepts of statistics and ML but not the art of data science, which includes curiosity and hit-and-trial experiments.

I am writing a one-of-its-kind book on Data Science that would be completely based on projects. The data will be messy, not everything we do will improve the performance. The aim is to give a feel of uncertainty and challenges that promotes learning from failure and develops an analytical bent of mind.

I agree with Joshi’s concerns, and it’s something that’s bothered me for a long time. Textbooks and research articles are full of examples that work, they’re full of estimates that are a bit more than 2 standard errors from zero, everything’s a bit too clean.

I’ve tried to push against this in my own teaching and writing. Here are some examples:

1. Decision analysis textbooks often use examples that are trivial and artificial (a college student is trying to choose an apartment or dorm room, balancing the factors of cost, amenities, and commute time) or important but vague (a company is deciding where or whether to build a new plant, it’s an interesting problem of decision making under uncertainty but the details are not given). A bit reason I wanted to do the full decision analysis with Phil for the home radon problem was that I wanted a real problem, with real data, going from beginning to end.

2. A few years ago there was a heart-stent study reported in the news. The experiment yielded results that were not statistically significant (the p-value was 0.20) and it was inappropriately reported as a null effect. We looked at the paper carefully, and a more reasonable analysis gave a p-value of 0.09–still not reaching conventional levels of significance. We wrote this up, and it’s a good example where the result is not clean. Again, something you don’t usually see much discussion of in textbooks, applied research papers, or articles on research methods.

3. Our book Active Statistics is full of stories with real-world complexity.

4. Appendix A of Regression and Other Stories, on computing in R, has examples where you have to download and clean the data. One problem with a lot of computer-friendly textbooks is that they’ll have clean data already loaded into packages so you can just click and do the analysis. I like there to be a bit of a struggle, to give a sense of what data analysis really feels like.

We try also to convey a sense of real-world complexity in the worked examples and homework problems in our textbooks.

The flip side of this is that, when you’re using live problems in your teaching, it’s good to come to class prepared! It’s fine to show students the steps of data cleaning and the frustration of messy data and models that don’t fit–but it’s best for you, the teacher, to encounter and resolve those problems on your own first, so that you don’t confuse the students even further. In class it’s good to have a sense of what’s coming next, so that when something doesn’t work, you’re not surprised and you can contextualize it for the students right away.

Your post this week on Integration and Differentiation got me thinking, never a good sign. Adding to this, I only really learned in January what Attention means, the foundation of LLM. And what it means is that in a vector space of 14,000 dimensions or so, you can express all manner of nuance… enough to start vitriolic arguments about the humanity of the output.

As I started thinking about this, and as your post crystallized, I have spent 50 years fighting the Curse of Dimensionality. I know this curse in my marrow. Brilliant inferences await me, but the space in which these insights are found is simply too vast to explore. So we simplify, reducing the dimensionality to something that while still vast, is confined to a hyperplane where we can, like Plato, see the projections of truth, not the truth itself.

But then what LLMs and their generation have taught me is that nuance, which is really just the inverse of inference (in that it’s the vast set of all things consistent with some inference) has an amazing boon of dimensionality. There appears to be no thought that can’t be described by a 14,000 dimension or so vector whose tuning has the huge advantage that 14,000-dimensional space is so empty that tiny nuances can be readily distinguished in such a space, so that you can hide uniqueness in the vastness of 14000-dimensional space that you couldn’t recover in a raw search in that same space.

I’m sure this inversion of the Curse of dimensional search into the Boon of nuance in high dimensions is not original to me, but I think it’s worth noting, and your post was the impetus.

I replied by pointing to our of our very earliest blog posts, The blessing of dimensionality, where I wrote:

The phrase “curse of dimensionality” has many meanings (with 18800 references, it loses to “bayesian statistics” in a googlefight, but by less than a factor of 3). In numerical analysis it refers to the difficulty of performing high-dimensional numerical integrals.

But I am bothered when people apply the phrase “curse of dimensionality” to statistical inference.

In statistics, “curse of dimensionality” is often used to refer to the difficulty of fitting a model when many possible predictors are available. But this expression bothers me, because more predictors is more data, and it should not be a “curse” to have more data. Maybe in practice it’s a curse to have more data (just as, in practice, giving people too much good food can make them fat), but “curse” seems a little strong.

With multilevel modeling, there is no curse of dimensionality. When many measurements are taken on each observation, these measurements can themselves be grouped. Having more measurements in a group gives us more data to estimate group-level parameters (such as the standard deviation of the group effects and also coefficients for group-level predictors, if available).

In all the realistic “curse of dimensionality” problems I’ve seen, the dimensions–the predictors–have a structure. The data don’t sit in an abstract K-dimensional space; they are units with K measurements that have names, orderings, etc.

For example, Marina gave us an example in the seminar the other day where the predictors were the values of a spectrum at 100 different wavelengths. The 100 wavelengths are ordered. Certainly it is better to have 100 than 50, and it would be better to have 50 than 10. (This is not a criticism of Marina’s method, I’m just using it as a handy example.)

For an analogous problem: 20 years ago in Bayesian statistics, there was a lot of struggle to develop noninformative prior distributions for highly multivariate problems. Eventually this line of research dwindled because people realized that when many variables are floating around, they will be modeled hierarchically, so that the burden of noninformativity shifts to the far less numerous hyperparameters. And, in fact, when the number of variables in a a group is larger, these hyperparameters are easier to estimate.

I’m not saying the problem is trivial or even easy; there’s a lot of work to be done to spend this blessing wisely.

It’s been over 20 years so worth sharing the point again.

Unlike the Moon, which hangs in the sky like a lonely grandparent waiting for someone to visit, Mars leads a rich orbital life of its own and is not always around to entertain the itinerant astronaut. There is just one brief window every 26 months when travel between our two planets is feasible, and this constraint of orbital mechanics is so fundamental that we’ve known since Lindbergh crossed the Atlantic what a mission to Mars must look like. . . .

We shouldn’t send human beings to Mars, at least not anytime soon. Landing on Mars with existing technology would be a destructive, wasteful stunt whose only legacy would be to ruin the greatest natural history experiment in the Solar System. It would no more open a new era of spaceflight than a Phoenician sailor crossing the Atlantic in 500 B.C. would have opened up the New World. And it wouldn’t even be that much fun. . . .

It wasn’t always like this. There was a time when going to Mars made sense, back when astronauts were a cheap and lightweight alternative to costly machinery, and the main concern about finding life on Mars was whether all the trophy pelts could fit in the spacecraft. No one had been in space long enough to discover the degenerative effects of freefall, and it was widely accepted that not just exploration missions, but complicated instruments like space telescopes and weather satellites, were going to need a permanent crew.

But fifty years of progress in miniaturization and software changed the balance between robots and humans in space. Between 1960 and 2020, space probes improved by something like six orders of magnitude, while the technologies of long-duration spaceflight did not. Boiling the water out of urine still looks the same in 2023 as it did in 1960, or for that matter 1060. . . .

Mars is also not the planet we took it for. . . . The surface might be dry, but in most places there was water ice just underneath. Dynamic surface features hinted that water (or at least brine) was flowing to the surface from deep underground. . . . The news from the ground also got better. Arriving at Gale Crater in 2012, the Curiosity rover found itself looking at an ordinary lake bed, complete with organic sediment and odd stick-like structures that would be called fossils if we found them on Earth. The crater had been habitable for millions of years in the past, and something in it was still emitting methane at night. Over in its own crater, the Perseverance rover found complex organic molecules of indeterminate origin.

But the really exciting news for Mars was the discovery of unexpected life on Earth. . . . not just dozens of unsuspected microbial phyla, but two entire new branches of life . . . These new techniques confirmed that earth’s crust is inhabited to a depth of kilometers by a ‘deep biosphere’ of slow-living microbes nourished by geochemical processes and radioactive decay. . . . This underground ecology, which we have barely started to explore, might account for a third of the biomass on earth.

The fact that we failed to notice 99.999% of life on Earth until a few years ago is unsettling and has implications for Mars. The existence of a deep biosphere in particular narrows the habitability gap between our planets to the point where it probably doesn’t exist—there is likely at least one corner of Mars that an Earth organism could call home. . . . if our distant relatives are still alive in some deep Martian cave, then just about the worst way to go looking for them would be to land in a septic spacecraft.

But the fact that a Mars landing stopped making sense has not had the slightest impact on NASA’s plan to go there in a rocket-propelled terrarium.

And more:

The chief technical obstacle to a Mars landing is not propulsion, but a lack of reliable closed-loop life support. . . . The technology program required to close this gap would be remarkably circular, with no benefits outside the field of applied zero gravity zookeeping. The web of Rube Goldberg devices that recycles floating animal waste on the space station has already cost twice its weight in gold and there is little appetite for it here on Earth, where plants do a better job for free. I would compare keeping primates alive in spacecraft to trying to build a jet engine out of raisins. Both are colossal engineering problems, possibly the hardest ever attempted, but it does not follow that they are problems worth solving. In both cases, the difficulty flows from a very specific design constraint, and it’s worth revisiting that constraint one or ten times before starting to perform miracles of engineering. . . . The only way to explore Mars in our lifetime is to ditch the requirement that people accompany the machinery. . . .

In recent years, there’s been a remarkable division in space exploration. On one side of the divide are missions like Curiosity, James Webb, Gaia, or Euclid that are making new discoveries by the day. These projects have clearly defined goals and a formidable record of discovery.

On the other side, there is the International Space Station and the now twenty-year old effort to return Americans to the moon. These projects have no purpose other than perpetuating a human presence in space, and they eat through half the country’s space budget with nothing to show for it. Forget even Mars—we are further from landing on the Moon today than we were in 1965.

In going to Mars, we have a choice about which side of this ledger to be on.

This all makes sense. I’ve never thought much about this Mars mission thing because it’s always seemed like a bit of a joke. But if powerful people are really gonna try to use this as pretext to take a big chunk out of our national resources, then, yeah, it’s good to have people like Cegłowski pushing back.

To adjust for lots of variables, Si et al. 2020 turned to MRP (Multilevel Regression and Poststratification) and equivalent weights based on these models (see “struggles with equivalent weights”, continued struggles, and “equivalent models, equivalent weights (locally)”).

In a simulation study they compare:

• Ind-P: MRP with commonly-used Independent Normal priors
• Str-P: MRP with a structured prior, see below.
• Ind-W: equivalent weights version of Ind-P
• Str-W: equivalent weights version of Str-P
• Rake-W: classical raking weights, a type of calibrated weights
• PS-W: classical poststratification weights, another type of calibrated weights
• IP-W: inverse probability of selection weights

They cover the “3 flavors of survey weights”: equivalent weights, calibrated weights, IP-W.

I won’t bury the lead, they found MRP performed best, then equivalent weights, then classical weights (calibrated or IP-W). See their Figure 4.1 for the simulation scenario without terribly many empty poststratification cells:

With many empty poststratification cells, Str-P outperforms Ind-P. (They don’t redo Figure 4.1 for this scenario, which confused me a bit.) So what is this structure that helps ?

In “improving with structure” we saw that Gao et al. 2021 found it helpful to use the ordinal structure of variables like age. Si et al. 2020 use the interaction structure:

We induce structured prior distributions to be able to handle deep interactions and account for their hierarchy structure, where the high-order interaction terms will be excluded if one of the corresponding main effects is not selected.

I asked about sparse priors for MRP back in “Sparsified MRP”. I didn’t remember that Si et al. 2020 had worked on this ! Ok so they write their structure more generally but I find it easier to read with a specific example. Consider just 2 variables from their motivating NYC Longitudinal Study of Wellbeing: age (5 categories) and race (5 categories). Here’s how their Ind-P prior differs from the Str-P:

(I had Claude type up my hand-drawn notes, though I still share Brendan Leonard’s preference for hand-drawn materials.)

Si et al. 2020 say these are similar to the Horseshoe prior. It differs in two ways, I think ? First, Si et al. 2020 have the selection at the batch level (e.g. age or race). The usual Horseshoe would have local scale lambdas for each age and race category. Second, the usual Horseshoe would use a half-Cauchy rather than half-Normal prior on these local scales.

Ok let’s get back to the original concern: adjusting for lots of variables can lead to very large weights. Si et al. 2020 show in Figure 5.1 that the equivalent weights based on this structured prior model look much less variable than calibration weights (I don’t see the IP-W weights in the figure itself):

Walnutpie is an MCMC sampler for continuously differentiable densities coded in Python, accepting models coded in Stan, PyMC, NumPyro, JAX, and plain old Python.

Walnutpie is not an official Stan project

I thought this was worth saying up front. It may eventually migrate to Stan, but for now, we followed the Nutpie approach of building a standalone sampling package that worked with a range of packages for defining models.

R version

We plan to develop an R interface after we release version 1.0.0 of the Python interface. So hopefully in 2026.

pip installable

Walnutpie is on PyPI, so it’s pip installable. The documentation includes information on getting started, running models, and posterior analysis. We have not yet included case studies for modeling tools other than Stan and Python.

• Walnutpie documentation

Stan through C++

Walnutpie runs Stan models through C++ using BridgeStan, so there is no Python dispatch overhead for Stan sampling. The basic architecture of the API is based on Adrian Seyboldt’s sampler Nutpie. We are working on doing that for other packages like NumPy and JAX to the extent that we can.

GitHub source

Development discussions and source code are managed through GitHub.

• Walnutpie on GitHub

Features of Walnutpie

We are almost ready to release the paper on arXiv with all of the gory pseudocode details of all of the algorithms used for Walnutpie. It will explain the following points in detail.

1. Walnuts: The underlying Hamiltonian Monte Carlo sampler is Walnuts. Walnuts uses Nuts for choosing the number of steps per iteration. It further allows step sizes within the Hamiltonian dynamics simulation to be lowered when necessary to preserve simulation accuracy. This helps with robustness and with accuracy in multi-scale distributions (i.e., ones where the curvature as represented by the Hessian varies around the posterior). With a high tolerance threshold, Walnuts reverts to Nuts’s behavior.

2. Mass-matrix warmup: The mass-matrix warmup strategy is an online form of Nutpie (links to: the paper and software). Nutpie minimizes Fisher divergence by estimating the inverse mass matrix as the midpoint (in the appropriate manifold) between an estimate based on the variance of the draws and the covariance of the scores (gradients of the log density). The target is better than Nuts’s variance of draws in both convergence speed and sampling efficiency. Walnutpie only supports diagonal mass matrices (Nuts supports dense matrices and Nutpie supports low-rank plus diagonal and even more general normalizing flow approaches). The approach is online in the sense that it is not blocked like warmup in Nuts or Nutpie—it updates every iteration by exponentially discounting the past to mimic Stan’s exponentially increasing history sizes. We also borrow Nutpie’s mass matrix initialization based on a regularized outer product of gradients at the initial point.

3. Step-size warmup: The step size adaptation strategy has not changed, but the underlying stochastic gradient descent algorithm is Adam rather than dual averaging. We found Adam to be faster to converge and much more stable. Matt Hoffman included a hack in the original Nuts approach to stabilize dual averaging, but even with that it is not as stable as Adam.

4. Concurrency and automatic stopping: The underlying sampler is multi-threaded (using C++11 threads) with shared data. On top of the multi-threading, we have layered a convergence monitor in a separate thread that communicates with the chains through lock-free, latest-only, single-producer/single-consumer (SPSC) buffers (specifically, a triple buffer). The monitor automatically stops warmup when the mass matrices and step sizes have converged within tolerance to their cross-chain averages. The monitor automatically stops sampling when a target (traditional, non-split, non-ranked) R-hat; threshold is satisfied for the unnormalized log density, which typically converges more slowly than any of the individual parameters. It can also be configured to run for a fixed number of warmup and/or sampling iterations. The link between the original R-hat and effective sample size makes this essentially an unscaled ESS target.

5. Ragged chain summaries: Asynchronous concurrent execution of chains with automatic stopping produces chains of different lengths. Because ArviZ does not accept ragged chain input of this kind, we have included posterior analysis tools for means, variances/standard deviations, quantiles, traditional R-hat, effective sample size, and Monte Carlo standard error that work with ragged chains.

6. C++20: Walnutpie is implemented in C++20. As a programming language type fanatic (I’ve written two books with “type” in the title!), I don’t know how I survived without C++ concepts before C++20.

7. ctypes FFI: The foreign function interface in Python uses ctypes rather than a higher-level interface, which sidesteps the requirement of ABI compatibility of C++ binaries.

Developers

We would also like to welcome new developers who may want to get involved. There are already a stack of improvements we’d like to make, which we have enumerated on the GitHub issues.

Sources of algorithms

The Walnuts algorithm was a joint effort among Nawaf Bou-Rabee, Sifan Liu, Tore Kleppe, and Milo Marsden. Nutpie was developed by Adrian Seyboldt. Nuts, in the form used currently in Stan, was originally developed by Matt Hoffman and Andrew Gelman, then improved with multinomial sampling and mass matrix adaptation by Michael Betancourt. We haven’t yet added cross-chain adaptation as developed by Ben Bales, but the pieces are all in place to do so.

Brian Ward and I wrote all of the version 0.0.1 code with design advice and code review from Steve Bronder. Claude (the LLM) helped with code review and testing, but we didn’t use it to write the actual code (not out of principle, but because Brian and I both prefer the control of doing things manually).

Feedback

We would very much appreciate any feedback people have, including feedback on the documentation and ease of use, the source code, and performance.

We are happy to get feedback through issues on GitHub, through replies to this post, or through mail to one of the developers.

Regarding your blog of today, believe it or not, I knew Marshall Brickman pretty well before he became famous. He was an undergrad at the University of Wisconsin in Madison when I was a grad student there. We belonged to the same very left-wing eating co-op, “The Green Lantern.” In that era, undergrad males at Madison and elsewhere, had to take ROTC classes, and he kept intentionally failing them because he was very leftwing politically.

Inasmuch as I am a helpful sort, I offered to nominate him for “Military Ball King,” but he declined. Marshall was a really gifted musician as well as a writer. In addition, he looked strikingly like the actor Richard Carlson who played Herb Philbrick in the 1950s television series I Led 3 Lives.

The gimmick is that you’d buy the book if you wanted to enter the comedy field. Which doesn’t seem very likely, but, then again, who reads books either? So maybe I’m one of the few people reading this one for entertainment value alone.

One of the interviews is with Woody Allen collaborator Marshall Brickman, coauthor of the 1970s classics Sleeper, Annie Hall, and Manhattan. Lots of interesting stuff in this interview, including this:

Marshall Brickman: [Woody] found a whole new area of insight: relationships of a certain kind, psychoanalysis, and the creation of the so-called loser—mostly with women. To some extent, the lower-with-women character was someone Bob Hope would play, but in a much more general and mainstream way. Woody’s character was more ethnically and culturally specific.

Mike Sacks: It takes true genius to develop a comic character like that.

Brickman: It does, but it also requires a collaboration with the audience. It’s the only way you can do it. You have to get out there and do a variety of material. Over times, certain things, statistically, will continue to work, and other things will drop away, and the audience will tell you what seems correct for you—for what you project onstage as a personality.

Sacks: But even with that said, you can work for twenty years and never connect with the audience half as much as Woody Allen.

Brickman: That’s right. That’s the genius. Creating something that somehow resonates with an audience that strongly.

If only Allen had died in 1986 (after releasing Hannah and Her Sisters) or maybe 1992 (after Husbands and Wives), just think how high his reputation would be. Even setting aside his personal life, releasing a series of flawed projects takes a toll. Not everyone’s a Scorsese or Spielberg who can keep it up into their old age. If Woody had gone in 1986, we’d still be speculating about the amazing work he didn’t have the opportunity to do.

But the thing that really interested me in the above-quoted interview is what Brickman said about collaboration with the audience.

This is true of teaching too! Teaching a class, even writing a book, is a collaboration with the audience. A book doesn’t read itself. With books, the challenge is that the audience doesn’t see it until it comes out. To continue the analogy, a live class is like stand-up; a book is like a movie.

When writing books, we always have the audience in my mind. The trouble is that the book might never reach the audience. That’s what I think has happened with Active Statistics. I’d like to rearrange it and rerelease it as a book called Statistics Stories. Or maybe, Important Concepts of Statistics in Story Form. Also I wish we’d given Regression and Other Stories the more serious and descriptive title, Applied Regression and Causal Inference. Maybe we can rerelease it under that title.

Anyway, yeah, “collaboration with the audience”: definitely.

He’s using the chickens model from our Slamming the Sham paper in the more general setting of placebo control tests.

In econometrics, a “placebo control test” does not need to literally involve a placebo treatment; it more generally is used to describe a procedure in which the same statistical analysis that was used to estimate a causal effect is applied to a different dataset, or a different part of the existing dataset, in which the treatment did not occur.

For example, if you want to measure the effect of an intervention that occurred in 2021, you could repeat the analysis but using data from a different year. Or if you want to measure the effect on a particular outcome, you could repeat the analysis but looking at a different outcome that should be unaffected by the treatment.

The idea is that, if your estimation method has an artifact or systematic bias, this should show up in the placebo analysis as well, indicating a problem. Conversely, if the placebo analysis does not show an effect, this is taken as evidence that there is no artifact.

In his post, Gazvoda argues that, rather than using the result of the placebo check to make a go/no-go decision, it should be possible to partially adjust the treatment effect to account for the information in that supplementary analysis.

This makes sense to me, and of course I’m happy that he’s using our chickens model.

There’s a tricky thing going on here with the placebo check, which, interestingly, arose in the chicken example too, and that is that we usually don’t have any good theory for where the effect is coming from in the placebo control. After all, if our causal identification is working as designed, we shouldn’t even need the placebo comparison, as we’re already getting an unbiased estimate of the treatment effect. The placebo control is typically there to address unspecified concerns of bias. And, indeed, in practice, researchers don’t always do placebo controls. And when, as hoped, the placebo control shows no statistically significant effect, the inclination is to take that as a reassurance and move on, in the same way that is done with other robustness checks.

The lesson Gasvoda takes from the chickens example is that if replications are available, you can assess the evidence for the placebo adjustments being relevant: you can fit a multilevel model to estimate how much adjustment needs to be done.

From a sociology-of-science point of view, it’s interesting to me that the conventions in biomedical statistics and econometrics go in opposite directions:

– In biomedical statistics, the default recommended behavior is to compute the difference in differences, taking the estimated effect from the main experiment and subtracting the estimate from the placebo experiment. As we explain in the chickens paper, this correction has the disadvantage of doubling the variance of the estimate, a true statistical crime if, as is often the case, the effect of the placebo treatment is indistinguishable from zero.

– In econometrics, the default procedure, if you’re calling it “difference-in-differences estimation,” is the same as above. But if you’re calling it “placebo control,” and the placebo estimate is not statistically significant from zero, the default is to ignore the placebo results entirely, not to adjust for them.

In general we recommend a partial adjustment, with the amount of adjustment depending on the problem at hand. If there is internal replication, as in the chickens example, the appropriate adjustment factor can be estimated from the data. If it’s a one-shot experiment, you’ll need to use prior information. I don’t have any good examples demonstrating how to do that; it’s something we should do.

Statistical methods can be characterized in terms of how they can go wrong. Some common modes of “how can things go wrong” include:
– biased measurements,
– selection bias into the observed dataset,
– differences between treatment and control groups,
– nonstationarity or, in general, differences between sample and population of interest,
– biased or noisy statistical estimates,
– missing data,
– problems with functional forms (for example, using a linear model for a nonlinear relation, or not including important interactions),
– problems with error terms (dependence, choice of distribution, etc.),
– unmodeled spillovers, hierarchical structure, or other violations of causal model assumptions,
– researcher degrees of freedom and forking paths,
and lots more!

Regression discontinuity is used for observational studies, and the #1 thing that can go wrong in an observational study is #3 on the above list: differences between treatment and control groups.

The problem is that when researchers perform regression discontinuity analysis, they focus on the problem with the functional form of the expected outcome given the one predictor, often ignoring other potential predictors even if they are screaming to be included (as with age in this example). There’s all this obsessing over the functional form (and I guess Guido and I contributed to this) that’s kind of missing the point (as we discuss briefly here).

So, the problem is not that these users of regression discontinuity don’t know what to do with other predictors; it’s that they don’t know that they don’t know this.

As Tolstoy might have said had he been a data analyst, Models that fit are all alike; every poorly-fitting model is poorly fitting in its own way.

P.S. Here’s some background from the Quote Investigator on various version of the saying that I used as the title of this post.

Your process limits information search, computation, and time so it seems fast and frugal to me. Happy you still associate me with that term. It was something I coined as a grad student.

In your proposal, only hit papers get audited. It reminds me a bit of Mel Brooks’ The Producers in which the protagonists use the logic “who would audit a flop?” and stay under the radar by intentionally producing a bad show. Fraudsters have likely attempted to make their work seem worthy of publication while ensuring it doesn’t attract too much attention. Just like in The Producers, though, this sometimes backfires.

I don’t know about that! My impression with fraudsters is that they think that fraud is normal science, perhaps out of some mixture of bad education in research methods, a view that “everybody does it,” and a general lack of understanding of how non-cheaters (like you and me!) think. Think of people like Wansink who gave general advice to to the world on how to p-hack, or Gino and Ariely, who published papers on dishonesty, or Mary Rosh, who surely believes that whatever shady statistical manipulations she does are nothing compared to the dastardly deeds done by the Democrats.

I’m sure there’s tons of below-the-radar cheating and bad science that we don’t hear about, but a fair number of prominent science fraudsters seem to enjoy the limelight. One reason for this seemingly self-sabotaging behavior, I think, is that cheating enabled these people to attain great professional success for years. They had no reason to think the juice would stop flowing.

To return to my proposal with Andy King: I think it’s ok that only the hit papers get audited. Bad papers that get no intention aren’t doing much damage, right?

Goldstein adds:

By the way, I was just having a conversation about your sensing that something was amiss with the LaCour study. For years now I have used this quote of yours in a talk I give about putting numbers into perspective. I argue that it’s really important that people learn how to put numbers into perspective because if they don’t, they won’t notice that something is unusual and worthy of a deeper audit. You somehow sensed something was up with the Lacour result. You didn’t think it was fraud yet but you knew it was strange because you know how to put such differences into perspective:

A difference of 0.8 on a five-point scale . . . wow! You rarely see this sort of thing. Just do the math. On a 1-5 scale, the maximum theoretically possible change would be 4. But, considering that lots of people are already at “4” or “5” on the scale, it’s hard to imagine an average change of more than 2. And that would be massive. So we’re talking about a causal effect that’s a full 40% of what is pretty much the maximum change imaginable. Wow, indeed. And, judging by the small standard errors (again, see the graphs above), these effects are real, not obtained by capitalizing on chance or the statistical significance filter or anything like that.

My colleagues and I recently wrote a paper on this general topic of average effect sizes. It’s our contention that people generally are way too optimistic about possible effect sizes, in large part because they don’t think about variation. If you ask someone to hypothesize an effect size, you’ll typically get a guess of the largest effect that might occur.

But what if you don’t really care about effect size–you just want to know about the effect?

For example, maybe you don’t believe that women during certain times of the month are three times more likely to wear red or pink shirts, but you are interested in some sort of evolutionary psychology theory of sexual display. In that case, why should the effect size matter? Why care that a study reported an estimate that was ridiculously implausible?

I have two to this questions, and thus two reasons why effect size is important even for problems where you don’t directly care about effect sizes:

1. Effect sizes vary. An treatment that has an effect (that is, a true effect, not just an estimated effect) of 0.1 for one group of people in one setting could have an effect of -0.2 in some other scenario. A treatment effect in an experiment is the sum of all sorts of things, positive and negative, and there’s no logical reason to think the sign of the effect will be preserved. Effect size matters. The issue is not just that a smaller and more realistic effect size is less important; it’s also that smaller effects can be more easily produced by other factors, and this reduces the generality of any claims, even if the experiment at hand was done well.

2. Experiments produce standard errors as well as estimates. If the standard error from a study is large compared to any realistic effect size, then the study contains very little information. Effect size is important in understanding the informativeness of an experiment, and to do this right you need to have some sense of what the true effect size could be. You can’t just use a point estimate from the study itself, as this estimate will inherently be too noisy to use to judge the information in the study. As I wrote in this note for the Annals of Surgery, Post-hoc power using observed estimate of effect size is too noisy to be useful.

I was reading ‘The Bestseller Code.’ The book reports results from some regressions of the form:

bestseller or not ~ features of content

This got me thinking about what you can recover from such an exercise.

Say that there are two types of novels: type x and y. Let’s say that people prefer reading novels of type x. They are k% more likely to read a novel of type x than y.

At time t, the total number of novels is T, with novels of type x being 50%. And we can recover k% by regressing the number of people who read a novel on the type of novel.

Reader preference for type x novels leads writers to produce more novels of type x. At time t+1, the percentage of type of novels x being produced is 70%. But say that the total number of novels of type x that people can read is fixed at n_t. When we regress the number of people who read a novel on the type of novel, we can get that people are less likely to read novels of type x than y because we have more novels of type x at t+1.

I wrote a brief post based on the point here.

I agree, and this sort of thing has always puzzled me. I’m sure economists have looked into the matter, but I don’t know the literature so I’m just guessing here. Whenever people talk about the relative profitability of different genres, I wonder about what happens when the more profitable genre gets flooded with content. That doesn’t mean that bestseller regressions are useless, just that they represent at best some equilibrium of a process that I don’t understand very well.

The idea is that the distribution of the test statistic depends on the true parameter vector–there is no simple pivotal quantity as would exist under a linear model with no constraints–but you can work out the distribution conditional on the true parameter vector, and then you can average this distribution over the posterior for the parameters.

You can think of this as Bayesian–the marginal posterior distribution of the test statistic–but at the time I was thinking of it more as a generalization of the existing “plug-in” approach that would use a point estimate of the parameter vector. From that perspective, posterior averaging is a technique for getting a distribution with better frequency properties than you’d get from the maximum likelihood estimate, and that’s because in an image reconstruction problem with positivity constraints and lots and lots of pixels, the maximum likelihood estimate will almost certainly be on the boundary of parameter space. So just about any sort of averaging should get you closer to the true parameter value.

As noted above, I started working in this area in 1988. Around 1991 I got my thoughts organized enough to write a paper and give a talk on the topic, and . . . it got a generally hostile reception! The Bayesians didn’t like it because they didn’t like chi-squared tests or frequentist hypothesis testing more generally. They wanted me to do Bayes factors, which even then I realized were generally a bad idea (for more on the topic, see chapters 6 and 7 of BDA3 (it was all in chapter 6 of the first two editions) or this article from 1995). The non-Bayesians didn’t like it because it was Bayesian, and because I just presented the method, without trying to justify it based on minimax properties or whatever.

Xiao-Li, Hal, and I finally published a version of the article a few years later, and I resigned myself to the fact that posterior predictive checking was going to remain in the Bayesian world. Predictive checking was a hard sell to the Bayesians, but, after a few decades of exposure to BDA and lots of applied examples, they started to get used to the idea. I gave up on trying to promote the idea outside the Bayesian community.

But . . . it turns out that posterior predictive checking has been taken up for non-Bayesian uses! Aki pointed me to this R package called “performance” that “provides posterior predictive check methods for a variety of frequentist models.” Here’s their vignette on checking model assumption – linear models,” and here’s the article, performance: An R Package for Assessment, Comparison and Testing of Statistical Models, by Daniel Lüdecke, Mattan Ben-Shachar, Indrajeet Patil, Philip Waggoner, and Dominique Makowski. It came out in 2021 and has already been cited over 2500 times.

So it looks like people really are using posterior predictive checks for non-Bayesian models. And it took less than 40 years for it to happen!

We publish interval forecasts under a pre-registration scheme: each forecast is serialised, hashed and timestamped into a Bitcoin block before publication, so the stated interval cannot be adjusted after the outcome is known. Across 56 resolved forecasts our stated 50% intervals contained 47 outcomes; at the 24-hour horizon, 36 of 44. Under honest 50% intervals, P(>=36 of 44) is about 1.3e-05.

The diagnosis, and the part I would most like criticised: walk-forward across 20,058 observations showed the miscalibration was conditional rather than uniform – 55.6% coverage across all days, 67.1% restricted to the calmest fifth. Our first hypothesis, that the sample carried too much old high-volatility history, was tested and rejected. The surviving explanation is that interval width ignored the current volatility regime. Conditioning the historical sample on regime measured at each window’s open (not its close, which leaks the outcome) brings coverage to 50.8%.

Two things I am unsure about. The instruments are correlated, so effective sample size is well below 56 and I have not done that properly. And at a 30-day horizon the conditional interval comes out wider than the unconditional one, which I have kept but cannot fully account for.

I asked him what was the application, and he replied:

Crypto prices. 24-hour and 7-day intervals on eight pairs (BTC, ETH, SOL, BNB, XRP, DOGE, ADA, LINK), scored against Binance closes.

I chose it as the substrate rather than the subject. Outcomes resolve within a day, the reference price is unambiguous, and there is no data-vintage or revision problem, so a coverage check accumulates evidence quickly and cheaply. The same construction is what we use for sports and prediction-market questions, but those resolve slowly and the sample is thin.

I realise crypto invites a certain reaction, and the reaction is mostly deserved. The calibration question doesn’t depend on it – the same test applies to any published range.

He adds:

If it helps frame it, the two things I’m least confident about are the ones I’d want readers to attack:

1. The eight instruments are correlated, so the effective sample size is well below 56 and I have not handled that properly. The binomial p-value I quoted assumes independence it doesn’t have.

2. At a 30-day horizon the regime-conditional interval comes out wider than the unconditional one. I kept the result because it’s inconvenient, but I can’t fully account for it beyond “calm periods have historically preceded larger monthly moves”, which feels like a description rather than an explanation.

Data and outcomes are CC-BY, one row per sealed forecast with its hash:
https://huggingface.co/datasets/neuportal/neuportal-sealed-crypto-forecasts

My only quick thought here is that it’s indeed a good idea to look at conditional calibration, but you have to be careful only to condition on things in the forecast, not on the outcome. So when he writes, “67.1% restricted to the calmest fifth,” this is relevant as long as “calmest fifth” is something that can be determined before the outcome occurs.

More generally, yeah, it’s hard to evaluate the calibration of forecasts for correlated outcomes–this comes up in election prediction too. My only answer is to try to model intermediate outcomes as much as possible to avoid getting stuck in a purely empirical mode of evaluations.

Maybe you in comments have other ideas?