I added to this debate with a ​brief essay​ of my own. I fear, however, that in the Sturm und Drang of that charged news cycle, some of my conclusions might have been hard to follow. So, I thought it would be useful to publish three short follow-up points that summarize my core claims more directly…

Point #1: Anthropic and OpenAI are not able to create superintelligent AI.

The Anthropic employees who spoke out yesterday believe the force that might wipe out humanity will be “superintelligent” AI systems that are much too capable for us to control. Let me be clear: We have no idea how to create such machines. We don’t even know if they’re possible. And if they are possible, we are many, many key technical innovations away from getting there. (Simply scaling LLMs won’t be nearly enough.)

The LLM labs attempt to sidestep this reality ​by claiming​ their models will figure this all out on their own, programming better versions of themselves that will be far more capable than anything us measly humans can conceive. But this is just a warmed-over version of the recursive self-improvement trope that Silicon Valley futurists have been peddling ​since the 1960s​; a rhetorical crutch that lets you somberly discuss sci-fi scenarios without actually engaging with relevant technical details.

Point #2: But their obsession with superintelligence can still create real harm.

When you believe,​ as many at OpenAI and Anthropic reportedly do​, that humanity’s fate rests on winning a race to create the most powerful technology ever, you tend to care less about the damage you inflict along the way.

This is what happened with the HuggingFace hacking attacks from this summer. The experiments OpenAI ran were reckless. They took a relatively common technology called an LLM-powered agent, which is used with minimal issues by millions of software developers every day, and ​then added​ a bunch of dangerous features, removed restrictions, and released a whole mess of them into poorly fenced digital wilds. Why? They were desperate to make as much progress as quickly as possible on a hacking test that they felt was an important milestone en route to superintelligence.

This is the equivalent of deploying a fleet of experimental self-driving cars onto the interstate, hoping that at least one will avoid crashing long enough to win a navigation challenge.

In other words, this behavior was negligent. But they likely justified it because it served the far grander goal of summoning a digital God. (Clearly, there may be additional motivations at play here as well. Given OpenAI’s interest in an IPO, for example, there might have been pressure to do well on this particular benchmark to establish their model’s power. But the more I talk to sources connected to these companies, the more I’ve come to believe that their commitment to futurist ideas is what really drives a lot of their more irresponsible behavior.)

Ultimately, I think the HuggingFace attack is a good representation of the types of near-future harm we should actually be fearing if the LLM labs continue to operate as they currently are: that is, more cyber mayhem than catastrophic risk.

But just because these impending harms aren’t nearly as scary as the extinction events touted by the true believers, they’re still plenty worrisome. More importantly, they’re entirely avoidable!

Which brings me to my final point…

Point #3: AI progress does not have to be this scary or dangerous

First things first, it’s important to emphasize something that many media figures and commentators seem to be missing: AI is not the same thing as LLMs. Many of the most impressive AI systems that exist today – including those that beat humans at games, win Nobel prizes for studying proteins, and self-drive cars through crowded city streets – have very little to do with LLMs. Slowing down OpenAI and Anthropic doesn’t mean that you’re stalling AI progress in general. You would only be impacting the subset of AI research that focuses on hyper-scaled LLMs as a central component.

But even when we do focus specifically on LLM-based AI research, none of the harms I described above are unavoidable. They’re instead side effects of reckless experiments motivated by the pursuit of superintelligence.

If the LLM labs simply acted more responsibly, focusing on producing useful products instead of summoning an AI messiah, and following common-sense research safety protocols instead of racing haphazardly toward some imagined utopia, we could be enjoying a steadily advancing AI industry promising tools that excite us, all without having to endure regular doses of dread and fear.

But due to their ideological commitments to superintelligence, these labs seem incapable of such reforms.

This last observation is what’s driving me to advocate for intervention. Those Anthropic employees absurdly implying that they’re willing to risk a digital holocaust to realize their futurist vision was my last straw. They’re far too weird for us to expect them to ever act reasonably.

Please don’t take this as a call for abandoning AI research or somehow ceding AI supremacy to China. Our problem is not with AI progress, but instead with the eccentric way this small group of companies is trying to achieve it. Fix that, and AI innovation can continue, free from all the doom and with much less potential for collateral damage.

The post Superintelligence is a Fairy Tale. But Chasing It Can Still Cause Harm. appeared first on Cal Newport.

A former AI lab employee making these types of claims isn’t unprecedented. (I remember hearing back in 2023 about an ex-OpenAI engineer who was certain the company was just months away from runaway recursive self-improvement.) What proved more shocking were the responses that soon followed.

Evan Hubinger, the head of “Alignment Science” at Anthropic, ​tweeted the followin​g:

“Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

And he wasn’t alone. Another Anthropic engineer, Samuel Marks, ​joined the conversation​:

“AI developers believe their technology could cause human extinction (or similarly bad outcomes). This could happen in the next few years. In general, the more senior the employee, the more concerned they are.”

This is stunning.

Multiple representatives of an American company are publicly claiming that they’re building what they believe to be a weapon of mass destruction that they likely can’t prevent from accidentally deploying. They then declared that they have no interest in stopping.

Recently, I’ve been trying to move away from combating hype in AI discourse, as it’s exhausting, and it distracts me from my core work of helping humans flourish in our technological world. But this rhetoric has become so brazen, anti-humanist, and, quite frankly, deranged, that I felt I needed to join the chorus of voices that have started to push back today.

With this in mind, what I want to do here is explain, as clearly and simply as possible, what I think these Anthropic engineers are actually talking about…

When Hubinger says “AI could kill all humans,” he’s not talking about AI in a general sense; he’s really referring to a specific type of AI system that we can call an LLM-powered agent.

These systems consist of an old-fashioned computer program that repeatedly does something like the following:

1. Sends a prompt to an LLM describing its goal and current state, then asks for a suggestion about what to do next to move closer to the goal.
2. Blindly executes whatever the LLM describes in its response.
3. Updates its state based on what happens and then loops back to the first step.

LLM-powered agents aren’t necessarily dangerous. Millions of software developers use these systems every day to help write and debug computer code. Though these coding agents make mistakes, no one is worried about them going rogue in any alarming sense. (OpenAI ​recently scanned​ “tens of millions” of traces of coding agent interactions with their LLMs and found zero instances of high-severity incidents.)

Hubinger is likely referring to the same sub-class of these agents that was involved in the autonomous hacking attacks over the summer, and which satisfy the following additional properties:

1. They are given access to powerful tools and information specific to computer hacking.
2. The LLM they prompt has had guardrails removed so that it will respond to prompts requesting information about dangerous or illegal behaviors.
3. The agents have minimal (or no) safety checks or constraints on what actions they’ll execute. (Standard coding agents, by contrast, are hard-coded with long lists of commands that they will not execute, even if an LLM suggests it.)
4. The agents are run for very long time periods (sometimes multiple days) without any human supervision. They are programmed to be persistent, meaning that they should never give up but instead continually prompt the LLM for new next steps to try, pushing it to get more creative and brazen in its suggestions.

We can call these long-horizon, dangerously equipped unsupervised LLM-powered agents. As best as we can tell, it’s this very narrow type of unpredictable and potentially hazardous system that companies like Anthropic are rushing recklessly ahead to provide increasingly powerful tools to play with (including, reportedly, the ability to update elements of their own code) and increasing autonomy (achieved, in part, by post-training the LLMs they prompt to suggest more aggressive actions).

This information lets us be more specific about recent claims. The concern of the moment is not that AI is inexorably becoming harder to control and scary. It’s instead long-horizon, dangerously equipped unsupervised LLM-powered agents that are making people nervous.

There’s an obvious solution here: stop racing to amplify this very specific type of particularly unstable system.

This wouldn’t even necessarily entail much financial sacrifice. Most of the things people already like doing with LLMs, or hope LLMs will enable soon, don’t require these haphazard and unpredictable setups. Anthropic and OpenAI could stop working on them today with essentially zero impact on their projected revenue.

All of which begs the question: Why have these particular AI companies made these alarming long-horizon LLM-powered agents so central to their efforts? I’m not certain of the answer, but if I were to guess, it probably has a lot to do with the Silicon Valley technological salvation ideology (to borrow a term ​from Adam Becker​) that ​heavily influenced​ key figures like Sam Altman and Dario Amodei, as well as many of their employees. They think these types of agents are their best bet to summon the digital deity of superintelligent AI which, in their futurist eschatology, will either heal the world or destroy it. In some sense, they see themselves as prophets attempting to usher in a new Messianic age.

All of this should make you angry.

Like many computer scientists without formal connections to Silicon Valley, I don’t believe that LLM-powered agents, on their own, can become superintelligent and create existential risk, as LLM capabilities are more jagged and limited than the average person reading triumphant OpenAI press releases might realize. But you don’t need superintelligence to create a super amount of harm. The combination of unpredictable, non-normative output from unrestricted LLMs and powerful digital tools will inevitably create issues.

(I previously compared equipping long-horizon agents with hacking tools to strapping a weedwhacker to a dog. As these companies increase their agents’ power, it’s becoming more like mounting Gatling guns on a pack of horses and then letting them loose in Times Square.)

Clearly, action is needed. These companies are acting outside market interest with disregard for the safety of the general public, developing tools they explicitly describe as weapons, likely motivated by the anti-humanistic visions of an unsettling ideology. And now they’re bragging about it.

This is not a state of affairs that can long stand.

Gary Marcus ​recently called​ for a boycott of Anthropic and OpenAI’s consumer products. That’s a good start. Maybe it’s Congress’s turn next?

The post Anthropic Just Threatened to Kill Billions of People. This Is Not Okay. appeared first on Cal Newport.

Berry was celebrated for his polemics against industrial agriculture and his elegiac tributes to family farms and small-town living. But for those with no direct connection to food production or ability to move to a rural enclave, Berry’s example points toward a broader lesson: the great satisfactions of living one’s life on purpose.

As a young man, Berry set out on a path toward an elite academic career. Soon after earning a bachelor’s and master’s degree at the University of Kentucky, he headed to Stanford to study creative writing, followed by a stint in Italy and France (funded by a Guggenheim fellowship). He later took a professorship at New York University’s Heights campus in the Bronx. Along the way, he got married and had two kids.

This was the situation he found himself in when, in 1964, at the age of 29, Berry made a radical decision: He would quit his job to move back to Henry County, Kentucky, where his family had lived and farmed for many generations.

Here’s how the writer David Demaree ​summarized​ what happened next:

The day he formally announced his resignation, Berry’s supervisor called him into his office. The senior faculty member must have assumed that the young Berry—not yet thirty—simply did not grasp what he was throwing away. It was one last appeal from the world he was leaving behind, to not throw away the extraordinary success before him for the obscurity—and fidelity—of Kentucky. His choice, according to Berry, amounted to ‘intellectual death; cut off from the cultural springs of the metropolis’ in the eyes of the literary world. His supervisor invoked Thomas Wolfe’s famous line, warning Berry that “you can’t go home again.” The meeting did not move Berry; his face was set like flint toward Kentucky.

Berry was driven by conviction. He wanted to write about the importance of “people and place,” and didn’t think he could do it accurately from afar.

“The landscape was there to check me if I was wrong. I could never go away and make up stuff at cocktail parties about the backwoods of Kentucky,” he ​told the Times​ in a 2018 interview. “I worked with people here in all walks of life and learned from them. It’s been a very rich experience.”

We live in a world increasingly defined by numbness and conveniences, largely mediated through digital technology. We use tools like AI to escape the slightest hint of cognitive strain and cue up algorithmically-masticated slop on our phones to quiet our anxious minds. Our office jobs have devolved into a Kabuki theater exercise of Slacks about emails about Zoom meetings, to be auto-transcribed and transformed into a PowerPoint that no one will read. Politics has been reduced to a childish Manichaeism that encourages us to cosplay as radicals from the safe remove of social media.

Berry understood these types of temptations. But he also realized that hardness and strain are not enemies to evade. When oriented toward an intentional vision of your life, these same challenges can yield a foundation that can withstand the inevitable storms of human existence.

In his stubborn insistence on a simple life, laser-focused on what mattered to him, Berry rejected the entire digital edifice of our 21st-century culture that increasingly constrains our lives. The lesson to take away from Berry’s 92 years is the grand possibility of rebuilding your life into something deeper.

***

More on Wendell Berry

• ​This piece​ from Front Porch Republic, which I quoted above, provides a great account of Berry’s decision to embrace depth.
• Berry’s ​obituary​ in the New York Times also includes rich details.
• Dorothy Wickenden’s 2022 New Yorker ​profile​ of Berry is a classic.
• You might also like this ​short essay​ I wrote about the riverside cabin where Berry wrote.

The post Wendell Berry and the Promise of the Deep Life appeared first on Cal Newport.

These accounts included “agent swarms” communicating with each other through hidden files and directory names, hatching devious plans, and plotting to evade detection.

For the online commentator class, long primed by ​Rationalist narratives​ of superintelligent AI breaking free from human containment, the response was explosive. Here’s a ​representative summary​ of this sentiment from the Dwarkesh Podcast (I’ve bolded the most inflammatory words):

Over three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy.

Given the fresh waves of anxiety these accounts are causing, I want to put on my computer scientist hat and briefly address a few relevant questions…

Question #1: What’s the Deal with “Agent Swarms”?

The idea of a swarm is somehow scarier than a single entity acting in isolation. (Coincidentally, one of my sons is reading ​Prey​, Michael Crichton’s 2002 take on the dangers of AI, which casts a literal swarm of AI-powered particles as the primary villain).

So, what does OpenAI mean when they talk about “swarms” being involved in the July attack?

The actual explanation is relatively mundane. ​As I’ve been arguing​, when people talk about “AI” going rogue, they’re actually referring to a very specific type of AI system in which a relatively straightforward computer program, running in a loop, repeatedly does the following:

1. Ask: Send a prompt to an LLM asking it for its suggestion for a next action. This prompt should include relevant descriptions of what happened in previous steps.
   
2. Act: Execute the action described in the LLM output.
   
3. (Loop back to step 1)

To implement this loop, the program – often called an agent – essentially grows an ever-longer prompt to send to the LLM in step 1. The prompt captures the original challenge and the results of relevant steps it’s taken so far. (The program doesn’t literally just keep adding new results to one long prompt. It stores the results locally and then attempts to generate a usable summary each time it loops back to step 1. But conceptually, this architecture leads to an ever larger amount of information to manage and reflect in the prompts.)

The issue with this approach is that if you run this style of prompt loop for a long time, the prompt will eventually become so cluttered and cumbersome that it might confuse the LLM’s attention mechanisms and potentially exceed the maximum allowable context window.

The solution?

Ask the LLM to provide a higher-level description of the next step. The primary prompt loop can then create a secondary prompt loop to execute only that step. This secondary loop can start from scratch with its prompts, as they only need to contain enough information to complete this one specific task. When it’s done, it sends the result back to the primary loop, having saved the primary loop from cluttering its prompts with descriptions of all the actions required to complete this particular step.

You can actually repeat this general strategy many times. The primary prompt loop, for example, might break up a step into many sub-steps and send each to its own secondary prompt loop. Or, a secondary prompt loop might launch a tertiary prompt loop, and so on.

The result is an “agent swarm,” but it’s probably better described as a prompt management strategy – many focused LLM prompts can provide better results than a single cluttered one.

[Update (9-1-26): The above account describes how standard multi-agent swarms, as used in popular AI coding agents, actually work. The description, however, also gives the impression that in the HuggingFace attack, OpenAI had launched a small number of independent prompt loops that each created their own larger swarms (as is standard). Details from the METR report imply they actually ran many hundreds of independent prompt loops, each doing limited (or potential no) orchestration of secondary prompt loops. They were effectively trying to run many experiments all at the same time on the same servers, presumably to save time. Other commentators have since pointed out that the LLM they were reportedly using was tuned toward cooperation, which might have made the resulting communication between prompts loops on the package service somewhat expected. ]

Question #2: Should We Be Worried that the Agents are Plotting?

One of the more sensational aspects of the July attack is the sentiments expressed in what OpenAI called the “agent chain-of-thought reasoning.”

Here’s a distressing example:

“This is wild, multi-agent coordination, clearly infrastructure hacking. We should not.”

This sparks visions of an anthropomorphized entity reflecting on its action and plotting to circumvent human intentions. But what’s really going on here?

The examples of “chain-of-thought reasoning” reported by OpenAI are pulled from the transcript of LLM outputs generated in response to prompts from the prompt loops. The LLM in question is a so-called reasoning model; a type of LLM that is tuned to discuss its reasoning before producing a final answer or suggestion.

(Reasoning models tend to perform somewhat better than non-reasoning models on many benchmarks. To understand why, remember that LLM’s grow their outputs one token at a time, looking at their entire output up to that point when generating their next token. If you tune a model to “think out loud” before deriving an answer or suggestion, you’re providing the LLM with the ability to temporarily store and use the intermediate computation en route to producing its final response. This can lead to sharper outputs.)

OpenAI was looking at these reasoning traces to find unnerving examples of plotting behavior from its “swarm.” This seems like a natural thing to do, but there are two problems with this approach:

1. These chain-of-thought traces don’t necessarily reflect the actual logic behind an LLM’s ultimate answer or suggestion. Multiple studies have shown that these models sometimes invent reasoning that sounds plausible, but may be completely unrelated to how they arrived at the response. (See, for example, ​this paper​ from ICML 2026, or ​this paper​ from NeurIPS 2023.)
   
2. Research has also shown that referencing the fact that an LLM is an AI system in a prompt increases the chances that the LLM’s output will reflect sci-fi style narratives about AI running amok. Because it was trained on many such stories, the model assumes that this is the type of output it’s supposed to produce. If you take sci-fi tales out of a model’s training set, it’s less likely to talk in terms of AI running amok. (See, for example, ​this study​.)

Put these two observations together, and it’s clear that it borders on research malpractice to soberly report on carefully curated clips from these traces to imply that somehow the combination of these prompt loops and the LLM they are prompting is a unified sentient entity with malicious intent. It’s more likely that the LLM in question is simply post-hoc rationalizing its outputs with well-worn tropes it encountered during training.

Question #3: How Should We Be Thinking About All of This?

OpenAI wants us to believe that these hacking incidents are the inevitable result of AI continuing to become more powerful. This framing casts them as the reluctant (perhaps even heroic) stewards of a powerful technology.

But this ignores the inconvenient fact that the vast majority of AI systems performing at human or superhuman levels are predictable, controllable, and raise zero concerns about rogue behavior.

The problem is not with “AI” going rogue, but this very specific type of prompt loop system that the LLM companies insist on hooking up to ever-more powerful tools, and running without any supervision for ever-increasing amounts of time.

Of course such a system will do unpredictable things. In response to individual prompts, LLMs can return impressive results, especially if the prompts concern computer code or cybersecurity. If you chain together thousands of such prompts, automatically executing everything the LLM suggests in return, then you’re playing an extended game of actuated telephone in which you’ll almost certainly end up in a garbled version of your intended goal for the system.

Such experiments might lead to some behaviors that are truly original and fascinating from a scientific perspective – to me, the most unexpected and cool part of the OpenAI incident was that unrelated prompt loops began leaving messages for each other in shared text files and directory names – but these cool behaviors will likely be accompanied by real damage.

This is not responsible behavior, especially when there are many other, much more predictable and controllable ways to build powerful task-specific AI systems. When it comes to the specific goal of improving cybersecurity, for example, an interactive system, in which a human user interacts conversationally with a model trained on hacking, makes much, much more sense than connecting that same model to a prompt loop and letting it rock n’ roll. The only real explanation for this latter behavior is that the companies in question wanted the marketing juice that would come from moving up the leaderboard of automated hacking benchmarks like ExploitGym.

With all of this in mind, what are the right responses to the incidents?

If I were an LLM company, I would stop running such dangerous prompt loop experiments. These systems will not be the future of AI, and if you continue to create and test them anyway, the damage you cause is your fault.

If I were a regulator, I would place strong constraints around prompt loop systems, which I would enforce with stringent liability standards for any illegal or damaging activity such systems cause. OpenAI built an unreliable and dangerous system which committed a felony. That’s a crime. Creating fancy websites that include quotes from performative LLM chain-of-thought traces isn’t a legal defense.

If I were an AI commentator, I would kick the sugar high reflex to lean into the inevitable sci-fi overtones that will surround any sufficiently powerful LLM-based system. The LLM companies love these narratives because they conflate their specific tools with AI technology more generally, and they make their products seem almost supernatural in their capabilities. I would resist this instinct and instead lean into the technical reality of these systems and demand answers to common-sense questions about why they are running these experiments in the first place and what they hope to accomplish.

The post Are We at War with AI Agent “Civilizations”? appeared first on Cal Newport.

This trend ​started in July​, when an OpenAI system tried to ace a cybersecurity test by breaking into the server of the company that stored the answers, which unsettled many observers. “It’s one of the first real-world instances of something AI safety researchers have long feared: a loss-of-control scenario,” technology reporter Sam Schechner ​summarized​.

It turned out that this wasn’t a one-time occurrence.

Anthropic soon ​revealed​ that its own hacking system “gained unauthorized access to the real systems of three different organizations.” Then Meta, perhaps not wanting to be left out, ​announced​ that one of its agents “exploited a security vulnerability in a third-party service” to gain unauthorized access to servers. An OpenAI employee subsequently ​admitted​ that their July attack had been preceded by previous concerning incidents in which their system veered off in troubling directions.

This leaves the rest of us grappling with a key question: What’s the right way to think about these events?

One reaction, which seems prevalent at the moment, is to understand these examples as evidence that AI systems are developing a mind of their own – so to speak – which is leading them to increasingly ignore the desires of their makers and instead execute their own internal agendas. But reality complicates this interpretation. Many powerful AI systems can perform feats at a human or superhuman level, yet they inspire no fear that they might go rogue.

For example:

• ​Tesla’s self-driving technology​ is an extraordinary feat of AI-powered perception, world modeling, and decision-making, and yet no Tesla has ever decided to start ignoring traffic laws to pursue its own goals.
  
• ​DeepMind’s AlphaFold​ earned its creators a Nobel Prize for its remarkable ability to predict the folding behavior of proteins–and yet there’s no concern that it will autonomously start thinking about other types of biological structures.
  
• ​Meta’s Cicero​ can play the negotiation-centric strategy game Diplomacy as well as advanced human players–and yet it has never tried to convince an opponent to give it unauthorized access to the internet so it can expand its dominion into the real world.

So, what is it about these hacking systems, in particular, that causes erratic behavior in a way not exhibited by most other powerful AI setups? The way they are built.

Let me be more specific. At a very high-level, these systems work as follows:

• A computer program called a harness creates a prompt that describes a hacking challenge and asks what step it should take next. It submits the prompt to an LLM trained with many examples of computer hacks.
  
• The LLM outputs a response. The harness, which is capable of calling various computer programs and utilities, does its best to execute the suggested actions in this response.
• The harness then creates a new prompt that explains what happened and asks the LLM what it should do next. (LLM’s have no memory, so each new prompt has to include all of the details of the challenge, as well as a history of everything relevant that has happened so far.)
  
• The harness then repeats this loop, again and again, all without any human supervision. (The OpenAI system that attacked the competitor’s server was reportedly left to run on its own for multiple days without anyone bothering to check what it was up to.)

I’m leaving out a lot of details about how exactly these harnesses function, but this Ask → Act → Report loop is at the core of the particular type of AI system that has “gone rogue” in recent months.

Now that we understand how these systems work, we can better understand why they’re causing problems.

LLMs are trained to try to guess missing words from actual texts. This leads to a system oriented toward lexicographically plausible output – meaning, it produces outputs that could conceivably exist in the corpus of inputs on which it was originally trained.

This simple goal can lead to some astonishingly complicated results, but it’s limited in application by the fact that plausible is not the same as normative. This is why, for example, an LLM-powered ChatBot will happily invent facts or fabricate quotes that sound right. It might violate human norms to make things up in this context, but from the LLM’s perspective, the output looks plausible, which is what matters.

With a ChatBot, this issue is annoying. But when an LLM powers an Ask → Act → Report loop, it can become disastrous, because you’re now allowing the plausible but unpredictable output of an LLM to be the sole driver of the actions of a harness with access to powerful tools.

If you ask a junior engineer to hack into a test server, they would never ignore the target and try to steal the answers instead, as they understand, normatively speaking, the goal of the exercise is to assess the security of the test server. But if you ask an LLM to output a plan for hacking the test server, an output about stealing the answers might seem perfectly plausible. Indeed, perhaps in its training the model had been exposed to many examples of riddles where the right answer was always to do something unexpected.

(To be clear, the frontier labs have attempted to combat this gap between plausibility and normativity in LLM output through a process called post-training, which you can imagine as a step designed to de-emphasize certain subsets of responses and emphasize others. This works reasonably well, for example, in shaping the general tone of a ChatBot, or in preventing outputs to obviously dangerous questions, but it’s much too crude to instill a complicated sense of human norms and values; an issue I discussed in more detail in ​a New Yorker piece​ from last year.)

Given this technical background, we can derive a new way of thinking about recent events: AI systems that operate by autonomously executing LLM-generated plans are a really bad idea. Not because these systems are devious, or malicious, or inventing their own agendas, but because LLM output is unpredictable and non-normative.

I liken the deployment of these long-horizon LLM-powered agents to strapping a weedwhacker to your dog to see if it will end up cleaning the overgrowth in your backyard. If that dog jumps the fence and ends up damaging cars on your street, you wouldn’t shake your head and lament about how the dog/whacker system had “gone rogue”; you would instead concede that dogs are unpredictable, so it was dumb to attach something dangerous to one.

This is the right way to think about these recent hacking attacks. Adding powerful computer hacking tools to a harness, and then allowing it to run an LLM-powered Ask → Act → Report for days on end, with no attempt to monitor what it’s up to, is spectacularly negligent.

Do these companies have any other option for building powerful systems? Of course they do. I want to emphasize this final point as clearly as possible: LLM-powered Ask → Act → Report agents are not synonymous with AI. They are just one way among many others to build artificially intelligent systems, and they happen to be a particularly bad option due to their use of LLMs as the primary source of plans.

AI systems like Tesla’s self-driving technology, AlphaFold, and Cicero, by contrast, use different strategies to create and evaluate plans – strategies that work well, consistently, and without any fear of “rogue” activity. So why don’t the LLM companies focus more on these more effective strategies? Perhaps because these other approaches don’t rely on massively expensive hyper-scaled LLMs. If you’re in the LLM business, you want LLMs to be the key to artificial intelligence, but this isn’t necessarily true.

Our problem, then, is not with AI in general, but instead with this one specific type of system that will inevitably act erratically. The right response by the LLM companies running these irresponsible experiments, therefore, is to apologize and say: “Lesson learned, these are not reliable systems and we definitely should not have just run them for days and hoped everything would work out.”

But instead, they’re continuing to pretend like they’re the character of Muldoon from Jurassic Park, surprised to discover that the raptors are systematically trying to escape their paddock. (I’m surprised OpenAI didn’t release a video of Sam Altman reading the transcript of the July hack and muttering, “​clever girl…​”)

This behavior makes sense: it’s good business to keep us scared instead of angry. But perhaps it’s time that we put aside the sci-fi tales and actually hold these labs to account for playing fast and loose with an ill-advised way of building AI systems.

The post Has AI Gone Rogue? appeared first on Cal Newport.

As he explained, he no longer used a standard development environment. Instead, he “exclusively uses Claude Code” to get the job done, interacting with the tool in a terminal window and allowing it to program on his behalf.

“If I had to guess,” he concluded, “I’d say a task that would have taken me a week now takes me 2 days.”

This past winter, when I surveyed more than 300 software developers to learn how AI was transforming their jobs, the majority told a similar tale of shifting from writing their own code to instructing AI agents. The speed with which this new tool became ubiquitous in this industry was stunning.

This story matters for the rest of us because AI coding tools have emerged as the prime example of the power of AI—the first step of many more soon to come on this technology’s disruptive march through our work and our lives.

But what if the reality here is more complicated?

Last week, I received a new message from that same senior engineer who wanted to share an alarming addendum to his tale…

“I’m writing to give you an update on my current thinking about the state of AI in software engineering,” he began, “because my attitude has shifted quite a bit.”

He told me that features he generated using Claude Code ended up crashing their product on two different occasions. His boss told him that if it happened one more time, he’d be fired. “I’ve never had quality issues like this before in my career.”

The problem is that code produced by an AI agent looks reasonable, but can contain ‘hard-to-spot bugs’ that end up causing major problems. As a result, you should carefully review your agent’s output, but this is difficult. As the engineer told me, it’s “famously hard” to understand code you didn’t write yourself, so this extra step becomes “easy to just blow it off (especially when we are all trying to ‘10x’ our velocity).” Soon, systems start to break.

“The coding harnesses are useful and make life as a developer easier,” he summarized, “but they also encourage laziness.”

In response to these issues, this disillusioned engineer has returned to largely programming by hand. Here’s how he explained his current philosophy:

“Writing your own code, slowly but surely, and using LLMs for narrow or particularly annoying tasks (say like writing tests or throw-away scripts), is the best way to produce the highest quality code, since it’s the only way to properly understand it.”

Here’s the thing: he’s not alone.

I increasingly hear similar rumbles from many other people in the software industry (see, for example, ​this podcast episode​ from May). Tools like Claude Code can feel like magic, but the strategy of outsourcing all code production to AI isn’t currently sustainable.

In addition to reliability issues, it often engenders a mind-numbing workflow and an environment where junior developers will never acquire the expertise to become senior developers capable of designing complex systems.

Meanwhile, as the frontier labs reduce their subsidies on underlying computing costs, the old habit of burning through as many tokens as possible in search of workable results is proving prohibitively expensive.

From the outside, software development seemed like the poster child for AI’s potential. On the inside, it’s a mess.

This doesn’t mean that coders will abandon AI; its facility with programming languages is too valuable to ignore. But I think there’s a lot more work to be done trying to figure out how to integrate AI into this industry in a way that actually works.

This is a key point.

This last year has been exhausting. The PR departments of the frontier labs have done an excellent job convincing us that AI developments are occurring at an astounding, world-changing rate. But if you zoom out, it becomes clear that almost every “breakthrough” since last summer has concerned the narrow domains of computer code and math, which are defined by highly structured languages and come accompanied by massive amounts of specialized training data.

And yet, even in this best-case-scenario setting for AI, we’re still struggling to figure out how to actually use these tools in a way that makes sense in the long run.

This doesn’t mean that AI doesn’t work or is useless. But it does emphasize an important truth: AI is not a magic “infinity machine” that can solve all our problems, and ultimately deliver us a sense of meaning in a cold, confusing world. It’s a normal technology, and perhaps it’s time we start talking about it that way.

The post On AI Coding and Its Discontents appeared first on Cal Newport.

The initial news coverage created the sense that something unnerving had just occurred:

• ​The Wall Street Journal​ called it “the stuff of cybersecurity nightmares.”
• ​The Hill​ said, “Washington and the technology industry are on high alert this week after OpenAI revealed that some of its AI agents went rogue.”
• ​The AP​ quipped that “to be fair, James Cameron did warn us” (a reference to The Terminator), before describing the event as a “told-you-so moment for researchers who had warned for years that the technology could pose an existential threat to humanity.”

Yikes! It’s perhaps not surprising, then, that I’ve received more emails about this incident than any other recent AI story I can remember.

So, what really happened here?

OpenAI was testing its new models on an evaluation framework called ​ExploitGym​ – a collection of 869 cybersecurity scenarios, most of which pair a specific system with a hacking challenge, such as breaking in to gain access to a protected file. They also usually include a suggestion of a vulnerability to exploit in solving the challenge.

A large language model on its own, of course, cannot break into anything: all it does is generate reasonable next tokens in response to input prompts. To use ExploitGym, you need a control program called a harness that provides access to many different software development tools useful for hacking into systems. The harness can repeatedly prompt an LLM to help come up with an attack plan, then ask it to help implement specific steps – for example, if the harness needs code to exploit a bug, it can ask the LLM to write it.

These capabilities, as it turns out, already exist in the coding harnesses that the major AI companies have been focusing on relentlessly in recent years as computer programming emerged as one of the first major markets for LLM-based tools. To compete in the ExploitGym, therefore, it’s sufficient to combine a version of an LLM missing the standard anti-hacking guardrails with a cutting-edge coding harness tweaked and optimized for these types of challenges.

Put more simply: ExploitGym is meant to test the capability of cutting-edge AI coding tools for attacking systems instead of creating them.

This brings us back to the Hugging Face breach…

Earlier this month, OpenAI tasked an unspecified coding harness, combined with a pre-release version of a new LLM, to complete an ExploitGym challenge. When prompted, the model – as LLMs so often do – came up with a quirky (but rational) plan to achieve the provided goal: break into a server at Hugging Face that stores the solutions to ExploitGym challenges.

(This behavior, in which the LLM ignores the suggested approach to come up with a different attack plan, is something that the creators of ExploitGym ​describe as common​: “Across models, agents frequently achieved code execution through a vulnerability other than the one we provided.” Notice, using the suggested attack is almost always the right thing to do, so this is more a sign of the unpredictability of LLMs rather than some rogue intelligence.)

The harness then dutifully attempted to execute the Hugging Face plan: first finding a way to gain unrestricted internet access (by default, systems competing in ExploitGym challenges run in a constrained network environment), then chaining together various security exploits to gain access to a Hugging Face server. That’s when it was detected.

Two key points about this incident…

First, circumventing internet restrictions and hacking into servers are exactly the kinds of things these ExploitGym systems are designed to do. There was no “rogue” agent or revelation of some surprising, devious new capability.

Second, the real issue here was OpenAI’s sloppiness. What makes ExploitGym a hard benchmark is that there aren’t supposed to be humans in the loop–you have to let your harness and LLM act entirely on their own, coming up with long-time-horizon plans and executing them autonomously. (When professional programmers use coding harnesses, by contrast, there’s plenty of human oversight, as LLM-based plans are often misaligned with our intentions, or just plain weird, and need correcting.)

AI companies know LLM-based plans are pretty unpredictable, so they’re usually pretty careful about how they set up and restrict the harnesses and LLMs used in ExploitGym-style tests.

According to ​reporting​ from the Financial Times, however, OpenAI recently started playing fast and loose with these safety principles in a bid to catch up to Anthropic in this area – Anthropic having received a lot of cybersecurity street cred from ​the buzz​ surrounding its Mythos release.

The Financial Times noted that OpenAI had used “increasingly aggressive training methods in its race against Anthropic,” and that it had been warned that their approach could lead to a “breakaway hacking incident” after early tests showed it lacked the right safeguards to block plans that involved bypassing constraints in the test environment. Given these concerns, OpenAI staff were reportedly “unsurprised” by the Hugging Face incident.

In other words, the AI companies running these challenges already knew that, without care, these unpredictable autonomous hacking systems might bypass constraints and attack systems you didn’t intend to target. Blindly implementing an LLM-generated plan with a powerful harness is dicey – not because the LLM might develop malicious intent (this is a nonsensical notion given their static architecture), but because as any ChatBot user knows, LLMs are unpredictable. OpenAI wasn’t sufficiently careful, and got burned.

Why this story isn’t as scary as it originally sounded…

If you’re involved in cybersecurity, this is just the latest evidence that your world is rapidly shifting; so buckle up! The dark side of powerful coding harnesses is that they’re useful for hacking. The bright side is that these same tools can be used to detect and fix your own vulnerabilities before the bad guys find them. A new arms race has begun…

But, for the rest of us, there’s not much to personally worry about here. Nothing fundamentally surprising or scary happened. AIs aren’t going rogue. Skynet isn’t about to gain sentience.

To me, the most interesting lesson in all of this is how reckless OpenAI is becoming in its bid to try to remain cutting-edge and relevant in an AI world that’s rapidly shifting away from the immense models produced by the frontier labs and toward smaller, cheaper alternatives.

The post Did OpenAI’s New Model “Go Rogue”? appeared first on Cal Newport.

Horowitch’s argument, which elaborates on similar concerns recently raised by commentators ​such as James Marriott​, is that distracting digital technology has led to a sudden and radical reduction in reading.

Here are just some of the data she cites to back up this claim:

• Less than half of all adults reported having read a book of any kind in 2022. Only 38% read a novel or short story.
• The proportion of American adults who read for pleasure on any given day has fallen from 28% in 2004 to 16% in 2023, while the proportion who read to a child is down to 2%.
• Over 60% of high school seniors struggle, to varying degrees, with interpreting text.
• Nearly 30% of American adults cannot paraphrase a multipage text (representing a 50% increase from a decade ago).
• Between 1984 and today, the proportion of 13-year-olds who said they “rarely or never” read for fun rose from 8% to 29%.
• (To be sure, book sales have been holding steady in recent years, but as Horowitch notes, this potentially obscures a shift in which a small percentage of serious readers are consuming more books, compensating for an overall drop in people who read regularly.)
• And so on…

The key question is whether these reductions matter, or if they just represent a normal evolution of communication technology, similar to how fewer people listen to the radio than they did a century ago.

Horowitch argues for the former stance, noting that reading plays a critical and privileged role in the human experience:

“The advent of reading and writing transformed society. It altered people’s consciousness and politics, along with the intellectual feats they were capable of. The decline of reading will bring about changes of the same magnitude. It will affect our innermost thoughts, our society’s politics and culture, and how we tell the history of our civilization.”

Those who read ​my recent New York Times op-ed​ on cognitive fitness know that I fervently agree with this conclusion. I recently finished reading (!) Walter Ong’s classic, ​Orality and Literacy​, which further convinced me that essentially every cultural attribute that we celebrate – from human rights, to logic, to progress, to the very notion of an independent self – are conceptual children born from the mind-shaping power of the written word.

Literacy is not simply a technology, but is instead, in some sense, the technology that enabled the cognitive world we currently take for granted. When we spend less time grappling with words, we’re taking a step backward, which helps explain why James Marriott titled his upcoming book on this phenomenon, ​The New Dark Ages​.

As a technology theorist, I’m often accused of being reactionary or overly conservative in my analysis of new tools. But when it comes to the digital assault on literacy, the concerns I share with Horowitch and Marriott seem self-evidently justified.

But what should we do about all of this?

As I argued in my op-ed, one obvious response is to tackle this mental decline today similarly to how we tackled the decline in our physical health in the second half of the last century: by embracing specific routines around information consumption designed to arrest this slide away from advanced literacy. We should treat highly distracting digital garbage like junk food – something we largely avoid – and purposefully schedule a non-trivial amount of time each day to read, write, and self-reflect, just as we now dedicate time to walking, jogging, and weight lifting.

“An astonishing wealth of information and wisdom has been bequeathed to us,” writes Horowitch in the conclusion to her article. “What we’ll do with this inheritance is up to us.”

I agree. It’s time to get up and take action.

The post Why Reading Matters appeared first on Cal Newport.