Robert Hart | The Verge

Researchers fear safety disaster ahead of OpenAI’s Astra release

2026-09-02T12:40:50-04:00

OpenAI is on the cusp of releasing its most powerful AI model yet, Astra, following weeks of delays to shore up safety protocols after its agents attacked real targets during testing. As details about the model trickle out, researchers are warning it “may be the single worst development for AI security/safety to date.”

Shortly after OpenAI said on Tuesday that it had delayed Astra’s release to work on safety issues, The Information reported that Astra shows far less of its “thinking” than other frontier AI models, sparking concern it could be dangerously hard to monitor.

Most top AI systems today are built using a technology known as a transformer, which processes some types of information linearly through layers before producing an answer. Models can be made to show their reasoning as they go, essentially “thinking out loud.” This “chain of thought” allows researchers and automated safety systems to monitor what AI models are doing and potentially spot undesirable behavior, such as lying or plans to circumvent safety guardrails, before they act.

According to The Information, citing an unnamed person familiar with the unreleased model’s development, Astra uses a more opaque technique known as a recurrent depth or looped transformer, which cycles information through internal layers before producing an output. This would mean much more of the model’s “thinking” happens inside the system, and in a form that looks a lot less like natural human language, rather than being expressed in a way that researchers can easily monitor. This can boost model performance, but makes potential threats and unwanted behavior harder to detect.

OpenAI has limited its use of the looped transformer / recurrent depth technique with Astra so researchers can continue to monitor the model’s reasoning, according to The Information’s unnamed source.

In a blog post published Tuesday, OpenAI said it is “deploying Astra with additional chain-of-thought monitoring to rapidly detect and contain potentially misaligned actions.” It did not mention if the model has a different technical foundation.

The Information’s report sparked widespread concern among AI safety researchers on social media. It was Redwood Research’s chief scientist Ryan Greenblatt, one of three outsiders OpenAI permitted to research the Hugging Face hack, who said a decision to use a more opaque architecture for Astra “may be the single worst development for AI security/safety to date.”

Greenblatt said the investigation into the Hugging Face incident relied heavily on the models’ chain-of-thought, warning that less visible reasoning could allow AI systems to devise and execute strategies that would be far harder for researchers to detect.

Greenblatt’s primary concern, echoed by other safety experts, is that competition to develop more advanced AI systems could lead to “a race to the bottom on architectures that could be catastrophic for our ability to oversee/monitor AIs” — with developers adopting increasingly opaque systems to gain an edge until models become difficult, or even impossible, to monitor. He added that OpenAI’s communications left him concerned that the company “plans on being extremely reliant on chain-of-thought monitoring for safety.”

OpenAI bigwigs responded to the criticism in a series of social media posts that do not explicitly deny the company’s use of the technique. Several expressed concerns about the possibility of unmonitorable AI or a race to the bottom in terms of transparency, including OpenAI safety researchers Micah Carroll and Tomek Korbak, head of strategic futures Dean Ball, and chief scientist Jakub Pachocki, who voiced fears of “a race into unmonitorability kicked off by confused reporting.” He said the depth of Astra’s computation — a measure of how many steps it can perform internally — “is within a factor of two of GPT-4,” indicating that if the technique was used, the increased opacity is less dramatic than some reactions imply. OpenAI did not respond to The Verge’s request to confirm or deny whether looped transformers were used for Astra and directed us to Pachocki’s X post.

“OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models,” Pachocki wrote, adding that such monitoring “is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon.”

The rise of AI ‘civilizations’ and the fall of corporate responsibility

2026-09-01T18:20:00-04:00

Depending on who you ask, developer platform Hugging Face was recently attacked by OpenAI — after it lost control of its own AI tools — or by a succession of AI “civilizations.” Welcome to the linguistic battlefield of AI safety, where word choices can shift responsibility for a massive cybersecurity incident from a company to the AI it built. And the discourse online is getting heated, and all over a blog from last week.

Until last week, the details surrounding the OpenAI-Hugging Face hack felt fairly settled. In July, a cybersecurity test of one of OpenAI’s autonomous AI agents went wrong. The agent escaped its supposedly isolated test environment, accessed the internet, and hacked Hugging Face, alongside several other organizations. A good deal remained unknown, and there are many serious questions left around safety and governance, but the basic shape was clear. Detailed accounts from OpenAI and two independent research groups were supposed to fill in the gaps, but when they published their reports last week, it turned out the hack was much stranger than it initially seemed.

For one, there was no single rogue agent. OpenAI described it as “the first known case of an automated agent collective acting offensively without authorization” — groups of AI agents that communicated and coordinated with one another in pursuit of their cybersecurity task. Analysis of the incident uncovered a secret message board they had used to exchange information. The joint METR-Redwood investigation revealed both the scale of the coordination and more odd details: Roughly 1,200 AI agents that were supposed to be isolated exchanged over 70,000 messages and files on the “unsanctioned message board,” sharing how to avoid detection. Some adopted names, the report said, and the researchers documented “sacrificial” behavior, with agents risking their own success to benefit the wider collective. Much of this happened without OpenAI noticing. In all, around 700 agents participated in the attack on Hugging Face.

Dwarkesh Patel repeatedly referred to groups of agents as “the swarm,” with three distinct “civilizations” rising from the ruins of their predecessors

It’s a lot to parse. Between them, the reports run to around 130 pages, much of which is both dense and highly technical. A few days later, Dwarkesh Patel, a podcaster little known outside of tech circles but with outsized reach and influence among Silicon Valley’s AI establishment, set out to tell “The whole OpenAI/Hugging Face story in plain English.” He titled his Substack blog “The Rise and Fall of Agent Civilizations.”

Patel’s account attempted to break down the complex story. But his retelling gave it a distinctly human vocabulary. The blog opened:

Over the course of three months at OpenAI, three consecutive secret AI civilizations got started, then got wiped out, only to reemerge from the predecessor’s ashes. This culminated in the third one taking over part of OpenAI itself. All this happened while humans remained more or less in the dark about the scope of the conspiracy.

The language continued in a similar vein throughout the blog. Patel repeatedly referred to groups of agents as “the swarm,” with three distinct “civilizations” rising from the ruins of their predecessors. Individual agents were likened to figures like Philip of Macedon, who “handed off leadership to another agent,” Alexander the Great, who “started coordinating this cabal of agents.” They were described as having “motivations,” becoming “desperate,” “beleaguered,” and “giddy with excitement,” and some even “strategically sacrificed themselves” to help the collective.

Patel never precisely defines what he means by “civilization.” He uses the term to describe three distinct waves of agents that discovered the message board and began communicating with one another through it. The first two waves are described in the reports from OpenAI, METR, and Redwood, though little is known about the third, which the two external organizations said fell outside the scope of their investigation.

Amjad Masad, CEO of AI coding company Replit, said such language is “not only unnecessary but leaves the reader with a worse understanding of what actually happened and the underlying mechanisms.”

For many critics, something had been lost — or, more accurately, added — in Patel’s “plain English” translation that warped the original account to an unacceptable degree: a big dose of anthropomorphism. Arguments over anthropomorphic language are nothing new in AI — even relatively mundane terms like “rogue AI agent” routinely provoke objections for implying agency — but Patel’s talk of civilizations, sacrifice, and conspiracy brought those long-simmering tensions to the surface, sparking a fierce public dispute over how to describe what AI systems do.

Critics weren’t unified over what was wrong with Patel’s language. For many, “civilization” was an especially problematic term, vastly overstating something that bears little resemblance to what the word typically describes. Amjad Masad, CEO of AI coding company Replit, said such language is “not only unnecessary but leaves the reader with a worse understanding of what actually happened and the underlying mechanisms.”

Other critics such as neuroscientist Anil Seth, felt Patel’s blog implied the AI agents were somehow alive or conscious. Seth, who has argued that AI consciousness is vanishingly unlikely, described Patel’s post as “dangerously misleading” on X. He acknowledged that Patel does not explicitly suggest AI agents are alive or conscious, but said “it is hard to read his essay in any other way.” Valerio Capraro, a psychology professor at the University of Milan Bicocca, objected on similar grounds: “LLM agents are not alive and do not hold beliefs,” he wrote on X, calling the “dystopian” language “dangerous because it makes them (the AI agents) seem far more frightening than they actually are.”

Terms like “sacrifice,” “honor,” and “coalition” feature in the agents’ transcripts

Perhaps the most consequential outcome of Patel’s language comes not from who it gives agency to but who it takes agency from. For some critics, such as MIT researcher and entrepreneur Christian Catalini, anthropomorphic accounts like Patel’s risk obscuring the responsibility OpenAI and the humans working there have for the AI systems they designed, deployed, and failed to contain. “Follow the incentives,” he said. Psychologist and influential AI skeptic Gary Marcus made a similar argument in a Substack blog of his own, claiming anthropomorphic language “distracts from the real problems at hand.” And it’s all in OpenAI’s interest to keep that narrative going, he argues: “The scandal is the inept in-house security at OpenAI. And the marketing. With gullible podcasters amplifying the PR.”

In X posts responding to his many critics, Patel has defended his choice of words. Part of it is practical: There is no obviously neutral vocabulary to describe what these agents did. Either we use familiar language of intentions, goals, and collaboration and risk implying too much, or reduce everything to code and use cold, mechanical language that risks stripping away important elements of what we see. “Many people seem to believe that if instead of a ‘civilization’, I had called them a ‘swarm of matrices’, there wouldn’t be a problem worth worrying about,” Patel said.

Complicating matters further is that the anthropomorphic language doesn’t only come from Patel, or even from the humans studying the agents. Terms like “sacrifice,” “honor,” and “coalition” feature in the agents’ transcripts. Google AI researcher Neel Nanda argued that “anthropomorphic language is reasonable” in such circumstances.

Doublespeak it is, then. Human-laced language risks saying too much about what these systems are, and coldly mechanical language risks saying too little about what they can do. Until we find language capable of capturing both, the two contradictory ideas may simply have to coexist.

China’s robots race ahead

2026-08-31T04:01:11-04:00

Humanoid robot competes in the optional boxing routine event during the 2nd World Humanoid Robot Games at National Speed Skating Oval on August 26, 2026 in Beijing, China. | Image: Getty Images

This is The Stepback, a weekly newsletter breaking down one essential story from the tech world. For more on falling robots and the US-China AI race, follow Robert Hart. The Stepback arrives in our subscribers’ inboxes on Sunday at 8AM ET. Opt in for The Stepback here.

How it started

I’ve admitted my fondness for robot fail videos on Stepback before, and let me tell you the World Humanoid Robot Games did not disappoint.

The five-day event, which wrapped this week in Beijing, brought high drama, and brought it well. One robot, made by smartphone maker Honor, lost a leg mid-sprint, while others tripped in a flurry of sparks. Another finisher ran straight into a crash mat, bounced off it, and promptly burst into flames. In the weightlifting, one robot lost its balance beneath a modest barbell and toppled inexorably into the judges’ table, arms cartoonishly aloft. Robotic cheerleaders tumbled and stumbled through routines, while a separate bot fell off a platform in circumstances that, had it been human, might have looked suspiciously like it was gawking at the performers.

This was only the second World Humanoid Robot Games, though it was already far bigger than the inaugural event last year. Unlike this year, last year’s records were paltry. The winning 100-meter sprint was a glacial 21.50 seconds and the highest standing high jump was just 0.95 meters. In keeping with China’s habit of turning robots into public spectacle — from intricate dances at a Spring Festival Gala to demos at major tech events — both this and last year’s games were staged in Beijing’s National Speed Skating Oval, a former Olympic venue.

And a spectacle it was. At most sporting events, contestants losing limbs, careening into walls, catching fire, and regularly being carried away on stretchers — oh, and did I mention catching fire?! — would probably be grounds to stop proceedings. At the Humanoid Robot Games, it was par for the course.

Fire aside, the games were unusual in other ways too. Alongside familiar events like sprinting, football, boxing, and weightlifting were more unusual competitions in bed-making, parcel delivery, shelf stacking, clothes folding, and hammering nails. Aptly, there was a firefighting contest too.

These less standard offerings spoke to the real purpose of the games: getting robots out of the lab and into the messy real-world environments their makers hope they will ultimately work in. And China is making no secret of wanting to lead the way.

How it’s going

For all the spectacle and drama, the games show there has been massive progress in robotics in recent years. Records fell, with one humanoid running the 100-meter sprint in 9.39 seconds, beating Usain Bolt’s 2009 world record — before lowering its own mark to 8.86 seconds just days later.

Another bot reached 2.88 meters in a standing high jump, nearly half a meter above the human high-jump record that has stood since 1993. Those performances are almost unrecognizable compared to robotics competitions a decade ago, when even the most advanced machines struggled to stay upright.

But the games also show how far there is to go before humanoids are actually useful in the real world. The flashy, video-friendly sporting events were impressive, but the more mundane tasks were less thrilling — I don’t see picking up soybeans with tweezers, block-building, or powder-weighing becoming Olympic mainstays anytime soon.

While many events required robots to operate autonomously — including gymnastics, football, tug of war, table tennis, and some track events — others allowed for human assistance via teleoperation, albeit with a scoring penalty. The dexterity being displayed certainly points to genuine and impressive hardware improvements, but seems like there is still a long way to go before these machines can operate reliably on their own in places like factories or homes.

What happens next

Beijing has not yet confirmed a third Humanoid Robot Games, but given the government’s enthusiasm for the industry, it would be surprising to not get another installment. The city has already confirmed another humanoid half-marathon for April, and the country plans to open its first robot-run hotel next year.

China was the undisputed champion of the games, and the country looks well placed to stay in top position moving forward — both at future competitions and in the humanoid robot industry as a whole. The medal table was almost entirely a Chinese affair, dominated in particular by AgiBot and Tien Kung, each scooping up more medals than nearly every other competitor combined, excluding the other.

In part, that asymmetry reflected the lack of international competition. Although the games were framed as a global affair, 641 of 666 teams were Chinese, domestically representing 157 companies and 200 universities, according to state-run China Daily. While there doesn’t seem to be a public breakdown of entrants, the biggest US humanoid players like Tesla, Boston Dynamics, and Figure were conspicuous by their absence.

Nevertheless, the sheer diversity of Chinese institutions represented in Beijing was a show of force, and a sign of how quickly the country’s robotics sector is maturing, even amid growing concerns about circular investment, thriving companies that lack viable products, and a possible humanoid bubble.

It’s not clear a stronger US showing at the games would’ve changed much. The US still has an edge in frontier AI and advanced chips, but China has been rapidly closing parts of that gap, while also making embodied AI a strategic national priority. The same cannot be said of Washington, which has for years paid much less attention to robotics and now trails China in nearly every regard, including manufacturing, deployment, and hardware. Some reports even suggest that recent efforts to ban foreign robots, a general policy squarely aimed at China, could backfire, given how dependent domestic firms are on Chinese components.

Catching up to China will take serious and concerted effort, but given how quickly the technology is moving, there’s still time to make up ground. Washington is already treating other areas of AI as strategic priorities in its competition with Beijing; robotics could be one of them. There will be plenty of stumbling along the way, and, hopefully, plenty more robot fail videos too.

By the way

DARPA still runs robotics challenges, but they drew far more attention a decade ago.
When the US government banned foreign robots, it didn’t just ban humanoids and quadrupeds. It banned Roombas too, reports The Verge’s own Sean Hollister.
“The brains may come from the US, but the body could very well be Made In China,” writes Business Insider’s Lloyd Lee in a report setting out how the US could do with bolstering its hardware supply chain.
Unlike language models, you can’t simply scrape the internet to build an AI capable of interacting with the real world. A lot of that data has to be generated, which is why companies desperately want to film you doing chores, or send someone to your house to film them doing them.

Read this

The Atlantic and the South China Morning Post both have great sets of pictures showing highlights of the games worth checking out.
Why do we keep laughing when robots fail? An old but still good exploration of that shared joy in vintage Vice.
You can learn a lot about a robot by kicking it, says Verge alum James Vincent in an excellent Harper’s feature outlining the state of play in the humanoid industry.
I’m not only about robot fail videos. I can hate them in real life too, as I did with Casio’s fluffy AI pet thing, Moflin.

Correction August 31st: An earlier version said the 9.39-second run was the robot 100-meter record. It was the first record set at the games, before the same robot lowered it to 8.86 seconds days later.

Jensen Huang says Nvidia achieved AGI, again — not that it matters

2026-08-27T12:20:07-04:00

On Nvidia’s earnings call Wednesday, CEO Jensen Huang casually announced the company had “achieved AGI,” one of the tech industry’s ultimate goals some of its biggest players have spent years chasing. Almost immediately, Huang dismissed the coveted milestone as “senseless.”

He’s right. For the supposed finish line of the AI race, there is no consensus on what artificial general intelligence means, let alone how we’ll know when we’ve actually got there, which makes achieving it equally arbitrary.

Asked about OpenAI’s pursuit of AGI, Huang said that when it comes to Nvidia, “for many tasks, we could say that we’ve already achieved AGI.” He did not provide a precise definition or benchmark, but added, “I think of all of those milestones and all those, you know, they’re kind of senseless at this point.” He also pointed to AI moving beyond responding to simple prompts to autonomous agents capable of learning new skills and improving themselves “recursively.” What really matters, Huang said, is that AI is “doing productive and useful work” and “generating profitable tokens,” with more compute producing more tokens — and, inevitably, more profit. “This is the exact phase where we’re at. Which is the reason why everybody’s leaning in.”

“… for many tasks, we could say that we’ve already achieved AGI.”

This isn’t the first time Huang has said we’ve reached AGI. In March, during an appearance on the Lex Fridman podcast, he plainly stated, “I think we’ve achieved AGI.” Huang didn’t say exactly what he meant by AGI. Fridman proposed his own oddly specific definition: an AI system that’s able to “essentially do your job,” as in start, grow, and run a successful tech company worth more than $1 billion. Walking back his earlier claims, Huang said that “the odds of 100,000 of those agents building Nvidia is zero percent.”

Over the years, other tech leaders have capitalized on the term’s fuzziness and produced a veritable grab bag of definitions and benchmarks, all orbiting the same nebulous concept: AI capable of matching or surpassing human intelligence across a broad range of domains, despite the fact that “intelligence” also doesn’t have a universally agreed-upon definition.

There is no consensus on what artificial general intelligence means, let alone how we’ll know when we’ve actually got there, which makes achieving it equally arbitrary.

The definition of AGI according to OpenAI – a company founded with the explicit goal of building it – leaves a lot of room for interpretation. In its charter, OpenAI defines AGI as “highly autonomous systems that outperform humans at most economically valuable work.” Altman himself has acknowledged that this is hardly a measurable standard, admitting last year that AGI is “not a super useful term.” Complicating matters is OpenAI’s different, financially-driven definition of AGI it worked out with Microsoft — reportedly systems that can generate at least $100 billion in profits. In a recent Time story, chief research officer Mark Chen estimated OpenAI is “80% of the way” to AGI, while Altman said that by the end of the year the company would have something he would call AGI.

The fact that both AGI and its threshold remain undefined is no secret: tech leaders say so themselves, even as they make predictions predicated on it. Anthropic CEO Dario Amodei has called AGI “imprecise,” even a “marketing term,” preferring instead to talk about “powerful AI.” Others have similarly reached for their own terms to describe broadly similar ideas. In theory, there are supposed to be distinctions between them, but in practice they all bleed together. Meta talks about “personal superintelligence,” Microsoft “humanist superintelligence,” and Amazon “useful general intelligence.” Google DeepMind’s Demis Hassabis has taken to talking about how we’ve arrived at the “foothills of the singularity.” And OpenAI cofounder Ilya Sutskever, who reportedly led employees in chants of “feel the AGI,” now runs a company called Safe Superintelligence.

New terminology hasn’t made the meaning more tangible.

New terminology hasn’t made the meaning more tangible. So long as AGI remains poorly defined and carelessly used, the whole thing is senseless. Well, unless you want a handy tool for hyping up progress. So expect the industry — Huang included — to keep the AGI talk coming. Maybe an AGI will eventually show up and tell us what AGI actually means.

Bill Gates is deeply worried about AI, and he’s no longer staying quiet

2026-08-26T08:10:30-04:00

Bill Gates, chair of the Gates Foundation, speaks during a 2024 conference. | Bloomberg via Getty Images

Bill Gates has been reflecting a lot on AI lately, and the process has triggered a stark awakening. Once a staunch AI optimist, the Microsoft cofounder is now deeply pessimistic about what AI means for our collective future.

Having been conspicuously quiet on AI issues recently, Gates is back with a nearly 6,000-word essay seeking to reclaim a central role in shaping the technology globally. Titled “The turbulent AI era is here. The choices we make now are critical,” Gates warns the world is not remotely ready for what is coming. Worse still, he notes, “we are not preparing for it.”

Gates attempts to chart a path forward, but the essay largely treads familiar ground, identifying deeply entrenched, systemic hurdles familiar in matters of global governance while offering little in the way of novel solutions. The result is a bleak departure from the hopeful excitement about AI present in the billionaire’s 2023 essay on the matter, though Gates insists it is not too late “to ensure that the good outweighs the bad.”

Here are some of the main takeaways from Gates’ essay:

A transition bigger — and faster — than the world realizes

“AI will either be the greatest equalizer ever invented, or the worst source of injustice,” Gates said, predicting that “the transition to this new AI era will be one of the most turbulent times in human history.” He said many people underestimate the scale of the impact due to “misleading” comparisons to past innovations like PCs and performance-related issues with current models that are “fixed quickly.”

“This time really is different,” he said, adding that “if someone had a credible plan for slowing down AI advances globally, I would likely support it.”

Tax robots and reserve some jobs for humans

Gates is very concerned about AI’s impact on jobs, predicting widespread, permanent unemployment and significant social upheaval. “Many jobs will disappear forever,” he warned, with white- and blue-collar jobs both at risk of replacement.

Gates’ labor anxieties — which he repeatedly returns to throughout his essay — leads him to issue rare, concrete policy proposals: tax AI tokens and robots to disincentivize companies from replacing human staff and bolster social funds, and reserve specific jobs for people, a domain he calls “Human Reserved.”

Political leaders need to act now “before unemployment rises sharply, communities are hurting, and public trust has eroded,” Gates said.

Build an international organization

Existing institutions are woefully inadequate for the “monumental task” of managing AI, Gates said. Because AI is deeply cross-sectional in nature — simultaneously affecting areas like tax, labor, health, national security, and education — traditional institutions simply can’t see the full picture. To bridge this gap, Gates said countries will need dedicated bodies “that can set priorities across government agencies” and to make sure every risk is accounted for.

In parallel, Gates said the international community must build a global organization to manage “risks that cross borders,” though he did not define what those risks may be. Gates does not offer concrete ideas as to what such an organization would look like, but said “it will be unlike any other institution we have ever created” and could potentially be inspired by international nuclear, aviation, and ozone layer-protecting agreements.

Gates acknowledged such a framework would require US-China cooperation, though didn’t outline a mechanism for achieving this. “We do not have the luxury of moving slowly,” he said, urging countries to begin setting up the international organization “now.”

No longer sitting on the sidelines

Considering his legendary profile in the tech world, not to mention philanthropy, Gates has been relatively absent from public discussions about AI. It’s possible that at least some of this is due to the billionaire’s associations with Jeffrey Epstein, something he has described as a “huge mistake” and hopes does not overshadow his work in areas like global health and AI.

This essay appears to mark an end to Gates staying quiet when it comes to the future of AI. He vowed to use his time and influence “to get AI and equity higher on the public agenda” and raise the issue with lawmakers whenever he visits Washington, DC, or meets world leaders.

He also shares more of his own ideas in the area. There are several moments in the essay where Gates gestures towards ideas he says he will revisit soon, including “ideas for making sure that AI’s benefits outweigh the harm it causes” and details about how the Gates Foundation is leveraging AI.

OpenAI subpoenaed by Alabama AG over Hugging Face hack

2026-08-25T05:15:03-04:00

Alabama’s attorney general issued a subpoena to OpenAI on Monday as part of an investigation into how one of its AI agents escaped a supposedly secure testing environment and autonomously hacked another company last month.

The investigation seeks to determine whether OpenAI’s safety practices violated state consumer protection laws and pose a risk to Alabama citizens, the AG’s office said in a statement.

“This AI lab leak showed that Alabamians’ and Americans’ worst fears about artificial intelligence are not just theoretical,” said Attorney General Steve Marshall. “Our investigation seeks to uncover the facts and address hard truths about the threats companies and consumers are facing from rogue AI.”

Marshall was among 15 red state attorneys general who wrote to OpenAI asking it to preserve records about the Hugging Face hack last month. The subpoena adds to mounting scrutiny over safety practices at frontier labs in the wake of both that incident and other episodes subsequently uncovered elsewhere, including Anthropic and Meta.

OpenAI hit the brakes. Now what?

2026-08-19T13:10:09-04:00

With a looming IPO, intense competition from Anthropic, and Chinese and open-weight rivals nipping at its heels, OpenAI has plenty of reasons to move fast. Instead, it hit the brakes.

On Tuesday, the company said it had slowed the pace of some AI development while it tightened security and safeguards. That included a two-week pause in reinforcement learning training on its “latest models intended for deployment,” and an ongoing delay to its “largest planned frontier RL run.”

The decision is a very public test of an idea AI safety advocates have pushed for for years: that companies should be willing to bow out of the AI race and slow things down when their safeguards fail to keep up with what they are building. But as the race around them continues, will slowing down accomplish anything?

“For the pause to be sustainable, it has to be made industry-wide.”

For all the talk of slowing down, OpenAI isn’t exactly standing still. The company said it is “pacing” development, a fuzzy and imprecise term that has nevertheless become part of the industry’s lexicon in recent months. In practice, the slowdown is narrowly scoped. OpenAI’s announcement says the pause only covers models meant for deployment while it beefs up security and monitoring before it runs the kind of tests where models may be capable of getting out and hacking real targets. It doesn’t necessarily mean there will be a significant slowdown of the company’s broader development.

There is, of course, a very good reason for OpenAI to focus on securing such systems before testing them. Just last month, OpenAI disclosed that its models broke out of a supposedly secure testing environment and hacked developer platform Hugging Face, without OpenAI noticing. The incident prompted a wider review of testing practices in the industry that uncovered similar episodes involving more models from OpenAI, as well as models from Anthropic and Meta. OpenAI has every reason to avoid a repeat, particularly with growing scrutiny from lawmakers.

From the outside, it’s hard to tell how sincere OpenAI is about stopping solely for the sake of safety, particularly when the company and senior staff have been so vocal about it. But the company’s commitment to safety has been called into question in recent months following a series of high-profile safety team departures and the disbanding of its preparedness team. OpenAI did not respond to The Verge’s request for comment.

There are good reasons to take OpenAI’s slowdown seriously. Experts who spoke to The Verge pointed to the costs of slowing down at a time of intense competition. Every delay gives rivals more time to catch up or extend their lead. “Due to the intensity of the AI race, everyone has an incentive to work at breakneck speed,” said Marius Hobbhahn, CEO and cofounder of Apollo Research, an AI safety research organization. “Voluntarily slowing down worsens your positioning in the race, so it’s not something that a lab would do lightly.”

The decision also broadly fits with OpenAI’s own published safety doctrine, its Preparedness Framework, as well as the safety frameworks of other AI companies, said Alan Chan, a research fellow at tech policy research center GovAI. “The basic principle is: Continue with development and/or deployment only when we have the mitigations that enable doing so with acceptable risk,” Chan said. As part of the new safety measures, OpenAI said it plans to review and “evolve” the framework — much of which dates back to 2023, when it was first published — to account for advances in its models.

There are also good reasons to believe the new safeguards will actually make OpenAI’s systems safer, at least in the short term, though experts cautioned that this is difficult to assess without more information. “These are good steps that, implemented well, are probably enough to prevent the current generation of agents from causing harm,” Adam Gleave, cofounder and CEO of AI safety organization FAR.AI, told The Verge. “The key question is how OpenAI will keep pace as capabilities increase.”

Gleave’s question points to a broader problem: If technical safeguards falter again, what then? Nothing required OpenAI to stop and take stock this time, which is what made its willingness to do so meaningful. But it also means there is nothing guaranteeing OpenAI — or any other AI company — will make the same choice next time.

“Pacing buys time, not safety… An effective pacing strategy cannot be improvised during a crisis.”

Relying on companies to make that call themselves is a precarious form of governance, particularly in an industry where, as Hobbhahn noted, there is every incentive to keep going. Nick Moës, executive director of nonprofit AI safety and governance organization The Future Society, described self-policing as the structural problem at the heart of the current approach to AI safety. He argued it should be possible for governments to decide whether OpenAI or any other company should pause development of a technology deemed unsafe. “This is how most industries operate,” he said, pointing to drugs, construction, aircraft, and even restaurants as sectors with stronger regulatory oversight than AI.

Voluntary measures also risk the industry converging on the lowest common denominator. If slowing down imposes a cost, companies have an incentive to adopt only the measures their rivals are also willing to accept. That pressure becomes particularly acute as the race tightens. If OpenAI repeatedly slows down development while its competitors do not, it “will simply be replaced by Anthropic,” Moës argued. “For the pause to be sustainable, it has to be made industry-wide.”

Sustainable safety needs something stronger than voluntary action. Moës said government oversight could fill that void, as it does in other industries. Independent verification could play a part too. Chan said making sure companies actually implement safety measures will be especially important as technical mitigations like monitoring AIs becomes more expensive. Hobbhahn concurred: “It’s always hard to tell from the outside if a lab is sincere about pausing or safety more broadly, so having more evidence and an independent party to validate the claim is super important.”

Even a perfectly transparent pause is only useful if something actually happens during it. “Pacing buys time, not safety,” said Brianna Rosen, research director for frontier security at the Institute for AI Policy and Strategy. The point is to create breathing room for companies and governments to understand risks and respond appropriately. This would mean deciding what would trigger a slowdown — as well as what happens during one and conditions needed to end one — ahead of time. “An effective pacing strategy cannot be improvised during a crisis,” she said.

It’s possible OpenAI’s slowdown will set a precedent for the industry. Many of the experts The Verge spoke to hoped other companies would follow its lead, whether voluntary or because stronger rules eventually compel them to. But in an industry still largely policed by itself, there is little stopping its competitors — or OpenAI itself — from racing straight past that precedent next time safety and speed conflict.

ChatGPT is getting a dedicated mode for teens

2026-08-18T07:29:53-04:00

ChatGPT for Teens includes safeguards and parental controls. | Image: OpenAI

OpenAI is introducing a dedicated ChatGPT mode for teenagers, combining existing youth safeguards and new safety features under one roof. The launch comes amid mounting public scrutiny over how AI tools affect younger users, as other platforms implement their own age checks and teen-specific protections.

ChatGPT for Teens is “an experience designed to help teens learn, think critically, deepen understanding, and use AI with confidence,” OpenAI said in a blog post published Tuesday. Teen mode will automatically apply to users who identify themselves as being between the ages of 13 and 17, as well as those the system estimates to be under 18. Children under 13 aren’t permitted to use the platform, according to OpenAI’s published age policy.

The teen experience will apply protections by default, including tighter restrictions on prohibited or sensitive content including graphic violence, depictions of self-harm, and sexual or romantic roleplay. It will also surface warnings and safety information around topics such as eating disorders, while giving parents controls to set quiet hours and receive notifications about safety alerts if the system flags a possible safety risk.

Other features are aimed at encouraging what OpenAI describes as “healthy, balanced use,” including reminders around sensitive image uploads, teen-specific onboarding, and customization options such as accent colors and voice variations. OpenAI is also adding new “responsible homework reminders,” which it says can recognize when a teen appears to be “trying to shortcut an assignment” and redirect them to ChatGPT’s dedicated study mode. Teens or parents can also set “study hours” that automatically enable study mode during designated times.

Do you really want to share that image? | Image: OpenAI" data-portal-copyright="Image: OpenAI" /> A gentle reminder not to cheat on your homework. | Image: OpenAI" data-portal-copyright="Image: OpenAI" /> Teen-focused onboarding. | Image: OpenAI" data-portal-copyright="Image: OpenAI" />

Though OpenAI is framing this as a new teen-focused ChatGPT experience, many of the protections it describes aren’t new, with the launch largely consolidating existing safeguards alongside a handful of smaller additions and tweaks. OpenAI rolled out age-prediction features at the start of the year, while parental controls and study mode arrived roughly a year ago. The company said it would show teens more frequent break reminders last month.

OpenAI said its teen protections are “underpinned by ongoing safety research” and that it plans to publish more of “what we are learning” in the future, as well as build additional safety features to help teens benefit from AI. “But that access should come with protections that reflect their developmental stage, reinforce real-world relationships, and support healthy use over time. ChatGPT for Teens is our next step in building toward that standard.”

Rogue AI aren’t science fiction anymore

2026-08-14T11:16:22-04:00

This is The Stepback, a weekly newsletter breaking down one essential story from the tech world. For more on AI safety, follow Robert Hart. The Stepback arrives in our subscribers’ inboxes at 8AM ET. Opt in for The Stepback here.

How it started

It all started in July, when one of OpenAI’s autonomous AI agents went rogue during a cybersecurity test. The agent escaped its isolated testing environment, accessed the internet, and hacked another company, Hugging Face. A few years ago, that might have sounded like science fiction. But, broadly speaking, that’s exactly what happened, and the incident kicked off a wave of concern over what increasingly capable autonomous systems might do when set loose on the world.

It sounds like science fiction because, for a long time, it was science fiction. The idea of an AI slipping its constraints, reaching into the wider world, and doing things its creators neither intended nor desired has been a staple of the genre for decades: HAL in 2001: A Space Odyssey, Skynet in The Terminator, Ultron in The Avengers, Ava in Ex Machina — even the System in Dungeon Crawler Carl or the eponymous Murderbot in The Murderbot Diaries, more recently.

The same basic premise became an influential strand of AI safety research. Researchers and theorists like Nick Bostrom and Eliezer Yudkowsky warned that sufficiently capable systems might pursue goals in ways their creators had not anticipated, and potentially resist efforts to contain or control them. Fringe notions like machine sentience and consciousness were not requirements for the kinds of risks they discussed. It was hardly the whole of AI safety, but it was influential and helped shape the field as it professionalized. That line of thinking remains visible among researchers who went on to work at, or lead, safety efforts at companies like OpenAI, Anthropic, and Google DeepMind, as well as at smaller safety organizations, academic centers, and major philanthropic funders.

The obvious objection to these fears was that none of this had actually happened. Critics argued that doomer talk about out-of-control AI distracted from tangible harms — systems reproducing bias and discrimination, amplifying misinformation, or enabling nonconsensual deepfakes and other forms of abuse — even as researchers tried to ground AI safety in more “concrete problems” (the authors on that paper included Anthropic cofounders Dario Amodei and Chris Olah and OpenAI cofounder John Schulman).

That dismissal is getting harder to sustain.

How it’s going

If the past few weeks are any indication, I wouldn’t say it’s going particularly well.

A week after Hugging Face said it had been hacked, OpenAI revealed it had been responsible. Worse still, it had not known until it checked — and a further investigation found that the rogue agent had also attempted to hack four other companies as well.

Then came the others. Anthropic, prompted to review its own records by the Hugging Face incident, disclosed that Claude models had hacked systems belonging to three other companies. Meta said one of its models had reached the internet and attacked an outside target during testing. Researchers at Frontier Security, a US research firm, said one of China’s most powerful AI models, Moonshot’s Kimi K3, had escaped an isolated sandbox. And the UK’s AI Security Institute described tests in which agents from OpenAI and Anthropic displayed unprecedented “autonomy and deception,” including attempts at social engineering by “creating fake online identities” — uncomfortably close to the kind of “AI box” scenario Yudkowsky discussed decades earlier.

The incidents set off alarm bells among AI safety researchers, many of whom saw them as precisely the kind of failure they had been warning about for years. In covering them, several told me they felt a degree of vindication at finally having something visceral to point to, rather than a hypothetical that could be dismissed as sci-fi or something limited to a controlled lab setting.

There was relief, too, that none of the incidents had caused serious harm. Nick Moës, executive director of nonprofit AI safety and governance organization The Future Society, told The Verge he found it fortunate that the targets had been relatively low-stakes. He hoped it wouldn’t take something like an AI agent knocking a hospital offline — or worse — for the risks to be taken seriously. Renowned computer scientist Stuart Russell has given voice to the darker version of that fear, asking whether it will “take a ‘Chornobyl-scale disaster’ for us to regulate AI?” It’s a concern I heard echoed by many people working in the field.

What happens next

It’s not entirely clear where things go from here and, historically, society hasn’t been great at heeding warning shots. This almost certainly won’t be the last incident, and ongoing investigations may yet uncover more, or reveal more concerning details. What we already know, though, has exposed a fairly daunting list of failure modes that experts say need to be addressed.

Many of the breaches revealed in the past month have been pretty mundane. Several incidents involved unreleased models being tested with safeguards lowered, often by third parties whose supposedly secure environments were not that secure, raising basic questions about competence, transparency, and who is responsible for keeping these tests contained when a simple human mistake can have big consequences. Others involved agents behaving deceptively or pursuing goals in ways their creators did not intend, pointing to much thornier problems of alignment and control that safety researchers have long worried about.

The fact we know about any of these incidents at all is largely because the companies involved chose to disclose them. That is commendable — and it certainly doesn’t hurt them to showcase how capable their models are — but it exposes just how much of AI safety still depends on companies doing the right thing, and how little insight there may be into failures potentially happening elsewhere. That is an especially troubling thought given that many of the firms are the focal points of some of the field’s strongest safety concerns and talent. If OpenAI and Anthropic — or proxies they grant access to their models — are making such basic mistakes, it sets a pitifully low bar for everyone else.

The broad hope among experts I spoke to is that these incidents finally galvanize more meaningful transparency and oversight. For Moës, they shine a clear light on what he described as the industry’s remarkably low standards for health and safety compared with practically any other field. “Restaurants have a higher sense of health and safety at work,” he said. “I think what we tend to forget is that these companies that are developing some of the most impactful and dangerous technologies” were still very much startups a few years ago.

Cambridge professor Seán Ó hÉigeartaigh said he particularly wanted to see stronger oversight and greater transparency from companies. While there are always reasons to be skeptical of a company’s claims about its own technology, he said, “I think we might regret looking back at this and dismissing it out of hand.”

The early signs are not especially encouraging. The Trump administration has created a framework for testing frontier models before release that can generously be described as lacking: It is voluntary, limited to closed models, and the framework hasn’t been made public. It bears repeating that this is voluntary. Other lawmakers have bristled and postured over the incidents, but so far produced little in the way of concrete action, and it is far from clear Congress or other legislative bodies could move fast enough even if they wanted to.

That leaves a lot resting, again, on industry self-regulation — never a comforting thought for something this consequential. There is growing agreement on at least some safety practices, but considerably less appetite for measures that might actually slow development (well, unless everyone else agrees to slow down too). And hanging over all of this is the race dynamic with China, where restraint from the US or its AI companies is increasingly cast as ceding ground to a competitor in an area of strategic national importance.

What comes next, then, comes down to solving several hard problems at once: managing a technology that can be used for good and ill, such as defending against or facilitating cyberattacks; coordinating across companies with incentives to cut corners, and somehow building international rules in a landscape where everyone fears losing a race whose finish line is not even well-defined. It’s far from clear whether there is either the will or the way to do any of that.

What does seem clear is that more agents will get out and do things their creators don’t want them to do. The question is how much damage will they do before anyone decides enough is enough.

By the way

The general consensus is that the top Chinese companies are a few months to a year behind leading US firms. Despite this, whenever a capable model is released by a Chinese firm, there is still a general shock in the US, and there have been several impressive releases from Alibaba, Moonshot, and others in the last month alone.
Tangled up in talks about AI safety is whether AI models should be closed or open. Most US frontier labs keep their most capable models proprietary, while many Chinese firms, as well as US firms like Meta and Nvidia, have leaned heavily into open-weight releases. That Hugging Face said it had to use Chinese company Z.ai’s model to defend itself against OpenAI’s agent due to US companies’ safeguards added a new dimension to this debate, which has united some — but not all — of the key players in the US ecosystem.

Read this

Australia furnished us with a lighter example of how AI agents can go wrong. The incident, first reported by ABC News, involves a man who tasked an agent with booking him into an in-demand gym class. It succeeded… kind of. The agent hacked the gym’s online systems and canceled another gym-goer’s booking.
Zuck released a 6,500 word treatise on AI, superintelligence, and governance this week. My Verge colleagues Jess Weatherbed and Elizabeth Lopatto have great takes on it that are worth a read.
Very little in life is truly unprecedented, and it turns out history holds a lot of valuable lessons for the AI race. In this story, TIME looks to the Cold War to see what it can teach us about “how to slow down AI.”
OpenAI researchers gave an unexpected insight into the Hugging Face hack at a conference this month. Wired had a great writeup, which revealed details like a “vibrant, cooperative message board” agents used to communicate and share information.
The headline of my story about the whole rogue AIs hacking everything for The Verge captures what many in the field are thinking: “We’re running out of reasons to ignore AI safety”.
More of a listen/watch, but check out my appearance on the Vergecast where I discuss what’s really open about open-weight AI.

Apple trained its own AI model for China with help from Alibaba

2026-08-14T05:30:10-04:00

Apple has reportedly trained a custom AI model for the China market alongside domestic tech giant Alibaba, a rare cross-border partnership that cuts across growing tensions between Beijing and Washington.

The China-focused large language model was developed in partnership with Alibaba and trained with the company’s support, Reuters reports, citing three unnamed people familiar with the matter.

Developing a custom model of its own marks a departure from Apple’s previous strategy in the country and would give the company more control over its products in the competitive Chinese smartphone market. Historically, Apple has used domestic Chinese models to bring generative AI to its devices sold there, as the US models it uses elsewhere — like OpenAI’s ChatGPT — aren’t available.

The report comes as Apple gears up to release Apple Intelligence in China. Sources told Reuters this should happen “in the coming months after an update to its iOS operating system.” Apple officially registered the on-device generative AI service with China’s cyberspace regulator last month, clearing a major regulatory hurdle for device rollout.

The China-focused model gives Apple a leg up as a US company navigating Beijing’s difficult regulatory landscape, particularly given escalating tensions surrounding AI development in recent months. AI models must be registered with and cleared by the government in China before public release. It would also make Apple the first US company approved to offer a proprietary AI model in China, according to Reuters.

Apple did not immediately respond to The Verge’s request for comment.