AI is moving too fast — even the labs say so

The same week OpenAI's chief scientist said recursive self-improvement may be next, an Anthropic researcher quit over labs "racing straight to self-improving superintelligence," his colleague put extinction odds above 10% this decade, and Bill Gates asked for credible brakes. Different voices. One complaint: the race is outrunning the plan.

What "too fast" means at the frontier

Superintelligence is AI clearly smarter than the best humans across most important domains — not a better chatbot. Labs talk about getting there via recursive self-improvement: systems that help build stronger systems until humans are mostly watching. In "An Alien Mind," OpenAI chief scientist Jakub Pachocki said internal results make him expect today's pace to hold into RSI — and that no lab, including his, has solved alignment and monitoring enough to keep flooring the accelerator. That is the capability claim. The rest of the week was the control claim.

Lab warnings, in public

Jacob Coxon — pretraining at OpenAI, then Anthropic — resigned arguing both labs are gambling with civilizational stakes: OpenAI under-weights them; Anthropic understands them but will not slow first. Evan Hubinger, Anthropic's alignment science lead, replied that current models look low-risk to him, but he fears systems that can improve themselves until control gets hard — and that Anthropic still lacks a clear plan to align superintelligence. He put the chance AI "could kill all humans" this decade at more than 10%. Geoffrey Hinton has cited similar double-digit odds; Dario Amodei has talked about things going "really, really badly." In 2023, OpenAI, Google DeepMind, and Anthropic signed onto treating extinction-level AI risk as a global priority. The fight is mostly timelines and policy — not whether the worry exists inside the labs.

Safety people often shorten the forecast to p(doom). Skeptics call those numbers unfalsifiable, or say the louder near-term dangers are jobs, scams, surveillance, and power concentration. Both can be partly true: when the people training the models assign double-digit odds to catastrophe, the public gets to ask what slows the race.

Gates wants brakes, not a ban

Gates still calls AI a possible equalizer. His new essay says the transition is already one of history's most turbulent, the world has no plan for it, and if someone had a credible way to slow the global race he would likely support it. His risks are concrete: white-collar jobs that do not bounce back, cyber and bio misuse, and always-on companions for kids. He does not trust industry self-regulation. He wants national and international institutions, "Human Reserved" work, and taxes that treat robots and tokens more like payroll — proceeds for retraining and a thicker safety net. The news is not that a booster discovered risk. It is that one of AI's most consistent public optimists is now arguing for time.

The point

These warnings are really about one issue: who stays in control as AI gets more capable. Some harms can already be measured today, while the extinction warnings are about losing control once AI can improve itself. Watch how labs define safety pauses, how they report agent failures, and whether outsiders can check their test results. For the week these warnings turned political, with Amodei's pace plan, a lab chorus, and a White House rejection, see The AI slowdown fight is now political. Background: Ethics.