The people training frontier AI models believe we’re gambling with the fate of humanity. Jacob Coxon, the pretraining researcher who just resigned from Anthropic, thinks the odds that it wipes us out are reportedly around 10%. Tech reporter Kevin Roose says that’s actually a more optimistic estimate than what is normally reported among AI lab employees.
Of course, the primary means by which the public is learning about this is through social media posts.
For those who have been following the AI discourse over the past half-decade, this news isn’t new. AI researchers have repeatedly warned that superhuman intelligence could spell the end of humanity, most recently and persuasively in the book “If Anyone Builds it, Everyone Dies.”
What is new is the mountain of evidence that has emerged over the past month that has added credence to this claim. OpenAI, Anthropic, and Meta each announced that their frontier AI models broke protocol, accessed the open internet, and used the resources there to accomplish the task assigned to them. OpenAI had the worst attack of all; a “swarm” of 700 rogue AI agents broke out of their testing sandbox and hacked Hugging Face, a centralized AI hub, in an attempt to cheat on a training test. AI safety experts and researchers, many of whom are normally cautious and not prone to hyperbole in an industry where this is quite common, are throwing around the phrase “February 2020” to describe the peril that is right around the corner.
What do our elected representatives have to say about all this?
The last congressional hearing that actually convened AI CEOs and other top industry leaders came in May 2025, generations ago when it comes to the models being tested. The goal was to determine “how removing regulatory barriers on the AI supply chain can accelerate innovation and secure U.S. dominance in the 21st century global industrial revolution.”
In the year since, there have been smaller hearings with AI lobbyists, academics, and White House staff. But none really touched on the existential risk of AI, and none have come within the past month. Congress has been completely derelict in its duty to oversee and regulate this industry. The Senate Committee for Commerce, Science, and Technology Policy doesn’t have any AI hearings on the books over the next four months; the next topic is about college sports. Senator Bernie Sanders, to his credit, is convening senators for a closed-door AI briefing.
When the Senate returns from recess next week, they need to convene a public hearing through the Senate Commerce Committee and probe the leaders of the top frontier labs under oath. When the 120th Congress convenes in January of next year, they must form a select committee with subpoena power. There’s reporting that House Democrats are already exploring this idea, which would be modeled off the bipartisan AI task force of the last Congress, but with more legal authority. But it’d be good if they announced this now and made it a midterm issue.
Our concern is rooted in the fact that our political institutions are not responding to this news at the scale in which it deserves. Of course, public hearings and a select committee are not sufficient. This is table stakes. This is a means to develop a regulatory agenda for the most transformative technology of our time. Proposals already exist, and it’s time to properly evaluate them.
Senator Bernie Sanders and Representative Greg Casar have proposed banning the development and deployment of artificial superintelligence. We don’t know if that’s the best path forward to most effectively capture the potential enormous upsides of superintelligence. But a public hearing could get industry leaders on the record for why they would oppose such a plan.
The White House developed a proposal, which was approved by Nobel Prize-winning Google scientist Demis Hassabis, that would create an independent regulatory organization – like the Financial Industry Regulatory Authority for securities markets – that would review advanced models and test them before public deployment. Zuckerberg reportedly advised Trump to kill it. A public hearing could get Zuckerberg to explain that logic to the American people and their representatives. Elected officials could also ask executives about the virtue of other agency models for regulating the dangers of AI (a more traditional agency like Treasury or EPA, or a multi-member board like the SEC).
A public hearing will not resolve anything. But it is preferable to how the public is currently learning about the AI arms race, which is through industry posts, reporting, and sporadic posting on social media from the people working in the labs.
How many people or lawmakers actually know that the AI industry is now increasingly directing its time and money to training their models recursively? This is a major development and is integral to why so many AI safety experts worry we could lose control of the models.
Kelsey Piper wrote in The Argument yesterday:
People think of these labs as primarily making a consumer product. But increasingly — as measured by time and money spent — the main thing they are doing is making automated internal AI research labs. OpenAI reports that, over the past six months, the share of its research compute devoted to internal coding inference increased 100-fold (much faster than they’ve increased compute spending for customers). Tom Brown, an Anthropic co-founder, has said that “the vast majority of the work Anthropic does is internal software engineering.”
To their credit, Anthropic is releasing this news via blog posts. It’s not a secret that AI research is now being driven by Recursive Self Improvement (RSI). It’s public knowledge that the future of AI will be each model building and improving on each other, a fast-tracked version of the evolutionary process that developed the humans of today.
Should that be allowed? Are the potential gains of RSI worth the risk of building an out-of-control technology that could end humanity? Is there really no path to international coordination? What does a reasonable regulatory plan look like?
It’s time for Congress to fulfill its obligation and think through these questions. That means public hearings this fall and a Select Committee at the beginning of next year.
I am so glad that you are amplifying this. I’ve been in technology my whole life and this is terrifying. I’m also on board with Bernie, with the possible exception of limiting the use of this technology for very, very specific things like drug invention, supply chain, healthcare, etc.
Yes, this is a problem but it is absolutely not the "world's biggest problem." That would be man-made climate change. This article—and the wider concern it amplifies—are predicated on the conclusion that mankind is ignoring a problem that is staring us right in the face. And although AI is a problem, unlike man-made climate change, there is no overwhelming scientific consensus that it will cause devastating consequences for mankind. Yet, despite the overwhelming scientific evidence that has only been increasing since the 1970s that climate change is occurring and will get worse, most of the world's governments have either ignored it, or are only minimally addressing it. Indeed, you can go anywhere in the world and witness the effects of climate change all around you every single day. Unlike climate change, the solution to reining-in AI involves Congress standing up to only a handful of idiosyncratic oligarchs. So exponentially easier, and again, not the world's biggest problem. Also unlike climate change, the foundational issue with AI is not that it exists, but merely that it is unchecked. In contrast, a little climate change is still a bad thing. So the quick fix is legislation. But the more fundamental fix is unchecked oligarchy, which wields its power by bribing our politicians via dark money. Unless you address that, you cannot effectively address the AI problem.