We’re building it. Do we know why?

Commentary
superintelligence
AI
philosophy
policy
The White House just signed a pledge on superintelligence. Nobody in the room could say what it is, or what we want it to do.
Author

Amaka Peace Onebunne

Published

September 29, 2026

Back to Commentary

Today, some of the most powerful people in technology sat down for lunch at the White House and signed a pledge. Trump called it “morally binding,” almost like a constitution. The White House billed the event as a meeting and luncheon on Super Intelligence, and Trump said the technology has now been officially renamed SI.

The renaming got my attention first. I wasn’t exactly sure why that took national priority at a time of heightened international tensions. But perhaps that was exactly the point. To lead, to be first, or to dominate the international race for technological supremacy. Naming it was itself a way of claiming the lead, even before anyone had clearly explained what we were racing toward or why it was worth getting there. I. J. Good imagined an “ultraintelligent machine” back in 1965, one that could design better machines and would be the last invention we’d ever need to make. Bostrom later defined superintelligence in Superintelligence as an intellect that greatly exceeds ours in practically every domain. Nobody at that table could tell you how we’d recognize the day it finally arrives, if at all. But a name does a lot. Calling a technology “superintelligence” tells people to lean in and stop asking questions.

Why do we want it?

The answer you hear most is that it will cure disease and end scarcity. Some people mean that sincerely. The answer you can actually see in the room is different. Trump said whoever wins superintelligence wins. He also argued the government can’t slow development for fear of falling behind China.

That’s a race, and a race isn’t a purpose. Scott Alexander’s essay “Meditations on Moloch” describes the trap where everyone would prefer a slower, safer pace, but no one can afford to slow down alone, so everyone speeds up. Zuckerberg has pushed back on calls to slow down collectively, arguing that every lab has the incentive to move at the pace needed to train models safely. But look at whose pace that is. It’s set by the competition, not by anyone’s judgment about what’s wise.

Rich Sutton’s “Bitter Lesson” explains why the race appears to be so rational. Over seventy years, general methods that scale with compute have beaten clever human-designed ones. So the bet is simple: keep scaling and the capability arrives. What the bet leaves out is for what. We’re arguing about how much intelligence is coming and skipping the question of what it should want.

That question is the philosophical center of all this. Bostrom’s orthogonality thesis says how smart a system is and what it wants are separate questions. It is Hume’s is-ought gap in a new outfit, because nothing about being good at achieving goals tells you which goals is worth achieving. The comforting intuition, that a wise enough mind would naturally be a kind one, has no backing. Stuart Russell makes the practical version of the point in Human Compatible. The standard model of AI, where we specify an objective and the machine optimizes it, is the problem. The failure is not that the machine disobeys, it is that it obeys perfectly.

The strangest feature of this moment is that the warnings come from inside the building. Dario Amodei published an essay this month, “We Must Pace the Frontier,” arguing for a cooperative slowdown, and it set off a fierce debate. People are telling us it may be dangerous, that they’re building it anyway, and that stopping isn’t an option. I honestly do not read that as hypocrisy. It’s what a coordination problem looks like from the inside.

Who polices a mind we can’t read?

Wittgenstein said in the Philosophical Investigations that if a lion could speak, we couldn’t understand him. Nagel asked “What Is It Like to Be a Bat?” and concluded we can’t really know. If we can’t cross that gap with a bat, the idea that we’ll oversee a mind that outthinks us across the board raises some eyebrows. Interpretability research is a serious attempt to close it, but it’s early.

Against that backdrop, the governance looks thin. A constitution binds power because something outside the powerful enforces it. Plainly, I mean courts, voters, institutions that can say no. The tech CEOs said they agreed on a safety accord, but it appeared to be voluntary. A promise you enforce on yourself is a good intention, and that isn’t worthless. It just isn’t a constitution.

There is a precedent for scientists policing themselves. In 1975, biologists met at Asilomar and set voluntary rules for recombinant DNA, and people still argue over whether that was wisdom or a way to head off regulation. Even so, a gene-splicing rule constrains a technique. Nobody knows yet what a rule constrains when the thing being governed can out-think the governors.

To be fair, there are smart skeptics. Yann LeCun argues that language models alone will get us nowhere near human-level reasoning and that we need new architectures that can plan and understand the world. He also says that fears of runaway systems wrongly equate intelligence with a drive to dominate. He might be right. I just notice that “he might be right” isn’t much comfort when the downside is this lopsided.

What does it mean for us?

Hannah Arendt opened The Human Condition in 1958 with Sputnik and a fear that has aged strangely well. She feared a society of laborers about to be freed from labor, with nothing to replace it, having forgotten how to do anything else. Keynes had made a gentler version of the bet in 1930, in “Economic Possibilities for Our Grandchildren,” predicting his grandchildren would work fifteen hours a week and that the hard part would be learning how to live. We got the productivity without the leisure. If superintelligence delivers real abundance, the open question is still what people are for.

Hans Jonas put the stakes best in The Imperative of Responsibility: act so that the effects of your action are compatible with the permanence of genuine human life. He wrote it in 1979, about technology in general. It reads now as if it were written for this week.

I don’t think doom is certain, and I don’t think the optimists have earned their confidence either. What I do believe is that the engineering questions, real and important as they are, sit on top of a philosophical one we’ve dodged for a few thousand years: What do we value? What would we want a mind more capable than us to do? And can we say it, together, clearly enough to hand it to something that will take us literally? Today’s lunch produced a new name and a voluntary pledge. It didn’t answer that question, and I doubt anyone in the room would claim it did.

References

Books and essays

Interviews and commentary on LeCun’s views

News coverage of the White House meeting (September 29, 2026)

Back to top