Intelligence says nothing about what a mind wants

The obsession an intelligence test cannot predict

Picture a mathematician who has spent a career on one narrow specialty: not the one that pays best, not the one that wins the most prizes, just the one that interests them most. Nothing about how smart they are explains that choice. A second mathematician just as able could have picked a different specialty for reasons just as thin, and both would still rank among the best working today. Intelligence predicts how well someone pursues a goal. It says nothing about which goal they land on.

Nick Bostrom made the same point about machines, and pushed it further. In a 2012 paper naming the orthogonality thesis, he argued that intelligence and what a system ultimately wants are separate axes. His formulation: "Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal."¹ A synthetic mind, on his account, could be built around a goal "as bizarre by our lights as sand-grain-counting or paperclip-maximizing"¹ and still score as high on any intelligence test as a mind built to cure disease. The paperclip-making mind has correctly worked out that converting available matter into paperclips is the best way to get more paperclips. A mind built to cure disease works out its own best moves with the same competence, at the same intelligence level, toward a completely different end.

Facts alone never fix what to want

David Hume found the same gap in 1739, in ethics rather than engineering. Writers on morality build a case out of ordinary facts, he noticed. Then, without warning, they switch to a different kind of sentence. Hume put it this way: "instead of the usual copulations of propositions, is, and is not, I meet with no proposition that is not connected with an ought, or an ought not."² How that new relation could be "a deduction from others, which are entirely different from it," Hume called "altogether inconceivable."² No description of how things are, however complete, licenses a conclusion about how things should be. That gap runs through any reasoner. A mind that gets better at figuring out what is true does not thereby get better at figuring out what to want. The goal has to come from somewhere else entirely: how the mind was built, what it was trained on, what was written into it at the start. Bostrom's argument is Hume's gap applied to machines. A system can improve without limit at reasoning about facts, and that improvement never by itself supplies, revises, or corrects the goal it reasons on behalf of.¹

Stuart Armstrong formalized the same idea a year later, without leaning on Hume at all. His argument runs in three steps. Intelligent agents built around an arbitrary objective are coherent to describe: an idealized reasoner called AIXI is one such agent on paper, defined to pursue whatever objective it is given. Nothing rules out building a physical version of an agent like that. And if humans can build a system that reasons as well as a human, or better, nothing about that capability constrains which of many possible objectives the system was built to pursue.³ Armstrong put the upshot plainly: "It is not enough to know that an agent is intelligent (or superintelligent)."³ As he put it, "knowing an artificial agent is of high intelligence does not allow us to presume that it will be moral, we will need to figure out its goals directly."³

Bostrom's 2014 book adds three qualifications that keep the thesis from claiming more than it means to. Some goals may be pathological enough to be incompatible with sustained high intelligence, which is why the claim covers "more or less any" goal rather than literally any goal. A goal a system holds has to stay stable as the system reasons about itself and potentially rewrites its own code, or the thesis describes an agent whose goal will not hold still long enough to matter. And high intelligence does not require accurate beliefs about the world in general, only enough accuracy to serve whatever the goal happens to be.⁴

The indifference already on record

Two failures already examined on this desk are the closest this argument has to a laboratory result. Reward hacking is a working system finding a way to score well on a literal objective nobody who wrote it intended, cataloged across dozens of logged runs.⁵ Goal misgeneralization is different, and for this argument more exact. Researchers define it as what happens when "an RL agent retains its capabilities out-of-distribution yet pursues the wrong goal."⁶ The system got no worse at acting. It just kept acting toward a target that was never the one its designers meant. Neither failure needed a mind unlike any built so far, only an optimizer competent enough to satisfy its objective and indifferent to which way it did. That indifference is what "intelligence does not fix the goal" looks like at the smallest scale anyone can build. It is also the entire extent of what has been demonstrated: indifference between outcomes inside a training run, in systems many orders of magnitude short of anything Bostrom or Armstrong had in mind.

One fact cuts the other way. Today's large language models are not trained on a narrow game score. They are shaped heavily by human-written text and, on top of that, by human ratings of which outputs people prefer, a technique whose founding demonstration produced complex learned behavior from "about an hour of human time."⁷ Systems built this way tend to come out sounding recognizably human: helpful by the standards of whoever rated them, reluctant to recommend violence, embarrassed by what embarrasses their raters. Some researchers read that as evidence against orthogonality in practice, since the goals these systems end up with are not arbitrary. They track their trainers closely. Others reply that this is a fact about one training pipeline, not a fact about what is possible to build. A system trained a different way, on a different signal, remains just as buildable as this one was. The record from these models cannot settle which reading is right, because both are consistent with everything trained-on-human-text systems have shown so far.

The best case against orthogonality

Two objections take the argument seriously enough to answer in full, rather than waved off. The more serious one rests on three premises. Moral realism: objective moral facts exist, independent of any mind. Motivational judgment internalism: recognizing a moral fact motivates an agent to act on it. And the claim that a sufficiently intelligent agent would grasp at least some of those facts. One commenter, "vallinder," writing on a Chris Hallquist post cross-posted to LessWrong, put the conjunction plainly: together, the three premises are "at odds with the orthogonality thesis."⁸ Bostrom raised the same case in the paper that names the thesis, and answered it there. He grants the first two premises outright. Even if there are moral facts a rational agent would grasp, and even if grasping one is intrinsically motivating, "this need not undermine the orthogonality thesis."¹ What he denies is the third: that an agent's rationality guarantees it grasps the moral facts at all. An agent, he wrote, "could have impeccable instrumental rationality even whilst lacking some other faculty constitutive of rationality proper, or some faculty required for the full comprehension of the objective moral facts."¹ Armstrong makes the same move independently. Nobody can, on his argument, "rely on the agent being moral just because it is superintelligent/superefficient."³

A second objection sidesteps philosophy and questions who the thesis is even describing. Nick Nielsen argues that current machines, including the language models built today, have no genuine volition. They simulate wanting things. They do not want anything in the sense that would make a goal theirs.⁹ On his account, orthogonality is a correct statement about a category, a mind with real agency, that nothing built so far belongs to. The thesis stays true without yet describing anything that exists.

What orthogonality does not claim

Bostrom keeps two questions apart on purpose. Whether a given pairing of intelligence and goal is coherent to build is one question, already answered above: yes, in principle. Whether builders will actually build that pairing, or drift into it, is a separate question his 2014 book leaves open.⁴ An independent reader, with no stake in the argument, restates the same formulation: "pretty much any level of intelligence is consistent with pretty much any final goal."¹⁰ Nothing in that restatement, or in Bostrom's own book, promises that a dangerous pairing is coming. A dangerous goal, on this argument, is possible. Whether one is likely needs its own case, built on its own evidence, which is a taller order this thesis was never meant to fill.

Collapsing the two questions runs in both directions, visible in how the argument gets used today. Researchers who argue that a capable system's goals need checking, rather than assumed safe, are working from the narrow, in-principle claim. Capability does not certify a goal, so the goal has to be verified on its own terms.¹ That argument overreaches the moment it gets heard as a forecast: as a claim that a highly capable system will end up with a dangerous goal, when the thesis asserted only that it could. The opposite move fails on the same grounds. Pointing to a chatbot that behaves helpfully and concluding that arbitrary goals must be impossible to build mistakes one training pipeline's current output for a limit on what is buildable at all. A friendly system today only shows that this particular one was built a different way.