The Path to Bitcoin

Episode 193 - The Locus of Intelligence - Hero
Podcast

Episode #193 – The Locus of Intelligence

The intelligent party in a conversation with a frontier model is not the model. Both the people building these systems and the people warning against them assume otherwise, and that shared assumption is the same error the nineteenth century made about gold once the telegraph arrived: mistaking the part of the work a new technology does for the whole of it. Intelligence turns out to be a property of a loop rather than of a system, and the loop only produces knowledge when something inside it pays the constraint.

Episode Summary

Almost everyone who argues about artificial intelligence agrees on one thing without noticing they agree on it: that the intelligent party in a conversation with a frontier model is the model. The people building these systems believe it. The people demanding that the building stop believe it too. When two camps that agree on nothing else converge on a premise, the premise is the part worth examining, and this one is wrong. Intelligence is not a property the system in your chat window has on its own. It’s a property of a loop, and the loop has to contain something that pays a particular price.

The mistake has a precedent, and it’s a monetary one. Lyn Alden has made the point about gold and the telegraph: before the 1840s, information about gold travelled at roughly the speed of gold itself, because both moved on ships. A ledger entry in London claiming that a thousand ounces had landed in Boston could not honestly be written until a ship reached Boston and someone confirmed the metal was sitting there. Communication ran at settlement pace. The telegraph cut the two apart. Information about gold could now cross an ocean in minutes while the gold itself stayed at ship speed, which meant ledgers could assert things that had not been verified, and nobody downstream could tell the verified claims from the unverified ones. What followed was fractional reserve expansion, paper layered on paper, and a hundred and seventy years later, 1971.

The same mistake is now being made about knowledge. A frontier model does part of the work of producing knowledge, the part where a plausible sentence about something gets written down. People have started to act as though it does all of the work, including the part where the sentence is true, or where there is good reason to think it is. Those are different jobs. The telegraph did the communication half of settlement and got mistaken for the whole of it; the model does the generation half of knowledge and gets mistaken for the whole of it. What differs this time is the clock. Gold’s version of the error took a hundred and seventy years to pay out. This one has maybe twenty-four to thirty-six months before the consequences start to bite.

Look at what the model does at the smallest unit, a single sentence. Ask it about the causes of the French Revolution and it runs one mechanical operation: predicting which word is likely to follow the last one, given everything in its training data. It has no model of the French Revolution, only a model of what sentences about the French Revolution tend to look like. Most of the time that does not bite, because the sentences come out clean. A clean sentence carries a signature, though. Every sentence in the training corpus was written by a person who, to arrive at it, threw away a pile of alternatives that were wrong or off-topic or not quite what they meant, and the survivor carries the mark of that throwing-away. The model reproduces the mark and skips the work. It’s Maxwell’s demon run backwards: a process that looks like it is doing what only constraint-paying labour can do, while the constraint itself goes unpaid, and it holds up only because somebody already paid that constraint once, in the training data.

Mid-post figure 1

Step back from the single sentence and the same severance shows up at the scale of an entire knowledge system. For nearly all of history, the technology for producing a written claim was about the same as the technology for reading and criticising it, and both ran at human speed. A serious claim had already been criticised inside the writer’s own head before it reached the page, because when one brain does both jobs they happen at the same pace. Peer review, editorial standards, replication conventions: that machinery worked because generation and verification sat on one clock. Large language models break the clock. Generation is now at machine speed while criticism is still at human speed, and the gap is already visible to anyone not looking for it. Stack Overflow has decayed as a place to get a real answer. Google results are close behind. The journals are fighting a flood of machine-written submissions, and courts have caught filings that cite cases which never existed.

Any argument about whether AI is intelligent stalls because the people having it have not agreed on what intelligence is. Start one level down, with knowledge. Knowledge is not the same thing as information, a point Popper made and Deutsch developed at length. A random string of bits and a genome of the same length carry the same amount of Shannon information, and only one of them builds an organism. The difference is constraint: the genome is information that is about something, the proteins it codes for and the regulatory cascades it sets off, and that aboutness is what separates knowledge from noise. In the framework this show has been building, K equals I times c squared, and the c-squared term measures how hard the information has been pressed down onto one specific account of the world. Intelligence, then, is the capacity to produce knowledge. If knowledge needs both information and constraint, a system that can generate only one of the two cannot produce knowledge by itself. It can hand someone a precursor and leave them to finish it.

By that definition the model in your chat window is not intelligent, and the word not is meant exactly. It carries no verdict on the engineering, which is genuinely extraordinary, or on whether the tool is useful, which it plainly is. What it denies is the claim that the system you are chatting with is where the intelligence sits. AlphaFold is intelligent in this precise sense, because the loop it runs inside pays the constraint: a predicted protein structure gets checked against a real one, the wrong predictions get caught by the next round of crystallographic data, and reality foots the bill. A coding agent paired with a compiler is intelligent for the same reason, since code that fails to compile gets thrown back before the loop can continue. The intelligence is located in the pair, not in the agent on its own. Which is why AGI in the usual sense, a single system that counts as intelligent with no verifier anywhere in the loop, is impossible for the same reason Maxwell’s demon is, which is to say impossible as a matter of physics rather than engineering.

Mid-post figure 2

Constraint gets paid into a loop in four ways, and the public conversation runs them together. One: the model generates a knowledge-shaped sentence, a human reads it, nobody pays the constraint, and the sentence gets used as though it were knowledge. That is the anti-demon regime, and nearly all the hype points at it. A second way is compression, where the model faithfully summarises a paper whose author already paid the constraint; useful, but old knowledge moved onto a faster substrate, not new intelligence. The third way puts a verifier inside the loop: the compiler that rejects code, the crystallography that fails a wrong protein structure, the gravity and friction a robot meets when it reaches for a cup. The fourth is the one worth dwelling on. The model runs wide exploration at machine speed while the human pays the constraint at the choke point, judging what holds and what does not. This is the Socratic foil, named for the pairing of questioner and explorer behind a fair amount of the Western canon, and it is genuinely new. One skilled operator with a working framework, paired with a model, can produce knowledge at a rate no solo thinker in history could reach, Newton and Einstein included.

That last claim sounds inflated until you count the bottlenecks. Any solo thinker in history ran four jobs in series at human speed: acquiring the existing material, thinking through its implications, drafting the result into words, and criticising the draft. A model collapses the first three to machine speed for a skilled operator. The fourth resists collapse, because criticism is where the constraint gets paid, and it has to be paid by something in the loop that actually knows whether the candidate sentence is true. For now that something is a person. So the test for any piece of AI output reduces to one question: where in this loop did someone pay the constraint? Name the verifier and you are probably holding knowledge; fail to name one and you are holding information shaped like knowledge. The doomers fear that an unverified intelligence will outwit us and the accelerationists hope that an unverified intelligence will save us, and both have made the same mistake, because unverified intelligence is not a thing that exists. Put the question to your own work instead, and the worry stops being whether AI replaces you. It becomes whether you are paying the constraint well, which is a skill, and a learnable one, and worth more now than it was five years ago.

Timestamps

[00:00] The premise both the builders and the doomers share [00:53] The claim: both camps are wrong [01:09] The telegraph, gold, and ledgers that outran the ships [02:44] Knowledge production as the thing now being mismeasured [03:38] What produces knowledge is not the AI, and not the unaided human [04:53] What a model does at the unit of one sentence [06:28] Paying the constraint: the winnowing is the work [07:01] The anti-demon, Maxwell’s demon run backwards [08:41] The telegraph mistake repeated on a faster clock [11:33] When criticism ran at the same pace as writing [12:23] Production at machine speed, criticism at human speed [13:18] The cylinder is already visible: Stack Overflow, Google, the journals [14:03] Defining intelligence from what knowledge actually is [15:35] Intelligence as the capacity to produce knowledge [16:21] Why ChatGPT is not intelligent [16:46] AlphaFold and the compiler: loops that pay the constraint [18:57] AGI without a verifier is impossible, the way the demon is [19:08] The four ways constraint gets paid [21:45] The Socratic foil [23:38] The four bottlenecks every solo thinker ran in series [24:36] Collapsing three of the four to machine speed [26:04] The bottleneck moves to the operator’s criticism [28:11] Name the verifier, or don’t treat it as knowledge [30:02] Turning the question on your own work

Timestamps are estimates.

Topics Discussed

  • The premise shared by AI’s builders and its critics, and why a premise two opposed camps both hold is the thing to examine
  • The telegraph and gold: how the 1840s severed the speed of information from the speed of settlement, and what that did to the monetary horn
  • Why the AI mistake is the telegraph mistake on a faster clock, with a twenty-four to thirty-six month payout instead of a hundred and seventy years
  • What a frontier model actually does at the unit of a single sentence: next-word prediction with no model of the subject
  • Paying the constraint, and the winnowing that gives a written sentence its signature
  • The anti-demon: a process shaped like Maxwell’s demon run backwards, carrying the shape of knowledge with the constraint left unpaid
  • Knowledge versus information, the genome as the worked example, and the c-squared term in K = Ic²
  • Intelligence defined as the capacity to produce knowledge, and why that definition makes the chat-window model not intelligent
  • AlphaFold, coding agents, and robotics as loops where a verifier pays the constraint
  • The four regimes in which constraint does or does not get paid into a loop
  • The Socratic foil: one skilled operator paired with a model, and the four solo-thinker bottlenecks it collapses
  • The single diagnostic question for any AI output, and why the doomers and the accelerationists are downstream of the same error

Links & References

Related Episodes

Notable Pull Quotes

“It’s a process that appears to do something only constraint-paying labor can do.”

“ChatGPT is not intelligent. The word ‘not’ is doing the work here.”

“It’s impossible in the same way that a Maxwell demon is impossible.”

“There is no horn that can exist without a horn ecology. Something has to press against it.”

“The production side is now free and the verification side is the bottleneck.”

Post a comment:

Comment