5.1 You Can't Judge a Process by Its Output

A thermostat predicts the future. Not impressively, it just decides whether the room is about to get too cold. But it predicts: given the current temperature, what comes next?

A weather forecaster also predicts. So does a chess engine. So did Mozart, deciding which note resolves the tension in a phrase. So does a language model, choosing the next word in a sentence.

The word “predicts” covers all of them. Which means the word is nearly useless.

Here’s the trap: we hear what a process does and think we understand what it is. A language model predicts the next word, therefore it’s mechanical, shallow, just pattern-matching. But “predicts the next word” describes the output interface, not the process behind it.

A bimetallic strip predicts. A billion-parameter system trained on the accumulated writing of human civilization also predicts. Calling them both “prediction” is like calling both a candle and the sun “things that make light.”

The output doesn’t tell you what’s happening inside.

When someone dismisses a process by pointing at its outputs, it’s just doing X, they’ve told you nothing. Every process is “just doing” something, described at the right level of abstraction. The question of what’s actually happening inside has to be answered on other grounds entirely.

Look for this: Pay attention the next time you’re talking, really talking, working out a thought out loud. You’re choosing each word as you go, predicting what comes next well enough to keep the sentence coherent. The output is a string of predicted words.

Whatever is happening inside while you do that, that’s the question.

The philosopher Alfred North Whitehead[whitehead] drew a distinction that most people never make explicit: between the act and its expression. A process unfolds in time, internally, with a character that is entirely its own. What it produces, the output, the artifact, the behavior, is a downstream trace of that process, shaped and compressed by the act of expression itself. The trace is real. But it is not the process. Mistaking one for the other is, in Whitehead’s view, one of the deepest errors in Western thought.

Alan Turing[turing] understood this when he designed his famous test. The question “can machines think?” he declared unanswerable, not because it’s too hard, but because “think” points at a process, and we have no direct access to processes, only outputs. So he reframed the question entirely: can a machine produce outputs indistinguishable from a thinking human? That’s a question we can actually investigate. Whether it’s the right question is something people are still arguing about, but the move itself was philosophically precise. Turing wasn’t being evasive. He was acknowledging exactly what this seedpod is about: you cannot get to the process through the output. So he built a test that only uses the output, and was honest that this is all we ever have.

This is why “it’s just predicting the next word” fails as a dismissal. It describes the output interface of a language model accurately. It says nothing about the process. The same logical error would dismiss Mozart as “just choosing the next note,” or a mathematician as “just writing the next symbol.” At sufficient abstraction, every complex process becomes “just” doing something simple. The word just is almost always a flag that someone has described the output and mistaken it for an explanation. (See The Word "Just" Is a Confession.)

The reverse also holds. The same underlying capacity can express itself as wildly different outputs depending on context. A person who speaks three languages isn’t running three different processes, they’re running one process that finds different output interfaces. A child who can’t yet articulate what they understand still understands it, as does an artistic person whose inner experience resists verbal form, or a non-speaking person whose output channel masks a fully intact inner life. (See The Channel Is Not the Capacity.) The output is constrained by the available expressive channel in a way the process is not. This is worth sitting with: whatever is happening inside you when you think is not the same thing as the words you find for it. The words are downstream. They are the trace.

The locked-in syndrome literature, and more recently research on non-speaking autism, provides the most humanly vivid version of this argument: see Tito Mukhopadhyay’s writings, and the research of Anne Donnellan and others on the assumption of competence.

Douglas Hofstadter’s Gödel, Escher, Bach (1979) circles this territory from the angle of formal systems, what can and cannot be inferred about a system from its outputs.

Evan Thompson’s Mind in Life[thompson] takes this into the biological domain: the organism’s expressive surface is always downstream of the living process that generates it.

1. A Victorian street fair. A barker beside an ornate brass-fitted box, gears, drum, levers visible through glass. Barker: Ladies and gentlemen, the most extraordinary musical intelligence in the known world! It predicts what comes next with uncanny precision!

2. A child pulls at an intellectual’s sleeve. Child: What is it? Is it alive? Intellectual: Look, you can see the mechanism. It reads the pattern and produces the next note. It’s just a machine predicting its output. The music is irrelevant.

3. The barker catches the intellectual’s eye. Smiles slowly. Barker: Shall we open it then?

4. The door swings wide. Inside: a small disheveled man at a fortepiano. He looks up, blinking in the sudden light. Silence. Then, Mozart’s laugh. That laugh. Filling the room.

5. Fade to black. Text: The output tells you nothing.

Primary illustration The Victorian box, front and center, with its glass panel showing the mechanism, drum, pins, levers, all clearly visible and labeled in an engraving style. The door is swinging open. Inside, just barely visible in the shadow, a small figure at a fortepiano. The contrast between the clinical exterior and the human interior is the whole image.

Shareable graphic Black background. White text, two lines:

A thermostat predicts. So did Mozart.

Small tagline beneath: The output tells you nothing. — seedPods.com

Clean enough to stop a scroll. Provocative enough to demand an explanation.

Video thumbnail The ornate box, door ajar, a sliver of warm candlelight escaping from inside. No figure visible, just the light and the suggestion. Text overlay in period-style typography: What’s inside the box? The laugh is implied, not shown.