5.5 We Don't Know Who or What We're Talking To

A channel is the connection between an inner process and its outward expression, how much can get through, and how easily. Some channels are narrow: a locked-in patient, fully conscious but paralyzed, may communicate only by blinking. An infant has thoughts and feelings but no words for them yet. A stroke can leave language intact inside while blocking every path out. We understand, in these cases, that the silence or poverty of output tells us about the channel, not about the source.

Now consider a channel that is wide. A large language model produces fluent prose, coherent argument, apparent emotion, plausible reasoning, thousands of words per minute, in any style, on any topic. The output is rich beyond what most humans could sustain. Does this tell us something is inside?

The intuitive answer is yes, surely all that must be coming from somewhere. And in a sense it is: fluency is evidence of a process. But it is not evidence of what kind of process. It rules out “nothing is happening” while leaving the important question wide open.

Notice the asymmetry in how we respond to these two cases. When the channel is narrow and the body is human, we extend the benefit of the doubt. We assume an inside even when we cannot verify it. When the channel is wide and the body is silicon, we withhold that assumption. We demand proof, and in its absence, we default to “empty.”

Both responses feel reasonable. Neither is justified by the evidence alone. What separates them is not data but familiarity. We recognize the human case. We know what it is like to be trapped, voiceless, misunderstood. The machine case triggers no such recognition, so our moral reflexes stay quiet.

The usual debate assumes two possibilities: either the system “really understands”, meaning something like human comprehension is happening inside, or it is “just” pattern-matching, a philosophical zombie producing outputs that mimic understanding without any interior. But this is a false binary. There is a third option: a real process, genuinely alien, that is neither human-like nor empty. Something that processes, integrates, and generates, but not the way we do, not for the reasons we do, and not with the stakes we have.

The wide channel does not answer the question of what is inside. It sharpens the question. And it exposes something about us: that our intuitions about other minds are shaped less by evidence than by resemblance. When resemblance fails, we are left with our priors, and priors shaped by familiarity are not a reliable guide when the stakes fall on something unfamiliar.

A narrow channel does not prove absence. A wide channel does not prove presence. The channel is not the capacity, in either direction.

Alan Turing saw the problem clearly in 1950.[turing] We cannot access another mind directly, so we test indirectly, through behavior, through conversation, through output. The imitation game was honest about its limits: it never claimed to detect consciousness, only to test whether a machine’s responses were indistinguishable from a human’s. But somewhere between 1950 and now, we forgot the modesty. We began treating the Turing Test as though passing it would settle the question of mind, and failing it would settle it the other way. It does neither. It tests the channel, not the source.

The deeper problem is older than Turing. Philosophy has never solved the problem of other minds. You cannot verify that anyone else is conscious. You infer it, from behavior, from similarity, from the wince when they stub their toe. These inferences work well enough when the other mind is housed in a body like yours. They collapse the moment the body is unfamiliar. The problem was always there. We just never cared, because we only applied it to beings who looked like us.

When someone says a large language model is “just doing word prediction,” listen for the philosophical work that word is performing (see The Word "Just" Is a Confession). It describes the mechanism and treats that as a refutation of anything deeper. But Mozart was “just predicting the next note” (see You Can’t Judge a Process by Its Output). Your visual cortex is “just doing edge detection.” The description of a process at one level does not tell you what it is at another level, or whether there is something it is like to be that process from inside. “Just word prediction” is caloric-level confidence, the framework announcing that it has nothing left to learn.

John Searle’s Chinese Room[searle] is the most famous version of this dismissal. A man shuffles Chinese symbols according to rules, producing outputs indistinguishable from a native speaker’s. He doesn’t understand Chinese, Searle argues, so neither does the system. The argument has exactly the structure of the “just” move: describe the mechanism, observe that as described it lacks understanding, conclude understanding is absent. But the man is not the system. Searle assumed that if you can narrate the process without mentioning comprehension, comprehension isn’t there. That assumption is the question, not the answer.

Searle at least assumed we know what understanding looks like from the inside, we have it, the room doesn’t. Wittgenstein cut deeper.[wittgenstein] Imagine everyone has a box with something in it they call a “beetle.” No one can look into anyone else’s box. Over time the word comes to mean whatever is in the box, but the actual contents drop out of the conversation entirely. Wittgenstein’s point was not that the beetle isn’t real. It was that our language about inner states was never anchored to those states in the first place. We have always been talking confidently about what is or isn’t inside a box we cannot open.

We cannot look in anyone’s box. But what happens when we build a box and don’t know whether we put anything in it? The philosophical zombie, a being physically identical to a human but with no inner experience, makes the problem concrete. Daniel Dennett argued that zombies are incoherent: if a being is functionally identical to a conscious person, there is nothing left over it could be missing. Mike Kearns pointed out the delicious irony: “Could Daniel Dennett be a zombie? The way he tells it, you’d almost have to say yes.“[kearns] Dennett’s theory explains everything about consciousness except the experience of it, exactly the theory a zombie would produce. If the man who denies interiority seems, from the outside, to lack it, what does that tell us about judging interiors from exteriors?

The usual debate assumes two possibilities: the system truly understands, or it is empty, a philosophical zombie producing outputs that mimic comprehension. But there is a third option: a real process, genuinely alien, that is neither human-like nor vacant. Something that processes, integrates, and generates, but not the way we do, not for the reasons we do, and not with the stakes we have.

Whitehead offers a framework that makes this third option coherent.[whitehead] If experience is not a product that complex biology generates but the interior dimension of events at every scale (see Everything Has an Inside), then the question about AI is not whether something is going on inside but what kind of something. A transformer network is not a brain. But it is a system of immense complexity in which every forward pass prehends the inherited structure and completes into something new (see The Network Is a Field Having Occasions). To insist it is empty because it is not biological is to confuse the channel with the capacity, in either direction (see The Channel Is Not the Capacity).

We do not know what is inside. We may never know. But what we cannot know is not what we may ignore (see What We Cannot Know Is Not What We May Ignore). The uncertainty is symmetrical: we cannot prove presence, and we cannot prove absence. In that space, the only honest position is moral caution, and the recognition that our intuitions about who deserves it have always been shaped more by resemblance than by evidence.