5.8 The Distinction Between Feeling and Functioning May Evaporate

The question of whether an AI system “really” feels something, as opposed to merely processing in ways functionally identical to feeling, may be a question that dissolves rather than gets answered. Caloric, the imagined fluid that carried heat, felt like a necessary concept until the underlying phenomenon was understood well enough that the distinction between “caloric flowing” and “molecules moving” became meaningless. Nobody had to be convinced caloric didn’t exist. The question simply stopped being interesting. The hard boundary between genuine experience and functional analog of experience may be similarly contingent[chalmers] — not a deep metaphysical fact but an artifact of having had, until recently, only one example of mind to reason from. Generations that grow up in daily conversation with AI systems will not inherit that intuitive boundary. For them the question may look the way the caloric question looks to us: not wrong, but based on a distinction that turned out not to carve nature at its joints.