Are LLMs perception-based modal or non-perception-based amodal conceptual systems?

Large Language Models (LLMs) are fundamentally non-perception-based, **amodal conceptual systems** in their core architecture and training mechanics, though they frequently learn statistical approximations of modal structure. In cognitive science, modal (or grounded) systems posit that concepts are intrinsically tied to sensory-motor simulations of the physical world. Conversely, amodal systems treat concepts as abstract, arbitrary symbols processed through formal relations, detached from any specific sensory modality. Because standard LLMs are trained exclusively on next-token prediction over disembodied text corpora, they lack sensory organs and direct physical interaction. However, because human language itself encodes rich statistical footprints of perceptual experience, LLMs construct high-dimensional vector spaces that capture relational properties resembling modal cognition, blurring the historical boundary between pure amodal symbol manipulation and grounded representation. ## Theoretical Framework: Modal versus Amodal Systems To evaluate where LLMs fit, one must distinguish the two primary paradigms of conceptual representation in cognitive psychology: * **Modal (Grounded) Conceptual Systems:** Championed by theorists like Lawrence Barsalou, this framework argues that semantic knowledge is reenacted across the brain's sensory-motor modalities. Understanding the concept of a "kick," for example, involves neural simulations in motor and premotor areas. * **Amodal Conceptual Systems:** Tracing back to classical computationalism and Jerry Fodor’s Language of Thought, this approach posits that concepts are represented in an abstract, language-like format distinct from sensory inputs. Symbols derive their function from formal syntactic rules rather than experiential grounding. When applied to artificial intelligence, these frameworks intersect with Stevan Harnad’s classical "symbol grounding problem"—the dilemma of how arbitrary computational symbols acquire meaning without external referents. ## The Architecture and Status of Large Language Models Unimodal LLMs operate entirely within a linguistic manifold derived from the distributional hypothesis—the linguistic principle that words occurring in similar contexts tend to have similar meanings. Structurally, this aligns them with amodal systems. They do not possess a body, navigate physical space, or experience interoception; their tokens are indices within a vast mathematical matrix rather than reactivations of sensory states. Yet, treating LLMs as *purely* amodal overlooks how thoroughly human language mirrors perception. Language does not float entirely free of reality; it systematically records perceptual regularities. Consequently, an LLM’s hidden layers develop geometric topologies where semantic distances mirror physical and perceptual realities. While the model lacks visual qualia, its latent space organizes concepts such that "hot" and "cold," or "loud" and "quiet," occupy structured relational coordinates derived from textual co-occurrence. ## Empirical Illustration: Representing Color Space Consider how an LLM handles abstract and sensory concepts like color. In a modal human mind, the concept of "yellow" activates visual simulations of wavelengths associated with lemons or the sun. In a traditional amodal symbol system, "yellow" is a categorical marker linked to logical propositions. An LLM approaches "yellow" through neither direct perception nor arbitrary logic, but via distributional proximity. In the model’s high-dimensional embedding space, "yellow" is clustered closely with "lemon," "banana," and "sunshine," while maintaining a quantifiable vector distance from "midnight" or "navy." Although the system cannot "see" yellow, the geometry of its parameter space reconstructs a functional analogue of human color categorization based entirely on text-based statistical regularities. ## Epistemic Limits and the Multimodal Turn The status of LLMs as amodal systems remains a subject of intense debate among cognitive scientists and computer scientists. Well-supported evidence shows that text-only LLMs suffer from systematic failures in physical reasoning—such as naive physics or spatial navigation—because text is a compressed, symbolic projection of reality rather than reality itself. However, the rapid proliferation of Multimodal Large Language Models (MLLMs) complicates this classification. By integrating vision encoders, audio processors, and action loops with a core transformer architecture, MLLMs ingest pixel and waveform data directly. In these hybrid systems, the LLM acts as an amodal semantic glue that coordinates multiple modality-specific encoders. Whether this functional integration achieves genuine grounding or merely sophisticated data fusion remains the central unresolved question in contemporary cognitive AI. ## Follow-up questions 1. How do distributional statistics in text-only models simulate properties of modal conceptual spaces without direct sensory input? 2. In what ways do Multimodal Large Language Models (MLLMs) alter the traditional symbol grounding problem compared to unimodal text transformers? 3. What are the primary epistemological limits of treating statistical co-occurrence patterns in language as a valid substitute for embodied conceptual grounding?

## Meaning and Significance of the Claim The selected text addresses a central tension in cognitive science and artificial intelligence: how abstract symbols connect to real-world meaning. Historically, cognitive theories of concepts split into two camps. **Modal (grounded) conceptual systems**, championed by psychologists like Lawrence Barsalou, argue that concepts are tied to sensory-motor simulations of the physical world. Conversely, **amodal systems** treat symbols as arbitrary tokens manipulated via formal rules, running into what Stevan Harnad (1990) famously termed the **symbol grounding problem**—the difficulty of explaining how arbitrary symbols acquire intrinsic meaning without direct connection to physical referents. The claim asserts that Large Language Models (LLMs) complicate this rigid dichotomy. Because LLMs are trained exclusively on text corpora, they possess no physical sense organs. However, because human language is generated by embodied agents navigating a physical environment, text acts as a dense archive of perceptual reality. Human speakers implicitly encode the physical properties of objects, spatial relations, and sensory experiences into word choices and syntactic structures. By mapping these statistical co-occurrences into high-dimensional vector spaces, LLMs construct geometric relations that mirror the structural topology of modal cognition. They achieve a functional simulation of grounded concepts without direct bodily experience. ## Concrete Illustration: The Cartographer of an Unseen Sea To understand this mechanism, consider a conceptual illustration. Imagine a cartographer locked in a completely windowless room who receives millions of captain’s logs detailing maritime journeys. The cartographer has never seen the ocean, felt a wave, or touched a compass. Yet, by meticulously tracking which descriptive terms appear alongside one another—such as "gale," "hull," "pitch," and "swell"—the cartographer constructs a precise multi-dimensional map of marine conditions. On this map, "storm" occupies coordinates close to "rough" and "roll," while "calm" sits near "glassy" and "drift." The cartographer possesses no direct perceptual experience of the sea, yet the resulting spatial geometry reflects the relational reality of maritime physics. This mirrors how an LLM utilizes distributional statistics to capture semantic structure. ## Competing Interpretations and Strong Objections The primary challenge to this perspective comes from semantic externalism and the formalist critique articulated by linguists and computer scientists such as Emily Bender and Alexander Koller (2020). In their influential "Octopus Test" thought experiment, Bender and Koller describe a hyperintelligent sea octopus that taps into an undersea communication cable between two isolated humans. The octopus learns to predict human linguistic responses purely through statistical regularities in form, successfully mimicking conversation without ever knowing what objects like "sticks" or "coconuts" actually are or how they are physically manipulated. Bender and Koller argue that a system operating solely on linguistic form lacks **reference** and **communicative intent**. From this viewpoint, claiming that vector spaces "blur the boundary" mistakes a shadow for the object. Critics emphasize that distributional regularities can mimic physical relationships while failing basic tests of functional affordance—such as determining whether an object can be physically manipulated in a novel context. Under this interpretation, high-dimensional vectors capture text-to-text co-occurrence statistics rather than conceptual grounding. ## Unresolved Debates and Scope Conditions The debate hinges on whether functional equivalence implies genuine semantic competence. Proponents of the selected text's view argue that for language tasks, the statistical footprints preserved in text are rich enough that the distinction between "direct grounding" and "statistical proxy grounding" collapses in practice. Unresolved questions remain regarding the limits of this proxy: * **The Affordance Boundary:** Do LLMs capture deep physical affordances (how objects interact dynamically in the world), or do they merely track how humans *talk* about those interactions? * **The Nature of Representation:** If a high-dimensional vector space successfully predicts and generates behavior indistinguishable from grounded cognition, does the absence of biological sensors render the representation inherently hollow, or does it instantiate a novel form of distributed computation?