Are LLMs perception-based modal or non-perception-based amodal conceptual systems?

Large Language Models (LLMs) are fundamentally non-perception-based, **amodal conceptual systems** in their core architecture and training mechanics, though they frequently learn statistical approximations of modal structure. In cognitive science, modal (or grounded) systems posit that concepts are intrinsically tied to sensory-motor simulations of the physical world. Conversely, amodal systems treat concepts as abstract, arbitrary symbols processed through formal relations, detached from any specific sensory modality. Because standard LLMs are trained exclusively on next-token prediction over disembodied text corpora, they lack sensory organs and direct physical interaction. However, because human language itself encodes rich statistical footprints of perceptual experience, LLMs construct high-dimensional vector spaces that capture relational properties resembling modal cognition, blurring the historical boundary between pure amodal symbol manipulation and grounded representation. ## Theoretical Framework: Modal versus Amodal Systems To evaluate where LLMs fit, one must distinguish the two primary paradigms of conceptual representation in cognitive psychology: * **Modal (Grounded) Conceptual Systems:** Championed by theorists like Lawrence Barsalou, this framework argues that semantic knowledge is reenacted across the brain's sensory-motor modalities. Understanding the concept of a "kick," for example, involves neural simulations in motor and premotor areas. * **Amodal Conceptual Systems:** Tracing back to classical computationalism and Jerry Fodor’s Language of Thought, this approach posits that concepts are represented in an abstract, language-like format distinct from sensory inputs. Symbols derive their function from formal syntactic rules rather than experiential grounding. When applied to artificial intelligence, these frameworks intersect with Stevan Harnad’s classical "symbol grounding problem"—the dilemma of how arbitrary computational symbols acquire meaning without external referents. ## The Architecture and Status of Large Language Models Unimodal LLMs operate entirely within a linguistic manifold derived from the distributional hypothesis—the linguistic principle that words occurring in similar contexts tend to have similar meanings. Structurally, this aligns them with amodal systems. They do not possess a body, navigate physical space, or experience interoception; their tokens are indices within a vast mathematical matrix rather than reactivations of sensory states. Yet, treating LLMs as *purely* amodal overlooks how thoroughly human language mirrors perception. Language does not float entirely free of reality; it systematically records perceptual regularities. Consequently, an LLM’s hidden layers develop geometric topologies where semantic distances mirror physical and perceptual realities. While the model lacks visual qualia, its latent space organizes concepts such that "hot" and "cold," or "loud" and "quiet," occupy structured relational coordinates derived from textual co-occurrence. ## Empirical Illustration: Representing Color Space Consider how an LLM handles abstract and sensory concepts like color. In a modal human mind, the concept of "yellow" activates visual simulations of wavelengths associated with lemons or the sun. In a traditional amodal symbol system, "yellow" is a categorical marker linked to logical propositions. An LLM approaches "yellow" through neither direct perception nor arbitrary logic, but via distributional proximity. In the model’s high-dimensional embedding space, "yellow" is clustered closely with "lemon," "banana," and "sunshine," while maintaining a quantifiable vector distance from "midnight" or "navy." Although the system cannot "see" yellow, the geometry of its parameter space reconstructs a functional analogue of human color categorization based entirely on text-based statistical regularities. ## Epistemic Limits and the Multimodal Turn The status of LLMs as amodal systems remains a subject of intense debate among cognitive scientists and computer scientists. Well-supported evidence shows that text-only LLMs suffer from systematic failures in physical reasoning—such as naive physics or spatial navigation—because text is a compressed, symbolic projection of reality rather than reality itself. However, the rapid proliferation of Multimodal Large Language Models (MLLMs) complicates this classification. By integrating vision encoders, audio processors, and action loops with a core transformer architecture, MLLMs ingest pixel and waveform data directly. In these hybrid systems, the LLM acts as an amodal semantic glue that coordinates multiple modality-specific encoders. Whether this functional integration achieves genuine grounding or merely sophisticated data fusion remains the central unresolved question in contemporary cognitive AI. ## Follow-up questions 1. How do distributional statistics in text-only models simulate properties of modal conceptual spaces without direct sensory input? 2. In what ways do Multimodal Large Language Models (MLLMs) alter the traditional symbol grounding problem compared to unimodal text transformers? 3. What are the primary epistemological limits of treating statistical co-occurrence patterns in language as a valid substitute for embodied conceptual grounding?

## Formulating the Problem: Harnad's Core Challenge Coined by cognitive scientist Stevan Harnad in his seminal 1990 paper, the **symbol grounding problem** addresses a foundational dilemma in cognitive science and artificial intelligence: how the semantic interpretation of a formal symbol system can be made intrinsic to the system itself, rather than merely parasitic on the human observer's understanding. Classically, computationalism (such as Jerry Fodor’s Language of Thought hypothesis) treated cognition as the syntactic manipulation of abstract, amodal symbols based strictly on their physical or formal shapes. Harnad asked a piercing question: > How can the meanings of the meaningless symbol tokens, manipulated solely on the basis of their (arbitrary) shapes, be grounded in anything but other meaningless symbols? (Harnad, *The Symbol Grounding Problem*) If a computer manipulates the token `CAT` by shifting it according to syntactic rules, the system itself does not know what a cat is. The semantic weight—the actual reference to a furry, four-legged feline—exists entirely in the mind of the human programmer or user. To break this infinite regress of definitions pointing to other definitions, Harnad argued that symbols must be anchored bottom-up in non-symbolic representations: *iconic representations* (proximal sensory projections of distal objects) and *categorical representations* (learned or innate feature-detectors that categorize sensory inputs). ## An Illustrative Analogy: The Chinese Dictionary Dilemma To grasp the mechanics of the problem, consider Harnad’s classic analogy of trying to learn Chinese using a purely Chinese-to-Chinese dictionary. If a monolingual English speaker who knows zero Chinese is given a massive Chinese dictionary, they can trace cross-references indefinitely. Symbol $A$ is defined by symbol string $B$ and $C$; $B$ is defined by $D$ and $E$. Because all entries are composed of the same unfamiliar shapes, the system remains entirely closed. Without an initial entry point connected to perceptual experience—such as pointing to a physical object or interacting with the world—the entire lexicon remains completely devoid of intrinsic meaning. For Harnad, a computer running a symbolic program is in this exact position: it navigates a vast, self-referential dictionary without ever tasting, touching, or seeing the world it describes. ## Competing Interpretations and Contemporary Objections While Harnad’s framework dominated classical AI and robotics, it faces significant pushback from proponents of connectionism and modern statistical learning. * **The Distributional Defense:** Proponents of distributional semantics argue that rich statistical regularities embedded in massive linguistic corpora can bootstrap semantic understanding. Under the distributional hypothesis, words occurring in similar contexts share meanings. * **The Functionalist Objection:** Scholars of natural language processing, such as Emily Bender and Alexander Koller, argue that while text-only models lack sensorimotor grounding, they nevertheless build complex internal models of linguistic form and communicative intent. They contend that meaning can emerge functionally from patterns of use within a linguistic community, even if the system lacks a physical body. | Approach | Primary Mechanism of Meaning | Main Limitation | | :--- | :--- | :--- | | **Sensorimotor Grounding (Harnad)** | Bottom-up anchoring in iconic and categorical perception | Struggles to explain abstract concepts (e.g., "justice") that lack direct perceptual correlates. | | **Distributional/Statistical (Modern LLMs)** | High-dimensional geometric mapping of word co-occurrences | Operates entirely within form (syntax/statistics) without direct causal links to worldly referents. | ## Connections and Scope Conditions As noted in the exploration of unimodal language models, treating LLMs as purely amodal systems brings Harnad’s critique squarely into contemporary debates. Unimodal LLMs learn from text alone, operating within a linguistic manifold rather than a sensorimotor one. Whether this constitutes a fatal barrier to genuine machine understanding depends on one's definition of semantics. If reference requires causal interaction with the physical world, LLMs remain ungrounded symbol manipulators. If meaning is instead constituted by inferential role and structural distribution within language, then the symbol grounding problem may misdiagnose how language functions altogether.