The visual system’s lingering mystery: Connecting neural activity and perception

Decades of vision neuroscience research have revealed how complex, abstract features are extracted from two-dimensional retinal images through parallel hierarchical pathways of the visual system. The prevailing assumption has been that the encoding of this visual information by the visual system is optimally integrated to form actions and decisions—a process that has been modeled by an optimal (and often linear) decoder.

Despite extensive research on the encoding side, the decoder’s computational constraints and physiological mechanisms remain largely unexplored. This asymmetry reflects the modular structure of neuroscience research itself. Vision researchers focus on the nature of neural encoding in the visual system, determining the tuning and representational properties of visual areas while assuming simple downstream readout. Meanwhile, decision-making and motor control researchers study action planning, treating visual input as an abstract signal.

This modularity of research has been invaluable for understanding specialized brain functions, but it has limited our grasp of inter-modular communication: How do downstream areas actually read out visual information to generate behavior? Unlike the general assumption of an optimal decoder, recent studies suggest this readout may be suboptimal (e.g., simple summation across neural populations) and surprisingly flexible, allowing different integration strategies even within single tasks. These findings paint a more complex picture of communication among areas, with significant implications for understanding neural computation.

To address this knowledge gap, we need an integrative, multi-module approach to visually guided behavior, going beyond single-domain studies in both theoretical and experimental work. The NeuroAI framework offers a promising path forward by building artificial neural networks (ANNs) designed to resemble brain function while performing complex behavior. The next generation of ANN models must incorporate multiple specialized modules for visual processing, planning and movement as they learn to perform complex sensorimotor tasks. Comparing these models against brain-wide neural data across scales and species will allow us to test both individual module dynamics—which has been done to some degree for each brain subsystem in isolation—but crucially, their inter-modular communication patterns.

This approach creates a symbiotic research loop between modeling and experiment: developing algorithmic constraints that reproduce the brain’s multi-area dynamics in ANN models, while the models continuously inform targeted experiments—including causal manipulations—that can further constrain and validate these models. As we move beyond single-domain models, we can more precisely characterize “readout” by understanding inter-area communication in the brain.

The idea that visual information must be “read out” from the visual cortex so it can be “used,” for example by the frontal cortex to plan and act, is intuitive and rooted in a modular view of the brain. This view—shaped early on by the phrenologists and furthered by great thinkers like Noam Chomsky and Jerry Fodor, as well as empirical evidence all the way from David Hubel and Torsten Wiesel to Nancy Kanwisher—holds that functionally specific modules produce outputs that must be coordinated, raising challenges such as the binding problem and even consciousness itself.

However, this modular view may require revision. For one, information flow in the brain does not solely proceed from the outside in—sensory to motor—but also from the inside out. Indeed, we do not perceive reality “as is,” but our goals and action repertoire co-determines what we perceive, as proposed in the active inference framework. The frontal cortex may shape activity in the visual cortex as much as it responds to it. As the famous saying goes, “To the man with a hammer, everything looks like a nail.” Indeed, evidence shows that visual processing is highly task dependent, with recent evidence of somatotopic maps in the visual cortex, as well as retinotopic organization of regions in the frontal cortex—suggesting bidirectional dependencies.

Relatedly, the metaphor of “readout” may be misleading. Rather than imagining information traveling up a cortical assembly line to be delivered to a homuncular central processor, we might instead view frontal activity as constrained both by sensory input and action possibilities, whereas visual activity is constrained by incoming stimuli and current goals. This reciprocal constraint suggests a brain organized not as modules but as a dynamic system—a Hopfield network of sorts, in which the whole system stabilizes into patterns through mutual dependencies. Whether one still views such a conceptualization as “readout” may depend on one’s scientific discipline and background.

This interactive view aligns with the growing recognition over the past 50 years that the brain features not just feedforward pathways but contains extensive feedback connections reaching back to early visual areas, even the lateral geniculate nucleus. Proposed functions for these feedback signals include figure-ground segregation, predictive coding, perceptual organization, attention, perception of visual detail and even conscious perception. More broadly, feedforward and feedback pathways may exist to reciprocally constrain each other, shaping perception and action together.

Thus, rather than as a perception-action cycle, the brain may be better understood as a system of action-infused perception and perception-infused action, reciprocally constrained by the action possibilities that the world affords us.

Asking what the function of a visual representation is without a behavioral readout is like asking whether a tree falling in a forest makes a sound if no one is there to hear it. Take selectivity for stimulus orientation in the primary visual cortex. Is it for edge detection? Or take category-selective responses in the human ventral visual cortex. Are they for stimulus categorization? The “for” implies a behavioral readout of the visual representation. The behavioral readout is to the visual representation as the listener to the air vibrations caused by the falling tree; visual function emerges from representation only when considering a corresponding behavior.

Behavioral readouts operationalize what is meant by edge detection or stimulus categorization by grounding these terms in specific tasks. This allows one to construct linking models; explicit hypotheses about how neural response properties are read out to produce behavior.

Such linking models are computational hypotheses that can be empirically tested, for example by altering visual responses through inactivation, stimulation or stimulus manipulations, and observing whether the resulting behavioral changes match model predictions.

Linking models reframe the question of what visual representations are for to a better one: What do they afford? Readout implies the potential for a multiplicity of functions; different readouts can achieve different behaviors. Thus, edge detection or image categorization are just one of a set of potential behavioral functions that visual representations might enable.

From this vantage point, orientation or categorical selectivity are not for edge detection or stimulus categorization. Rather, they reformat visual inputs according to an efficient basis set from which simple linear readouts can achieve otherwise complex, entangled functions.

Behavior is thus the key to unraveling how visual representations are read out. But what behavior? Psychophysical tasks offer the ability to craft specific questions. However, as noted by Grace Lindsay’s piece, extensive training can lead to apparently inconsistent conclusions. Perhaps overtraining on low-dimensional psychophysical tasks results in overly specific task solutions, similar to how adversarial images revealed that deep neural networks learn to categorize with subtle high-spatial-frequency patterns. The solution is to examine naturally occurring behaviors. Months of training may be required for a monkey to press a button to report which image is a banana, but the animal needs no training to grab a banana from a tray full of objects. Leveraging such naturalistic cognition tasks is thus the key to understanding how visual representations are read out.

Lindsay points toward an important issue: the disconnect between research on the sensory processing of complex, natural visual information and research on higher-level task solving, which is often performed in simpler experimental setups. How are the two systems efficiently connected in the brain to enable real-world task solving? A quick tour through computer-vision history may offer clues.

In computer vision, various tasks were originally solved by training separate expert systems, i.e., using a different encoder and readout. Although this works in silico, biology lacks the space and energy to maintain a dedicated subsystem for the many tasks we face (although some special visual categories, such as faces, may indeed operate on their own subsystem because of their importance or their specific feature-processing requirements).

Later, computer vision shifted to large-scale foundation models trained on web-scale data, expecting that these would produce feature spaces rich enough to serve many downstream tasks via separate, task-specific readouts. Perhaps analogously, the list of features encoded in the primate higher-level visual cortex keeps expanding, including features that the inferior temporal was previously thought to be invariant to, such as position, size, rotation, etc., and modeling our 2021 work. So does the visual system effectively function as a foundation model plus a range of readouts?

This view is challenged by the need for a fixed readout to operate on varying stimulus features. Consider, for example, a rule switch after which different visual features now require the same motor response. Instead of rewiring output weights, what seems needed is a way to dynamically change what reaches those weights. A solution to this issue may lie in information routing via attention, also used in the latest transformer-based computer vision systems. As NeuroAI modeling work has recently demonstrated, injecting a lightweight, automatically inferred, task-dependent gain field, in the form of a context-vector, into a frozen visual network redirects information flow across all layers so effectively that a single, unchanged readout can work for a new task set.

Importantly, these three sketches—dedicated experts, rich-feature backbones with adaptable heads and context-gated routing onto a fixed head—outline a continuum rather than mutually exclusive camps. The key, then, is not to identify a monolithic “readout” but to chart how these different ways of mapping between visual input and behavior may co-evolve, which strategy is used in which condition, and what computational advantages, such as robustness or task compositionality, the different approaches afford. Computational models now let us implement and systematically perturb each option, offering an image-computable framework for disentangling how the visual system turns natural scenes into goal-directed action.

The program sketched here is easy to admire: Record multiple areas during naturalistic tasks, perturb tagged cell types, train models on the data, and, finally, explain how visual signals drive behavior via downstream areas. However, despite the piece’s opening, vision research has used active, task-engaged experiments for more than 50 years, from eye tracking in humans to single-unit recordings in awake monkeys. But perhaps most strikingly, versions of every proposed step and some synthesis proposed here already exist in neighboring fields.

Daniel Dombeck and colleagues’ head-fixed, closed-loop naturalistic VR navigation with hippocampal imaging was adopted by Matthias Minderer and his team to relate V1 optic flow to parietal choice sequences. Georg Keller (2012) and Alexander Attinger (2017) showed real-time prediction errors in V1 depending on coupled visuomotor experience. In 2019, Nicholas Steinmetz and his colleagues demonstrated Neuropixels recordings across cortex, striatum, and hippocampus at millisecond resolution. Peter Zatka-Haas and his colleagues showed in 2021 that sensory signals in the visual and frontal-motor cortex play a causal role in task performance. Carsen Stringer (2019) and Simon Musall (2019) traced cortex-wide widefield recordings during spontaneous and naturalistic movement. Mixed-selectivity neurons in the prefrontal cortex are shown to combine sensory and task variables in high dimensions, whereas brief beta and gamma bursts gate which subspace determines behavior. Computationally, deep convolutional neural networks predict single‐unit responses in the macaque IT and mouse V1.  LFADS learn low-dimensional trajectories that capture single-trial neural population dynamics with sequential auto-encoders, and recurrent neural networks capture context-dependent dynamics in the monkey PFC.

Moreover, despite the feed-forward metaphor of a readout, “downstream” areas do not simply read sensory input; they apply task-specific transformations to the same visual signal, reflecting the animal’s internal state and intended action. MT lesions show that MT biases perceptual choices, and this was validated by causal manipulations. Simultaneous recordings from the primary visual cortex and hippocampal CA1 in mice running through a virtual corridor shows that population activity in both regions rises and falls together, tracking the animal’s perceived, rather than purely physical, position along the track.

This demonstrates that visual cortical signals are dynamically aligned with hippocampal spatial representations and update in step with the animal’s internal estimate of location. Prefrontal research demonstrates flexible re-weighting of color and motion cues when task rules switch. Micro-stimulation of the frontal eye field modulates V4 at the attended location, showing top-down gain control. Together these findings demonstrate that visual information is continuously reshaped by recurrent fronto-sensory interactions that translate perception into cognition and behavior.

There is plenty to build on and much room for synthesis. True, “closed-loop naturalistic tasks” and “broad, tagged, millisecond multi-area recordings” have not simultaneously appeared in one study with end-to-end modeling. But given the literature, the proposal seems more modest and incremental, far less uncharted territory. Vision science does not need to (re)invent these paradigms; it can join them.

The notion of a “readout” has proven to be a useful theoretical construct. But if we are going to test our theories, I think that the gold standard has to involve causal interventions. That is, if we really think that visual information is read out in service of some behavior, then manipulating visual information should change that behavior. Thus far, the results of these kinds of causal manipulations have been somewhat humbling.

For example, area MT is famous for its robust encoding of motion direction, but inactivating it sometimes has no effect on performance of a motion task. Lindsay mentioned a similar disjunction between encoding and readouts in the context of depth perception, and we recently showed the same thing in V4 for form discrimination. The (lack of) readout in these cases confounds intuition; why doesn’t the brain use information that is readily available in each area?

Importantly, all these studies yielded different results depending on how the animals were trained, even for a fixed set of stimuli and behavioral tasks. This suggests that readouts cannot be predicted from stimulus representations alone, no matter how good our decoders are. Rather, what is read out from any particular patch of visual cortex neurons depends heavily on both stimulus representations and recent visual experience (or context more generally). The interaction of these factors is arguably the most important thing that we don’t understand about readouts.

With further experimentation, we might find that the malleability of readouts is an artifact of the degeneracy of learning in laboratory environments. Indeed, in the lab, we often ask our subjects to categorize two types of inputs into two types of outputs, and maybe this is just not something that biological brains are wired to do. But I doubt it. Even in natural environments, we learn to recognize new faces and places all the time, and this likely requires a change in readouts as well. So a useful direction for future research will be to apply causal manipulations to neural activity while exploring a larger variety of tasks and techniques.

Lindsay asks, “How is visual information used by the rest of the brain?” I agree that decodability is only a partial answer, and that probing multiple brain regions, modeling trial-wise variability or employing more naturalistic tasks may help us find an answer. Two complementary points:

First, answering the question may benefit from studying individual differences in brain and behavior. One way to show that decodable information is being used by downstream systems is by predicting behavior based on a psychological model applied to neural responses of a brain region. For example, in so-called “model-based” cognitive neuroscience, formal models of behavior from mathematical psychology are directly linked to neural responses. This research reveals a puzzle. On the one hand, the models typically characterize the process linking stimulus to behavior as occurring in a limited number of stages (e.g., perception to decision to behavior). On the other hand, the same (decision) stage of these models has been linked to responses of many different (visual and non-visual) brain regions. How do we reconcile these two facts? Perhaps many regions represent the same visual information in different ways, but whether the information is used depends on context. To explore this, one might use experimental designs that manipulate task difficulty while also modeling individual differences in neural response and behavior. Although different brain regions encode the same information, readout may occur from certain regions based on the varied decision process individuals use to perform a task.

Second, answering the question may require precisely articulating what we mean by information use or “readout.” For example, it has sometimes been claimed that decodability is a proxy for showing that information is “explicitly” represented in a brain region. The reasoning is that if the representation is structured so that stimulus information can be decoded from patterns of neural response, then it is at least available for use by downstream systems. However, as David Kirsh discusses in his 1990 paper “When is information explicitly represented?”, it is not obvious what such availability amounts to. Kirsh illustrates the inconsistencies between our intuitions about explicit representation using (informal) examples from mathematics and computer science. He suggests explicit representation is less about how representations are structured and more about process and how quickly the information they contain can be accessed. I encourage those interested in theorizing on how the brain uses visual information to consider revisiting Kirsh’s paper (a personal favorite).

My work explores how minds create useful and efficient representations of sensory input, and most recently it has been centered on the flexibility of those representations under task variation. Within just a few seconds of being asked an unexpected question, people change the way that they store information so that they are prepared to answer that question in the future. We have been focused on the computational mechanisms that allow multiple readouts to be available for a given stimulus and then simulate how specific readouts are routed into working memory depending on the current task.

Interest in the visual representations that form memories, especially for visual information, has been pronounced recently because of the proliferation of neural decoding of noninvasive neuroimaging data. However, a mismatch between our assumptions of brain decoding and the internal readout mechanisms between brain areas can be misleading.

Because neuroimaging relies on the accumulation of data from pools of neurons, decoding such data benefits from being applied to large redundant representations within the brain. This dependency can make it easier to successfully decode information in certain brain regions such as V1 where representations are distributed over more cortical area. However, if cognitive processing projects information into progressively more efficient and low-dimensional representations, then decoding tools will be biased toward the very early levels of processing in the sensory system and less sensitive to the deeper and more meaningful representations that are formed afterward. Evidence of this bias toward early representations is seen when decoding from electrical brain waves shows a rapid loss of accuracy in the first second after a stimulus has been seen. However, this decline in decoding does not match the memory contents, as people can still report the information several seconds later.

The inability to find sustained representations is sometimes interpreted as an activity-silent form of coding, in which neural firing would no longer be required to hold the memory. However, this silencing of the decoder may reflect the conversion of memories into more compact representations that cannot readily be decoded with these tools. Thus, the inability to decode internal neural readouts from the brain does not mean they do not exist.

Another confounding issue is that decoding does not tell us what kind of information is being represented. For example, images of cars contain many horizontal elements, whereas vegetable images have more variability in their edges. Being able to discriminate whether someone was looking at cars versus vegetables in one brain region could reflect the predominance of orientation channels, whereas successful discrimination in another area could reflect a representation that is more akin to the conceptual difference between these categories.

We should bear in mind these limitations of neural decoders if we are going to use them as foundational knowledge for theories of cognitive neuroscience. One remedy is to use models that expand our understanding of the many kinds of representations that could explain both behavior and neural correlates and then develop experiments to test for those representations.

To generate intelligent behavior, our brains must extract from our past experience the bits of information that are most relevant for accomplishing our future goals. This process can be characterized by rate-distortion theory, the branch of information theory that deals with optimal data compression under limited capacity resources. The visual system plays a key role in this process, taking in massive amounts of information from our visual environment and forming compressed representations at different levels of abstractions. What makes a compressed representation useful for an intelligent system? That depends on the downstream tasks it needs to support and the representational resources available to it. Rate-distortion theory formulates optimal representations in terms of a trade-off between the informational complexity, or compressibility, of the representation and the expected utility that the representation supports. This general organizing principle is not specific to the visual system and not even to biological systems. It applies to any system that attempts to support intelligent behavior, whether it is natural or artificial. A key prediction for readout mechanisms in the brain is that the encoding and decoding mechanisms should be co-adapted to each other in a way that optimally balances complexity and utility.