We are witnessing great excitement about what are loosely called “foundation models of neural activity.” These neural foundation models are, in essence, like the artificial-intelligence (AI) chatbots that have become an inescapable part of our lives: They have read through large amounts of neural data and learned their underlying statistics through training. After training, these neural foundation models can generalize what they have learned to predict new neural activity patterns, motor output or responses to sensory stimuli.
Arguably, the most intriguing aspect of neural foundation models is that they can learn from activity across different animals, related brain areas or similar tasks. This suggests something remarkable about the brain. Just as AI chatbots exploit the finite number of characters and grammatical rules of language to learn the statistics underlying individual authors’ work in order to write sonnets, suggest recipes or discuss the meaning of life, neural foundation models’ capacity for generalization suggests that neural activity must also be composed of “characters” and “grammatical rules” that are shared across brains.
In this piece, I will try to explain why these neural foundation models work, by linking their performance to recent neuroscientific findings, and I will highlight what I think are key opportunities for neuroscientists to use neural foundation models to understand the brain—and the challenges ahead.
O
Importantly for us, these collective patterns seem to be relatively consistent across the neurons within a population you record from, meaning that you can potentially recover any neural process from any subset of recorded neurons. Moreover, these collective activity patterns are also preserved across animals performing the same motor behaviors or navigating the same environment. They are also often similar across related tasks, although neuroscientists do not have a way to know a priori when this will or will not be the case. Given that brain function is well described by collective patterns of activity that can be uncovered from completely different subsets of neurons within a population, and that these collective activity patterns are invariant across individuals and similar across related tasks, it follows that the powerful statistical machinery underpinning neural foundation models can learn these invariances and commonalities by aggregating data across individuals and tasks. In other words, recent neuroscience findings explain why neural foundation models should and do work.
Beyond combining recent observations, neural foundation models are powerful tools that will enable us—and have already enabled us—to learn new things about the brain, at least in two main directions.
First, neural foundation models can help us improve our understanding of the language of neural populations by continuing to quantify—and maybe even “qualify”—the organization of the activity patterns that mediate behavior. For instance, we could use them to characterise how different neurodevelopmental and neurological conditions alter neural population activity. Neural foundation models could also help us define how different ways of solving an interesting task, such as playing the same game of Tetris, affect the collective activity of neural populations. Finally, these models also offer an opportunity to define relationships across different species, by quantifying their similarities in neural population activity. This line of inquiry will be crucial to establish homologies across the evolutionary tree that are based not only on neuroanatomy and behavioural repertories but also on neural function. Given that many neurotechnologies target this “systems level” to alleviate motor and psychiatric conditions, establishing similarities between the activity patterns of different model organisms and humans could accelerate translational research and have a tremendous societal impact.
Second, neural foundation models could potentially help us to establish links between the different levels of brain function, from neurotransmitters to single neurons, circuits and brain regions. The collective patterns of neural activity that I have been discussing are without question implemented by neurons, meaning that they can, in principle, be reduced to their constituent neurons—although whether they can be “constructed” from their constituent neurons is a key question I do not have room to discuss here. Yet there is an important looming question: Is there a tractable mapping between aspects of these collective activity patterns and well-defined subpopulations of neurons? Recent work shows that it is possible to predict fairly well, based on the neural activity alone, from which area of the mouse visual system a neuron has been recorded, and even to distinguish among as many as 11 different excitatory cell types. These intriguing results were based on examining the individual mappings of single neurons onto the collective patterns learned by the neural foundation model, suggesting that there is indeed a discoverable mapping between single neuron subclasses and the collective.
T
Another looming question is whether it will be possible to integrate brain-wide recordings across many behaviors. Most results to date focus on a certain brain region across fairly similar tasks or a series of related regions during the same task or behavior, although of course this concern is relevant for all subfields in neuroscience. One fundamental challenge to achieve this overarching goal, with models that are as data-hungry and parameter-hungry as current neural foundation models, is whether we will end up in the situation anticipated by Jorge Luis Borges: In his short story “On Exactitude in Science,” a society of cartographers created a perfect map of their empire that coincided point by point with the empire itself, yet no one cared about it because they could not learn anything useful from a map—for us, a model—that had as much detail as what it stood for.
Finally, the kind of neural foundation models that I have been discussing do not have a body. It could be argued that, in principle, sensing and acting on the world through a body can be abstracted away as “transformations” that may be learnable given enough neural and behavioral data. Yet how much data will be enough data? My intuition is that this problem may become intractable in practice. For example, it is not “only” about learning complex transformations—such as from the retina to the lateral geniculate nucleus and then V1, or from motor cortex output to spinal circuits, motoneurons and eventually movement—but the fact that these transformations depend on many factors such as top-down modulation, attention, engagement, the state of the body, and perhaps even the developmental trajectory and lifelong experience of the organism. This challenge has motivated work on foundation models of brain-body-world interaction loops, which could potentially tease apart all these factors from what a neural population actually “does.”
Of course, neural foundation models can be useful even if we cannot understand the relationships among neurons, their collective activity and the behavior they uncover. For example, these relationships could be leveraged to develop new neurotechnologies that “know” what neurons to target to drive the collective brain activity away from disease states, leading to new cures. And perhaps by using tricks from machine learning we might even avoid falling into the same trap as Borges’ cartographers, making neural foundation models a new instrument to effectively chart brain function. Ultimately, only by navigating this unknown territory will we discover whether neural foundation models can transform—no pun intended—neuroscience and neurotechnology.
