PhD thesis

1. 

   Getting ahead: Prediction as a window into language, and language as a window into the predictive brain

   M. Heilbron

   Radboud University Nijmegen 2022

   link PDF

Journal articles and conference proceedings

2024

1. 

   Language Models That Accurately Represent Syntactic Structure Exhibit Higher Representational Similarity To Brain Activity

   Abraham Jacob Fresen, Rochelle Choenni, Micha Heilbron, Willem Zuidema, and Marianne Heer Kloots

   Proceedings of the Annual Meeting of the Cognitive Science Society 2024

   Abs

   We investigate whether more accurate representation of syntactic information in Transformer-based language models is associated with better alignment to brain activity. We use fMRI recordings from a large dataset (MOUS) of a Dutch sentence reading task, and perform Representational Similarity Analysis to measure alignment with 2 mono- and 3 multilingual language models. We focus on activity in a region known for syntactic processing (the Left posterior Medial Temporal Gyrus). We correlate model-brain similarity scores with the accuracy of dependency structures extracted from model internal states using a labelled structural probe. We report three key findings: 1) Accuracy of syntactic dependency representations correlates with brain similarity, 2) The link between brain similarity and dependency accuracy persists regardless of sentence complexity, although 3) Sentence complexity decreases dependency accuracy while increasing brain similarity. These results highlight how interpretable, linguistic features such as syntactic dependencies can mediate the similarity between language models and brains.

2023

1. 

   Lexical processing strongly affects reading times but not skipping during natural reading

   Micha Heilbron, Jorie Haren, Peter Hagoort, and Floris P Lange

   Open Mind 2023

2022

1. 

   Cortical activity during naturalistic music listening reflects short-range predictions based on long-term experience

   Pius Kern, Micha Heilbron, Floris P Lange, and Eelke Spaak

   eLife Dec 2022

   Abs link PDF

   Expectations shape our experience of music. However, the internal model upon which listeners form melodic expectations is still debated. Do expectations stem from Gestalt-like principles or statistical learning? If the latter, does long-term experience play an important role, or are short-term regularities sufficient? And finally, what length of context informs contextual expectations? To answer these questions, we presented human listeners with diverse naturalistic compositions from Western classical music, while recording neural activity using MEG. We quantified note-level melodic surprise and uncertainty using various computational models of music, including a state-of-the-art transformer neural network. A time-resolved regression analysis revealed that neural activity over fronto-temporal sensors tracked melodic surprise particularly around 200 ms and 300–500 ms after note onset. This neural surprise response was dissociated from sensory-acoustic and adaptation effects. Neural surprise was best predicted by computational models that incorporated long-term statistical learning – rather than by simple, Gestalt-like principles. Yet, intriguingly, the surprise reflected primarily short-range musical contexts of less than ten notes. We present a full replication of our novel MEG results in an openly available EEG dataset. Together, these results elucidate the internal model that shapes melodic predictions during naturalistic music listening.

2. 

   A hierarchy of linguistic predictions during natural language comprehension

   Micha Heilbron, Kristijan Armeni, Jan-Mathijs Schoffelen, Peter Hagoort, and Floris P. Lange

   Proceedings of the National Academy of Sciences Aug 2022

   Abs link PDF

   Understanding spoken language requires transforming ambiguous acoustic streams into a hierarchy of representations, from phonemes to meaning. It has been suggested that the brain uses prediction to guide the interpretation of incoming input. However, the role of prediction in language processing remains disputed, with disagreement about both the ubiquity and representational nature of predictions. Here, we address both issues by analyzing brain recordings of participants listening to audiobooks, and using a deep neural network (GPT-2) to precisely quantify contextual predictions. First, we establish that brain responses to words are modulated by ubiquitous predictions. Next, we disentangle model-based predictions into distinct dimensions, revealing dissociable neural signatures of predictions about syntactic category (parts of speech), phonemes, and semantics. Finally, we show that high-level (word) predictions inform low-level (phoneme) predictions, supporting hierarchical predictive processing. Together, these results underscore the ubiquity of prediction in language processing, showing that the brain spontaneously predicts upcoming language at multiple levels of abstraction.

3. 

   Dampened sensory representations for expected input across the ventral visual stream

   David Richter, Micha Heilbron, and F. P. De Lange

   Oxford Open Neuroscience Aug 2022

   Abs link PDF

   Expectations, derived from previous experience, can help in making perception faster, more reliable and informative. A key neural signature of perceptual expectations is expectation suppression, an attenuated neural response to expected compared with unexpected stimuli. While expectation suppression has been reported using a variety of paradigms and recording methods, it remains unclear what neural modulation underlies this response attenuation. Sharpening models propose that neural populations tuned away from an expected stimulus are particularly suppressed by expectations, thereby resulting in an attenuated, but sharper population response. In contrast, dampening models suggest that neural populations tuned toward the expected stimulus are most suppressed, thus resulting in a dampened, less redundant population response. Empirical support is divided, with some studies favoring sharpening, while others support dampening. A key limitation of previous neuroimaging studies is the ability to draw inferences about neural-level modulations based on population (e.g. voxel) level signals. Indeed, recent simulations of repetition suppression showed that opposite neural modulations can lead to comparable population-level modulations. Forward models provide one solution to this inference limitation. Here, we used forward models to implement sharpening and dampening models, mapping neural modulations to voxel-level data. We show that a feature-specific gain modulation, suppressing neurons tuned toward the expected stimulus, best explains the empirical fMRI data. Thus, our results support the dampening account of expectation suppression, suggesting that expectations reduce redundancy in sensory cortex, and thereby promote updating of internal models on the basis of surprising information.

4. 

   Reconstructing the predictive architecture of the mind and brain

   Floris P. Lange, Lea-Maria Schmitt, and Micha Heilbron

   Trends in Cognitive Sciences Sep 2022

   Abs link PDF

   Predictive processing has become an influential framework in cognitive neuroscience. However, it often lacks specificity and direct empirical support. How can we probe the nature and limits of the predictive brain? We highlight the potential of recent advances in artificial intelligence (AI) for providing a richer and more computationally explicit test of this theory of cortical function.

2020

1. 

   Word contexts enhance the neural representation of individual letters in early visual cortex

   Micha Heilbron, David Richter, Matthias Ekman, Peter Hagoort, and Floris P. Lange

   Nature Communications Jan 2020

   Abs link PDF

   Visual context facilitates perception, but how this is neurally implemented remains unclear. One example of contextual facilitation is found in reading, where letters are more easily identified when embedded in a word. Bottom-up models explain this word advantage as a post-perceptual decision bias, while top-down models propose that word contexts enhance perception itself. Here, we arbitrate between these accounts by presenting words and nonwords and probing the representational fidelity of individual letters using functional magnetic resonance imaging. In line with top-down models, we find that word contexts enhance letter representations in early visual cortex. Moreover, we observe increased coupling between letter information in visual cortex and brain activity in key areas of the reading network, suggesting these areas may be the source of the enhancement. Our results provide evidence for top-down representational enhancement in word recognition, demonstrating that word contexts can modulate perceptual processing already at the earliest visual regions.

2019

1. 

   Tracking Naturalistic Linguistic Predictions with Deep Neural Language Models

   Micha Heilbron, Benedikt Ehinger, Peter Hagoort, and Floris P. Lange

   2019 Conference on Cognitive Computational Neuroscience Jan 2019

   Abs PDF Poster

   Prediction in language has traditionally been studied using simple designs in which neural responses to expected and unexpected words are compared in a categorical fashion. However, these designs have been contested as being ‘prediction encouraging’, potentially exaggerating the importance of prediction in language understanding. A few recent studies have begun to address these worries by using model-based approaches to probe the effects of linguistic predictability in naturalistic stimuli (e.g. continuous narrative). However, these studies so far only looked at very local forms of prediction, using models that take no more than the prior two words into account when computing a word’s predictability. Here, we extend this approach using a state-of-the-art neural language model that can take roughly 500 times longer linguistic contexts into account. Predictability estimates from the neural network offer a much better fit to EEG data from subjects listening to naturalistic narrative than simpler models, and reveal strong surprise responses akin to the P200 and N400. These results show that predictability effects in language are not a side-effect of simple designs, and demonstrate the practical use of recent advances in AI for the cognitive neuroscience of language.

2. 

   Confidence resets reveal hierarchical adaptive learning in humans

   Micha Heilbron, and Florent Meyniel

   PLoS computational biology Jan 2019

   Abs link PDF

   Hierarchical processing is pervasive in the brain, but its computational significance for learning under uncertainty is disputed. On the one hand, hierarchical models provide an optimal framework and are becoming increasingly popular to study cognition. On the other hand, non-hierarchical (flat) models remain influential and can learn efficiently, even in uncertain and changing environments. Here, we show that previously proposed hallmarks of hierarchical learning, which relied on reports of learned quantities or choices in simple experiments, are insufficient to categorically distinguish hierarchical from flat models. Instead, we present a novel test which leverages a more complex task, whose hierarchical structure allows generalization between different statistics tracked in parallel. We use reports of confidence to quantitatively and qualitatively arbitrate between the two accounts of learning. Our results support the hierarchical learning framework, and demonstrate how confidence can be a useful metric in learning theory.

2018

1. Great expectations: Is there evidence for predictive coding in auditory cortex?

   Micha Heilbron, and Maria Chait

   Neuroscience Jan 2018

   Abs link PDF

   Predictive coding is possibly one of the most influential, comprehensive, and controversial theories of neural function. While proponents praise its explanatory potential, critics object that key tenets of the theory are untested or even untestable. The present article critically examines existing evidence for predictive coding in the auditory modality. Specifically, we identify five key assumptions of the theory and evaluate each in the light of animal, human and modeling studies of auditory pattern processing. For the first two assumptions – that neural responses are shaped by expectations and that these expectations are hierarchically organized – animal and human studies provide compelling evidence. The anticipatory, predictive nature of these expectations also enjoys empirical support, especially from studies on unexpected stimulus omission. However, for the existence of separate error and prediction neurons, a key assumption of the theory, evidence is lacking. More work exists on the proposed oscillatory signatures of predictive coding, and on the relation between attention and precision. However, results on these latter two assumptions are mixed or contradictory. Looking to the future, more collaboration between human and animal studies, aided by model-based analyses will be needed to test specific assumptions and implementations of predictive coding – and, as such, help determine whether this popular grand theory can fulfill its expectations.

2. How Do Expectations Shape Perception?

   Floris P. Lange, Micha Heilbron, and Peter Kok

   Trends in Cognitive Sciences Sep 2018

   Abs link PDF

   Perception and perceptual decision-making are strongly facilitated by prior knowledge about the probabilistic structure of the world. While the computational benefits of using prior expectation in perception are clear, there are myriad ways in which this computation can be realized. We review here recent advances in our understanding of the neural sources and targets of expectations in perception. Furthermore, we discuss Bayesian theories of perception that prescribe how an agent should integrate prior knowledge and sensory information, and investigate how current and future empirical data can inform and constrain computational frameworks that implement such probabilistic integration in perception.

2017

1. Dynamic Causal Modelling of GRIN2A-related epileptic abnormalities in sleep

   Micha Heilbron, Richard Rosch, and Karl Friston

   Society for Neuroscience Annual Conference Sep 2017

   PDF Poster

2016

1. Visual input that matches the content of visual working memory requires less (not faster) evidence sampling to reach conscious access

   Surya Gayet, Leendert Maanen, Micha Heilbron, Chris LE Paffen, and Stefan Stigchel

   Journal of Vision Sep 2016

   link PDF