Qi 2023 Neural Dynamics of Causal Inference in the macaque frontoparietal circuilt

Key Takeaways

• This paper investigates how the brain represents and updates the hidden causal structure between visual and proprioceptive signals during multisensory processing. 
• Monkeys can combine previous experience and current multisensory signals to estimate the hidden common source of visual and proprioceptive signals. 
• The premotor cortex integrates previous experience and sensory inputs to infer hidden variables, while the parietal cortex updates the sensory representation to maintain consistency with the causal inference structure. 
• The dynamic loop of frontal-parietal interactions provides a potential neural mechanism for understanding how circuits represent hidden structures related to body awareness and agency. - 
• Premotor neurons integrate bimodal information for small disparities and segregate the information for large disparities between proprioceptive and visual information. 
• Parietal cells show reaching tuning changes that support the updating sensory uncertainty between tasks.

Intro

discusses how the brain infers the hidden causal structure of the environment during natural perception. explains that the brain combines information from multiple sensory inputs to infer the properties of a single entity based on the quality and uncertainty of the sensory stimuli. The example of the ventriloquism illusion is used to illustrate this concept.

Lit Review: 

cites several previous studies that have investigated the neural mechanisms underlying multisensory integration and causal inference. These studies have shown that the brain combines information from multiple sensory inputs to infer the properties of a single entity based on the quality and uncertainty of the sensory stimuli. The paper also discusses the role of the premotor and parietal cortices in multisensory processing and how they contribute to the representation and updating of the hidden causal structure.

Methods

• Uses a virtual reality system to train monkeys to infer the probability of a potential common source from visual and proprioceptive signals based on their spatial disparity. 
• Involves single-unit recordings in the premotor and parietal cortices to investigate the neural mechanisms and functional circuits essential for representing and updating the hidden causal structure and corresponding sensory representations during multisensory processing.

Results

• monkeys were able to combine previous experience and current multisensory signals to estimate the hidden common source and subsequently update the causal structure and sensory representation. 
• Single-unit recordings revealed that neural activity in the premotor cortex represents the core computation of causal inference, characterizing the estimation and update of the likelihood of integrating multiple sensory inputs at a trial-by-trial level. 
• In response to signals from the premotor cortex, neural activity in the parietal cortex also represents the causal structure and further dynamically updates the sensory representation to maintain consistency with the causal inference structure. 
• This dynamic loop of frontal-parietal interactions in the causal inference framework may provide the neural mechanism to answer long-standing questions regarding how neural circuits represent hidden structures for body awareness and agency.

Conclusions

• premotor cortex integrates previous experience and sensory inputs to infer hidden variables and selectively updates sensory representations in the parietal cortex to support behavior. 
• provides insights into the neural mechanisms and functional circuits essential for representing and updating the hidden causal structure and corresponding sensory representations during multisensory processing. 
• findings suggest that the dynamic loop of frontal-parietal interactions in the causal inference framework may provide the neural mechanism to answer long-standing questions regarding how neural circuits represent hidden structures for body awareness and agency.

Limitations

• generalizability of findings from monkeys to humans. 
• study focused on one specific type of MSI task, but different tasks may involve different neural mechanisms. 
• translation of VR to real-world. '

How do monkeys combine previous experience and current multisensory signals to estimate hidden common sources, as demonstrated by proprioceptive drift reported in this study?

In this study, monkeys were trained to perform a reaching task in a VR environment where visual and proprioceptive signals were dissociated. The monkeys were able to combine previous experience and current multisensory signals to estimate the hidden common source of the two signals. This was demonstrated by the proprioceptive drift reported in the study, which showed that the monkeys updated their estimate of the common source based on the spatial disparity between the visual and proprioceptive signals. The monkeys used this information to update their causal structure and sensory representation, allowing them to perform the reaching task more accurately.

In what ways does the dynamic loop of frontal-parietal interactions observed during causal inference provide a potential neural mechanism for understanding how circuits represent hidden structures related to body awareness and agency?

The dynamic loop of frontal-parietal interactions observed during causal inference provides a potential neural mechanism for understanding how circuits represent hidden structures related to body awareness and agency in several ways. 

1. premotor cortex integrates previous experience and sensory inputs to infer hidden variables, such as the common source of visual and proprioceptive signals.

2. the parietal cortex updates the sensory representation to maintain consistency with the causal inference structure. 

3. frontal-parietal interactions allow for the selective updating of sensory representations based on the estimated causal structure, which is critical for accurate behavior. 

Overall, the dynamic loop of frontal-parietal interactions provides a potential neural mechanism for understanding how circuits represent hidden structures related to body awareness and agency.

What are the key novel results of this study regarding premotor neurons' representation of bimodal information for small disparities versus segregation for large disparities between proprioceptive and visual information?
they integrate the information for small disparities between proprioceptive and visual information, while segregating the information for large disparities. This suggests that premotor neurons play a critical role in the estimation and updating of the likelihood of integrating multiple sensory inputs at a trial-by-trial level.

Questions

- How do the findings of this study contribute to our understanding of the neural mechanisms underlying multisensory processing?
- How might the findings of this study be relevant to the development of prosthetic devices or other technologies that rely on multisensory integration? - What are the potential implications of this study for our understanding of body awareness and agency? - Are there any limitations to the methods used in this study, and if so, how might they be addressed in future research?

Steop 2021 - Addictive Nature of Human Multisensory Evoked Pupil Response

Van der Stoep N, Van der Smagt MJ, Notaro C, Spock Z, Naber M. The additive nature of the human multisensory evoked pupil response. Sci Rep. 2021 Jan 12;11(1):707. doi: 10.1038/s41598-020-80286-1.

Abstract: This research paper investigates the nature of multisensory pupil response by combining methodological approaches from previous studies and using suprathreshold stimuli, to determine whether it is linear (additive) or non-linear (sub-additive/super-additive).

Key Takeaways

• The study investigated the nature of multisensory pupil response by combining methodological approaches from previous studies and using suprathreshold stimuli. It provides insights into how spatial orienting may be modulated by saliency, focused spatial attention or motor coordination through changes in pupil size.
• Reaction time (RT) data indicated MSI as shown by race model inequality violation, however, the multisensory pupil response in both experiments could best be explained by linear summation of unisensory responses. 
• This suggests that for supra-threshold stimuli, the multisensory pupil response is additive in nature and cannot be used to measure MSI since only a departure from additivity can unequivocally demonstrate an interaction between senses.

Data Used: two experiments were conducted using auditory and visual stimuli to evoke an (onset) response in observers. The data used for these experiments included reaction time (RT), unisensory  pupil responses, and multisensory pupil responses

Main Approaches in Paper

• Combining methodological approaches from previous studies and using suprathreshold stimuli to investigate the nature of multisensory pupil response.Comparing RT data with unisensory and multisensory responses, as deviations from additivity can be used as a strict criterion for MSI.
• Investigating whether spatial orienting is modulated by saliency, focused spatial attention or motor coordination through changes in pupil size.

Results in Brief

• Results indicate that the multisensory pupil response for supra-threshold stimuli is additive in nature and cannot be used as a measure of MSI, since only a departure from additivity can unequivocally demonstrate an interaction between senses. 
• Additionally, it was found that spatial orienting may be modulated by saliency, focused spatial attention or motor coordination through changes in pupil size.

IN MORE DETAIL

INTRO  

Multisensory integration (MSI) is the process by which the brain combines information from different senses to create a unified perception.

MSI importance in spatial orienting eg:  using senses to quickly determine the location of an approaching car when crossing the street and adjust our actions accordingly.

MSI mediated by SC, which contains multisensory neurons that respond to input from different sensory modalities and contribute to the multisensory enhancement of orienting behavior. 

• Integrates sensory input and generates eye-movements to unisensory and multisensory events. 
• Involved in  transient changes in pupil size. It has been suggested that the pupil’s response to sensory events plays an important role in orienting responses as it is modulated by saliency, focused spatial attention, and motor coordination.
• The underlying neural computation of these multisensory neural responses has been characterized as linear (additive: equal to the sum of the unisensory responses), or non-linear (i.e. sub-additive: less than the sum, or super-additive: larger than the sum).

Implications of the multisensory pupil response for clinical applications.

• If the multisensory pupil response is larger or smaller than the linear sum of the unisensory pupil responses, then it can be concluded that the multisensory response is driven by integrated sensory input and that patients can integrate sensory input.
• However, if the multisensory pupil response is additive, then it is likely that the observed multisensory behaviour is the result of the independent processing of sensory input. This is important for clinical applications, as it can be used to determine whether a patient is able to integrate sensory input.

Conflicting results of research on the nature of the multisensory pupil response.

The multisensory pupil response is the change in pupil size in response to a multisensory stimulus. 

Previous research in monkeys showed the multisensory pupil response to AV (audiovisual) events to be additive or sub-additive. But recent study in humans suggests super-additive responses which suggests that multisensory pupil response is driven by MSI. 

Conflicting results suggest that more research needed. 

Experiment 1: (Figure 1 left panel)

Figure 1 (left panel) shows the experimental setup for Experiment 1. Twelve Participants took part in a response and no-response block, where they were instructed to respond as fast as possible to an onset of sound or light in the response block, and passively observe stimuli in the no-response block. Pupil responses were recorded during both blocks. 

Experiment 2: Figure 1, right panel 
Figure 1 (right panel) illustrates the experimental setup for Experiment 2 which was similar to that of experiment one but with two additional conditions: bright visual stimulus and dark visual stimulus. In this second experiment participants had to respond when there was a change detected either visually or auditorily while withholding their response if neither changed occurred

Results of Experiment 1

Figure 2 shows the results of Experiment 1. The left panel illustrates response times (RTs) for auditory only, bright visual stimulus and dark visual stimulus conditions in both active and passive blocks. It can be seen that responses were faster to multisensory stimuli than unisensory stimuli - i.e., RTs decreased when a sound was presented together with either a bright or dark target compared to just one modality alone (A > AV Bright , A > AV Dark ). The right panel displays pupil size data from Experiment 1 which indicates an additive nature of the multisensory pupil response as there is no significant difference between audiovisual targets versus sum of uni-modal targets across all participants tested in this experiment (AV vs cSum). This suggests that any observed speedup due to MSI could not explain beyond statistical facilitation effects such as crossmodal spatial attention or switch costs etc..

Response Time: The results of the experiments were measured using response times (RTs). The results indicated that participants responded faster to multisensory stimuli than unisensory stimuli in both the bright and dark stimuli.

The results were measured using a Bayesian repeated measures Analysis of Variance (ANOVA) and post-hoc tests corrected for multiple testing. This ANOVA indicated very strong evidence for an effect of Target Modality compared to a null model assuming no effect. Essentially results indicated that responses in the multisensory conditions were faster than in the unisensory conditions, for both the dark and light conditions.

Multisensory Response Enhancement (MRE)

Multisensory response enhancement (MRE) is a measure of the speed-up in the multisensory condition relative to the fastest unisensory condition.

It is calculated by analysing the grey area in Figure 2B . Bayesian one-sample t-tests were used to determine if the MRE was larger than zero in the bright and dark AV target condition. The results showed that the MRE was larger than zero in both conditions, with a mean of 30 ms and 39 ms in the bright and dark conditions respectively.

There was only anecdotal evidence for a difference in the amount of MRE between the dark and bright condition.

To investigate whether the observed MRE could be explained by statistical facilitation, the cumulative response time distribution in the multisensory condition (the blue line in Figure 2C) was compared to the sum of the unisensory cumulative RT distributions (the race model, the black line in Figure 2C). If responses in the multisensory condition are faster than the upper limit of the race model, the race model inequality is violated.

This means that multisensory response enhancement cannot be explained by independent processing of sensory input, which is indicative of MSI

Bayesian one-sided one-sample t-tests indicated there was strong evidence for the amount of RMI violation being larger than zero in both the Dark and Bright condition (see Figure 2E).

RMI violation was observed both for dark and bright AV targets There was no evidence for or against a difference in violation area between the Dark and Bright condition.

Race model inequality (RMI) violation is a measure used to determine whether multisensory response enhancement can be explained by independent processing of sensory input. 

It compares the cumulative response time distribution in the multisensory condition with that of the sum of unisensory responses, known as race models. 

If responses in the multisensory condition are faster than those predicted by race models, then RMI has been violated and this indicates that there is evidence for MSI - i.e., an interaction between senses which cannot simply be explained away through statistical facilitation or other processes such as crossmodal spatial attention or switch costs.

"Overall, these results are indicative of MSI in the Bright and Dark target condition as the observed multisensory response enhancement cannot simply be explained by independent processing of sensory input"

Pupillometry Data 

Discusses the pupillometric measures used to investigate the nature of multisensory pupil response. Pupil size is a measure that can be used as an indicator for various sensory processes, and has been shown to be modulated by saliency, focused spatial attention or motor coordination. 

In this study, eye-link 1000 was used to collect eye position and pupil size data from participants' right eyes with a sampling rate of 1000 Hz in order to compare responses between unisensory (sound/light alone) stimuli and audiovisual targets presented simultaneously. The amount of speed-up in the multisensory condition relative to fastest uni-sensual condition was analysed using Multivariate Response Enhancement (MRE). Additionally, Bayesian one sample t tests were conducted on RMI violation area which compared cumulative RT distributions under different conditions - i.e., whether MSI could explain any observed differences beyond statistical facilitation effects such as crossmodal spatial attention or switch costs etc..

Discussion Expt 1

The results of the experiment showed that the response time data indicated multisensory integration (MSI) as shown by race model inequality violation. However, the multisensory pupil response in both experiments could best be explained by linear summation of the unisensory pupil responses. This means that the multisensory pupil response was equal to the sum of the unisensory pupil responses, indicating that the multisensory pupil response is additive in nature. This suggests that the multisensory pupil response may not be a good measure of MSI in populations for which response time data collection is not feasible. The researchers then conducted a second experiment using a change-detection paradigm similar to a previous study that did show super-additivity of the multisensory pupil response. The second experiment was conducted to further investigate the nature of the multisensory pupil response.

Experiment 2

Figure 3 shows the results of Experiment 2. The left panel illustrates response times (RTs) for auditory only, bright visual stimulus and dark visual stimulus conditions in both active and passive blocks. It can be seen that responses were faster to multisensory stimuli than unisensory stimuli - i.e., RTs decreased when a sound was presented together with either a bright or dark target compared to just one modality alone (A > AV Bright , A > AV Dark ). The right panel displays pupil size data from Experiment 2 which indicates an additive nature of the multisensory pupil response as there is no significant difference between audiovisual targets versus sum of uni-modal targets across all participants tested in this experiment (AV vs cSum). This suggests that any observed speedup due to MSI could not explain beyond statistical facilitation effects such as crossmodal spatial attention or switch costs etc..

Figure 4 shows the results of a Bayesian repeated measures Analysis of Variance (ANOVA) for RTs in the dark target condition. It indicates very strong evidence for an effect of Target Modality compared to a null model assuming no effect (A, V Dark , AV Dark ; BF 10 = 625,233). Post-hoc tests corrected for multiple testing indicated that responses in the AV Dark condition were faster than those observed with either unisensory stimulus alone - i.e., A or VDark conditions (M = 249 ms vs 285 and 319 respectively; SD's 40/47/57). This demonstrates additivity of the audiovisual pupil response as there is no significant difference between multisensory targets versus sum of unimodal targets across all participants tested in this experiment.

Figure 5 shows the results of a Bayesian one sample t test on RMI violation area which compared cumulative RT distributions under different conditions. It indicates that there was no significant difference between audiovisual targets versus sum of uni-modal targets across all participants tested in this experiment (AV vs cSum). This suggests that any observed speedup due to MSI could not explain beyond statistical facilitation effects such as crossmodal spatial attention or switch costs etc..

Terms Explained

• MSI: Multisensory Integration
• SC: subcortical structure that contains multisensory neurons that respond to input from different sensory modalities and contribute to the multisensory enhancement of orienting behavior
• Additivity: measure of how the multisensory response compares to the sum of the unisensory responses
• Additive Response: If the multisensory response is equal to the sum of the unisensory responses
• SubAdditive Response: If the multisensory response is less than the sum of the unisensory responses. 
• SuperAdditive Response:  If the multisensory response is larger than the sum of the unisensory responses