Van der Stoep N, Van der Smagt MJ, Notaro C, Spock Z, Naber M. The additive nature of the human multisensory evoked pupil response. Sci Rep. 2021 Jan 12;11(1):707. doi: 10.1038/s41598-020-80286-1.
Abstract: This research paper investigates the nature of multisensory pupil response by combining methodological approaches from previous studies and using suprathreshold stimuli, to determine whether it is linear (additive) or non-linear (sub-additive/super-additive).
Key Takeaways
- The study investigated the nature of multisensory pupil response by combining methodological approaches from previous studies and using suprathreshold stimuli. It provides insights into how spatial orienting may be modulated by saliency, focused spatial attention or motor coordination through changes in pupil size.
- Reaction time (RT) data indicated MSI as shown by race model inequality violation, however, the multisensory pupil response in both experiments could best be explained by linear summation of unisensory responses.
- This suggests that for supra-threshold stimuli, the multisensory pupil response is additive in nature and cannot be used to measure MSI since only a departure from additivity can unequivocally demonstrate an interaction between senses.
Data Used: two experiments were conducted using auditory and visual stimuli to evoke an (onset) response in observers. The data used for these experiments included reaction time (RT), unisensory pupil responses, and multisensory pupil responses
Main Approaches in Paper
- Combining methodological approaches from previous studies and using suprathreshold stimuli to investigate the nature of multisensory pupil response.Comparing RT data with unisensory and multisensory responses, as deviations from additivity can be used as a strict criterion for MSI.
- Investigating whether spatial orienting is modulated by saliency, focused spatial attention or motor coordination through changes in pupil size.
Results in Brief
- Results indicate that the multisensory pupil response for supra-threshold stimuli is additive in nature and cannot be used as a measure of MSI, since only a departure from additivity can unequivocally demonstrate an interaction between senses.
- Additionally, it was found that spatial orienting may be modulated by saliency, focused spatial attention or motor coordination through changes in pupil size.
IN MORE DETAIL
INTRO
Multisensory integration (MSI) is the process by which the brain combines information from different senses to create a unified perception.
MSI importance in spatial orienting eg: using senses to quickly determine the location of an approaching car when crossing the street and adjust our actions accordingly.
MSI mediated by SC, which contains multisensory neurons that respond to input from different sensory modalities and contribute to the multisensory enhancement of orienting behavior.
- Integrates sensory input and generates eye-movements to unisensory and multisensory events.
- Involved in transient changes in pupil size. It has been suggested that the pupil’s response to sensory events plays an important role in orienting responses as it is modulated by saliency, focused spatial attention, and motor coordination.
- The underlying neural computation of these multisensory neural responses has been characterized as linear (additive: equal to the sum of the unisensory responses), or non-linear (i.e. sub-additive: less than the sum, or super-additive: larger than the sum).
Implications of the multisensory pupil response for clinical applications.
- If the multisensory pupil response is larger or smaller than the linear sum of the unisensory pupil responses, then it can be concluded that the multisensory response is driven by integrated sensory input and that patients can integrate sensory input.
- However, if the multisensory pupil response is additive, then it is likely that the observed multisensory behaviour is the result of the independent processing of sensory input. This is important for clinical applications, as it can be used to determine whether a patient is able to integrate sensory input.
Conflicting results of research on the nature of the multisensory pupil response.
The multisensory pupil response is the change in pupil size in response to a multisensory stimulus.
Previous research in monkeys showed the multisensory pupil response to AV (audiovisual) events to be additive or sub-additive. But recent study in humans suggests super-additive responses which suggests that multisensory pupil response is driven by MSI.
Conflicting results suggest that more research needed.
Experiment 1: (Figure 1 left panel)
Figure 1 (left panel) shows the experimental setup for Experiment 1. Twelve Participants took part in a response and no-response block, where they were instructed to respond as fast as possible to an onset of sound or light in the response block, and passively observe stimuli in the no-response block. Pupil responses were recorded during both blocks.
Experiment 2: Figure 1, right panel Figure 1 (right panel) illustrates the experimental setup for Experiment 2 which was similar to that of experiment one but with two additional conditions: bright visual stimulus and dark visual stimulus. In this second experiment participants had to respond when there was a change detected either visually or auditorily while withholding their response if neither changed occurred
Results of Experiment 1Figure 2 shows the results of Experiment 1. The left panel illustrates response times (RTs) for auditory only, bright visual stimulus and dark visual stimulus conditions in both active and passive blocks. It can be seen that responses were faster to multisensory stimuli than unisensory stimuli - i.e., RTs decreased when a sound was presented together with either a bright or dark target compared to just one modality alone (A > AV Bright , A > AV Dark ). The right panel displays pupil size data from Experiment 1 which indicates an additive nature of the multisensory pupil response as there is no significant difference between audiovisual targets versus sum of uni-modal targets across all participants tested in this experiment (AV vs cSum). This suggests that any observed speedup due to MSI could not explain beyond statistical facilitation effects such as crossmodal spatial attention or switch costs etc..

Response Time: The results of the experiments were measured using response times (RTs). The results indicated that participants responded faster to multisensory stimuli than unisensory stimuli in both the bright and dark stimuli.
The results were measured using a Bayesian repeated measures Analysis of Variance (ANOVA) and post-hoc tests corrected for multiple testing. This ANOVA indicated very strong evidence for an effect of Target Modality compared to a null model assuming no effect. Essentially results indicated that responses in the multisensory conditions were faster than in the unisensory conditions, for both the dark and light conditions.
Multisensory Response Enhancement (MRE)
Multisensory response enhancement (MRE) is a measure of the speed-up in the multisensory condition relative to the fastest unisensory condition.
It is calculated by analysing the grey area in Figure 2B . Bayesian one-sample t-tests were used to determine if the MRE was larger than zero in the bright and dark AV target condition. The results showed that the MRE was larger than zero in both conditions, with a mean of 30 ms and 39 ms in the bright and dark conditions respectively.
There was only anecdotal evidence for a difference in the amount of MRE between the dark and bright condition.
To investigate whether the observed MRE could be explained by statistical facilitation, the cumulative response time distribution in the multisensory condition (the blue line in Figure 2C) was compared to the sum of the unisensory cumulative RT distributions (the race model, the black line in Figure 2C). If responses in the multisensory condition are faster than the upper limit of the race model, the race model inequality is violated.
This means that multisensory response enhancement cannot be explained by independent processing of sensory input, which is indicative of MSI
Bayesian one-sided one-sample t-tests indicated there was strong evidence for the amount of RMI violation being larger than zero in both the Dark and Bright condition (see Figure 2E).
RMI violation was observed both for dark and bright AV targets There was no evidence for or against a difference in violation area between the Dark and Bright condition.
Race model inequality (RMI) violation is a measure used to determine whether multisensory response enhancement can be explained by independent processing of sensory input.
It compares the cumulative response time distribution in the multisensory condition with that of the sum of unisensory responses, known as race models.
If responses in the multisensory condition are faster than those predicted by race models, then RMI has been violated and this indicates that there is evidence for MSI - i.e., an interaction between senses which cannot simply be explained away through statistical facilitation or other processes such as crossmodal spatial attention or switch costs.
"Overall, these results are indicative of MSI in the Bright and Dark target condition as the observed multisensory response enhancement cannot simply be explained by independent processing of sensory input"
Pupillometry Data Discusses the pupillometric measures used to investigate the nature of multisensory pupil response. Pupil size is a measure that can be used as an indicator for various sensory processes, and has been shown to be modulated by saliency, focused spatial attention or motor coordination.
In this study, eye-link 1000 was used to collect eye position and pupil size data from participants' right eyes with a sampling rate of 1000 Hz in order to compare responses between unisensory (sound/light alone) stimuli and audiovisual targets presented simultaneously. The amount of speed-up in the multisensory condition relative to fastest uni-sensual condition was analysed using Multivariate Response Enhancement (MRE). Additionally, Bayesian one sample t tests were conducted on RMI violation area which compared cumulative RT distributions under different conditions - i.e., whether MSI could explain any observed differences beyond statistical facilitation effects such as crossmodal spatial attention or switch costs etc..
Discussion Expt 1
The results of the experiment showed that the response time data indicated multisensory integration (MSI) as shown by race model inequality violation. However, the multisensory pupil response in both experiments could best be explained by linear summation of the unisensory pupil responses. This means that the multisensory pupil response was equal to the sum of the unisensory pupil responses, indicating that the multisensory pupil response is additive in nature. This suggests that the multisensory pupil response may not be a good measure of MSI in populations for which response time data collection is not feasible. The researchers then conducted a second experiment using a change-detection paradigm similar to a previous study that did show super-additivity of the multisensory pupil response. The second experiment was conducted to further investigate the nature of the multisensory pupil response.
Experiment 2
Figure 3 shows the results of Experiment 2. The left panel illustrates response times (RTs) for auditory only, bright visual stimulus and dark visual stimulus conditions in both active and passive blocks. It can be seen that responses were faster to multisensory stimuli than unisensory stimuli - i.e., RTs decreased when a sound was presented together with either a bright or dark target compared to just one modality alone (A > AV Bright , A > AV Dark ). The right panel displays pupil size data from Experiment 2 which indicates an additive nature of the multisensory pupil response as there is no significant difference between audiovisual targets versus sum of uni-modal targets across all participants tested in this experiment (AV vs cSum). This suggests that any observed speedup due to MSI could not explain beyond statistical facilitation effects such as crossmodal spatial attention or switch costs etc..
Figure 4 shows the results of a Bayesian repeated measures Analysis of Variance (ANOVA) for RTs in the dark target condition. It indicates very strong evidence for an effect of Target Modality compared to a null model assuming no effect (A, V Dark , AV Dark ; BF 10 = 625,233). Post-hoc tests corrected for multiple testing indicated that responses in the AV Dark condition were faster than those observed with either unisensory stimulus alone - i.e., A or VDark conditions (M = 249 ms vs 285 and 319 respectively; SD's 40/47/57). This demonstrates additivity of the audiovisual pupil response as there is no significant difference between multisensory targets versus sum of unimodal targets across all participants tested in this experiment.
Figure 5 shows the results of a Bayesian one sample t test on RMI violation area which compared cumulative RT distributions under different conditions. It indicates that there was no significant difference between audiovisual targets versus sum of uni-modal targets across all participants tested in this experiment (AV vs cSum). This suggests that any observed speedup due to MSI could not explain beyond statistical facilitation effects such as crossmodal spatial attention or switch costs etc..
Terms Explained
- MSI: Multisensory Integration
- SC: subcortical structure that contains multisensory neurons that respond to input from different sensory modalities and contribute to the multisensory enhancement of orienting behavior
- Additivity: measure of how the multisensory response compares to the sum of the unisensory responses
- Additive Response: If the multisensory response is equal to the sum of the unisensory responses
- SubAdditive Response: If the multisensory response is less than the sum of the unisensory responses.
- SuperAdditive Response: If the multisensory response is larger than the sum of the unisensory responses