Neuroscience-inspired approaches for visual‑audio multimodal visual search

Abstract

This project aimed to investigate the effect of auditory stimuli on visual search in humans through human eye-tracking experiments and to explore various visual-audio search models to examine how a multi-sensory approach would influence visual search efficiency and accuracy. The human experiment involved exposing the subject to 240 different trials of images and audio stimuli under seven different conditions and 20 different target categories. The eye fixations are then collected and processed to determine the effects of auditory stimuli on visual search. The results found that, while audio semantics has a slight effect in aiding visual search, the difference was not significant. However, human participants heavily relied on visual cues instead of audio cues. The model experiment aimed to explore various sound localisation models and integrate them with the human-inspired visual search model, IVSN. The results showed that the integration of the IVSN significantly boosted the weak performance of the sound localisation model in our dataset, but concluded that future work needs to reduce the hybrid model’s reliance on the IVSN representation.