Abstract
In media reproduction, there are many situations in which audio and visual signals, coming from the same object, are presented with a spatial offset. When the offset is small enough the spatial conflict is usually resolved by the brain, merging the different information into one unified object; this is the so-called ventriloquism effect. With respect to evolving immersive technologies such as virtual and augmented reality, it is important to define the maximally accepted offset angle to create a convincing environment. However, in literature on the ventriloquism effect, values for the maximally acceptable offset angle vary greatly. Therefore, a series of experiments was devised to investigate the influencing factors leading to this great variation. First, the influence of participants’ background and sensory training in hearing and vision was assessed. In a second step, the influence of the stimulus properties such as their semantic category was examined. In both cases, a forced-choice yes/no experiment was conducted evaluating participants’ thresholds in perceived spatial coherence. The third set of experiments strived to evaluate ventriloquism indirectly using reaction times measurement to circumvent the observed influencing factors. The results show that auditory sensory training greatly influences the measured offsetangles with a nearly doubled acceptable offset angle for untrained participants (19°) compared to musically trained ones (10°). The measured offset is further dependent on signal properties linked to localisation precision with variations in the range of ±2°. Both findings can be explained along the current model of bimodal spatial integration. Compared to these results, the reaction time measurements reveal that offsets as small as 5° and less can influence human bimodal integration independent of the sensory training. The divergent results are discussed along the lines of the two-stream processing in the brain for semantic and spatial information to derive recommendations for media reproduction taking into account the different use-cases of various devices and reproduction methods.