Although we rarely stop to appreciate its complexity, our everyday experience of the world is dependent upon the seamless integration of information in different modalities. The process by which this “illusion of unity” arises from a constant stream of sights and sounds is both complex and informative. Due to its multi-sensory nature, music provides an especially rich and vibrant domain in which to explore audio-visual integration, particularly through an illusion in which percussionists use body movements to shape the way performances “sound.”
A Musical Illusion
- The main debate: Is it possible to produce long and short notes through variations in striking motion?
Gesture Analysis
- Analyzing the types of gestures that musicians may make to achieve their sounds.
Virtual Marimbist
- Deconstructing the gestures of musical movement to discover what factors lead to the audio-visual illusion.
Cross-Modal Causality
- Does it matter if the audio of an impact lags behind or leads ahead of the video? What if the sound no longer sounds like an impact?
Biological Motion
- The differences in how we perceive and understand the motion of an inanimate object versus a biological being.
- What makes biological motion unique from non-biological? How far can reduce the qualities of human movement until it no longer appears human?
Although much of our work on this topic is inspired by musical situations, this research holds broad implications for theories of sensory integration that generalize beyond music listening. For example, further exploration of this illusion builds a strong case for the role of causality as a cue for audio-visual integration. Furthermore, conflicts between the patterns of audio-visual integration observed in our paradigm and previous research on this topic have now led to new insights on the role of amplitude envelope in auditory perception general, as well as its role in audio-visual integration in particular.
We are currently working with a network of collaborators interested in applying our techniques and approaches to explore audio visual integration in individuals with autism spectrum disorder, as well as integration in hearing impaired populations. Both of these studies will help extend our basic research by informing treatments for conditions that affect many individuals around the world. For more information on this work, please see an archived PDF of Nedra Floyd-Pautler’s feature story on our work on the Hearing Lab.