Chapter 56. Sensory Interaction: McGurk Effect

Learning Objectives

McGurk effect

perceptual illusion created by mixing video of a person speaking sounds with audio containing different sounds; if the mouth movements are distinctive, vision dominates; if the mouth movements are not distinctive, the perceived sounds are a blend of vision and hearing

sensory interaction

information from one of the senses often influences how we perceive information from a different sense

visual capture

when two senses send conflicting information to the brain, the information from vision tends to dominate perception

Depth Cues

true

Learning Objectives:

Understand the principles of sensory interaction and visual capture.

Describe how these principles produce the perceptual illusion called the McGurk effect.

Review

concept_review

Review

Select the NEXT button to continue with the Review.

36-year-old hispanic magician with a rabbit

Photodisc/Getty Images

1. The McGurk effect, named after researcher Henry McGurk who first reported the effect in 1976, is an example of visual capture, the tendency for vision to dominate the other senses. The mouth movements of a ventriloquist’s dummy can trick our brain into believing that the dummy is speaking the words. The McGurk effect is also an example of sensory interaction (the principle that one sense may influence another).

Review

concept_review

Review

Select the NEXT button to continue with the Review.

asset/activities/mcgurk_effect/images/review_2.svg

2. The McGurk effect can be demonstrated by combining a silent video of a person repeating one syllable with the audio of a person repeating a different syllable.

Review

concept_review

Review

Select the NEXT button to continue with the Review.

asset/activities/mcgurk_effect/images/review_3.svg

3. What would happen if we combine a silent video of a person repeating the syllable "BA" with an audio clip of a person repeating the syllable "DA"? When listening with eyes closed, most people will accurately hear "DA." But with eyes open, the visual information conflicts with the auditory information. Because the mouth movements associated with the syllable "BA" are distinctive, the video usually wins (visual capture in action). Most people who watch the combined clip hear “BA” rather than "DA."

Review

concept_review

Review

Select the NEXT button to continue with the Review.

asset/activities/mcgurk_effect/images/review_4.svg

4. On the other hand, most people who see the less-distinctive mouth movements for "GA" while listening to the sound of "BA" will perceive "DA" (or perhaps "THA"), a blend of the two syllables that demonstrates sensory interaction.

Practice 1: Video Clip A

video_quiz

true

Practice 1: Video Clip A

Play the video clip and then report what you heard. Next, play the clip again and listen with your eyes closed. Did you hear something different with your eyes closed? After you have reported what you heard each time, select the SHOW RESULTS button.

Watch Clip A and try to guess what syllable the person is saying.

"fa" "va" "ba"

Next, listen to this clip again with your eyes closed. Try to guess what syllable the person is saying.

"fa" "va" "ba"

Tom Ludwig

Your results:
With eyes open, you heard FAVABA.
With eyes closed, you heard FAVABA.

Explanation:
This clip combines video of the person repeating one syllable with audio of the person repeating a different syllable. In this conflict situation, visual capture predicts that the distinctive visual mouth movements for FA will be dominant. Most viewers think the person is saying FA (or perhaps VA), while those listening with eyes closed correctly hear the syllable BA.

Practice 2: Video Clip B

video_quiz

true

Practice 2: Video Clip B

Play the video clip and report what you heard. Then, play the clip again and listen with your eyes closed. Did you hear something different with your eyes closed? After you have reported what you heard each time, select the SHOW RESULTS button.

Watch Clip B and try to guess what syllable the person is saying.

"da" "tha" "ba"

Next, listen to this clip again with your eyes closed. Try to guess what syllable the person is saying.

"ga" "da" "ba"

Tom Ludwig

Your results:
With eyes open, you heard DATHABA.
With eyes closed, you heard GADABA.

Actual stimulus:
Video showed the mouth motions for the syllable GA while the audio played the sounds for the syllable BA.

Explanation:
Because the mouth movements for the syllable GA are not very distinctive, sensory interaction leads the viewer to blend the video and audio to perceive a completely new syllable. Most viewers think the person is saying DA (or perhaps THA), while those listening with eyes closed correctly hear the syllable BA.

Quiz

dnd_test

Quiz

Drag each term to the appropriate description. When both terms have been placed, select the CHECK ANSWER button.

Perhaps you should go back to review visual capture and sensory interaction.

Select the NEXT button and move to the Conclusion.

sensory interaction

visual capture

a general principle that explains why most people who view the mouth movements for GA while listening to the sound BA will perceive the syllable DA

a more specific principle that explains why most people who view the mouth movements for FA while listening to the sound BA will perceive the syllable FA

Conclusion

end_slide

Congratulations!

You have completed the activity Title