Speech Perception and the McGurk Effect
It is often assumed that speech perception is primarily an auditory process. However, in recent research it has been proven that it is now known that seeing the face of the speaker is just as important for the hearing impaired, for speech development, and also when dealing with a complicated or degraded auditory signal. Audiovisual speech also provides a rich forum in which to study more general issues in Cognitive Science. Auditory information is the major source of information for speech perception, visual The McGurk Effect is just one example of how audiovisual speech perception is processed. Another example of the way people routinely use and process information provided is by the speaker’s lip movements. Our facial and lip movements help us to understand what is being said and how it is being said in a noisy environment information can also exert a strong influence on what we hear this is known as the McGurk Effect. While you might think of each sense as independent from the others, the McGurk Effect shows just how linked everything is. Sight and sound seem like two entirely adverse senses which can definitely have an effect on each other. The McGurk Effect is often seen as evidence for gestural theories because such theories provide a good account for why the auditory and visual information are integrated during perception.
Most conversations occur face-to-face , that involves both auditory and visual inputs. Although people are able to understand words without visual input, congruent visual and auditory information is important in understanding a conversation (Sumby and Pollack 1954). When visual and auditory information are incongruent, people sometimes fuse or combine the two types of information erroneously, perceiving a new sound. This phenomenon is known as the McGurk Effect (McGurk and MacDonald 1976). This paper will attempt to show that visual information is very important in speech perception and how cultural differences indicate that people with different language backgrounds exhibit different levels of the McGurk effect.
McGurk Effect
Harry McGurk (1936-1998), was a senior developmental psychologist at the University of Surrey in England where his primary field study was psychological research of human development. He conducted infant perception studies back in the early ’70s. His research were to great extent in conflict with the, at the time, authority in this area, Tom Bower. He rebutted the research of Aronson and Rosenbloom, stating intermodal spatial dislocation, and instead went on to investigate and research the unity of the senses hypothesis by way of intermodal conflict. It has been understood for many years that speech perception occurs not only on what is heard but also upon what is seen (Sumby and Pollack 1954). Speech is more intelligible when visual information is congruent with auditory information (MacDonald, Andersen et al. 2000). Seeing the speaker facilitates the understanding of a conversation in a noisy environment (Sumby and Pollack 1954; MacDonald, Andersen et al. 2000). However, prior to The McGurk Effect, it was believed that visual information aids speech perception only when auditory information is distorted. This explanation remained unchanged until McGurk and MacDonald discovered what is now simply referred to as "the McGurk Effect" (McGurk and MacDonald 1976). Findings from both behavioral (the McGurk effect) and neuroimaging studies support the view that Audiovisual speech perception is an integrated procedure. Studies using functional Magnetic Resonance Imaging (fMRI) show that the primary auditory cortex is active during pure lip reading, while the primary visual cortex is active with just auditory inputs (Calvert, Bullmore et al. 1997). These studies as well as those reported below strongly support the notion that visual information is very important in speech perception.
Speech Perception
Speech perception, the process by which we employ cognitive, motor, and sensory processes to hear and understand speech, is a product of innate preparation ("nature") and sensitivity to experience ("nurture") as demonstrated in infants' abilities to perceive speech. We are constantly being bombarded by acoustical energy. The challenge to humanity is to translate this energy into meaningful data. Speech perception is not dependent on the extraction of simple invariant acoustic patterns in the speech waveform. The sound’s acoustic pattern is complex and greatly varies. It is dependent upon the preceding and following sounds (Moore, 1997). Studies of infants from birth have shown that they respond to speech signals in a special way, suggesting a strong innate component to language. Other research has shown the strong effect of environment on language acquisition by proving that the language an infant listens to during the first year of life enables the child to begin producing a distinct set of sounds (babbling). The babbling stage is where the content of their articulations bears no relationship to the language of their elders. They can and do produce all the speech sounds of their language as well as many sounds which do not form part of the phonemic structure of their language and which they will be unable to reproduce later in life. At a later stage they auditory compare their articulations with those of their elders and eventually weed out the sounds which do not belong to their parents' language. Speech is more readily decoded in the left hemisphere than in the right cerebral hemisphere. This is evident in people with brain lesions. The left hemisphere plays a primary role in speech perception (Moore, 1997). According to Fant (1973), speech perception is a process consisting of both successive and concurrent identification on a series of progressively more abstract levels of linguistic structure. Nature of speech sounds phonemes are the smallest unit of sound. Nature of Speech Sounds Phonemes are the smallest unit of sound. In any given language words are formed by combining these phonemes. English has approximately 40 different phonemes that are defined in terms of what is perceived, rather than in terms of acoustic patterns. Phonemes are abstract, subjective entities that are often specified in terms of how they are produced. Alone they have no meaning, but in combination they form words (Moore, 1997). In speech there are vowels and consonants. Consonants are produced by constricting the vocal tract at some point along its length. These sounds are classified into different types according to the degree and nature of the constriction. The types are stops, affricates, fricatives, nasals, and approximants. Vowels are usually voiced and are relatively stable over time Moore, 1997). Consonant sounds have a lower intensity than vowels, fluctuate more rapidly, and last for a shorter time than vowels. Therefore, the acoustic patterns of consonants frequently decay rapidly. Another explanation is that boundaries, which separate one speech sound from another, tend to lie at a point where discrimination is optimal. The last explanation is that it comes from experience with a person’s own language. In this explanation it is believed that a person learns to attend to acoustic differences that affect the meaning of a word and ignore the differences that do not affect the meaning. The natural consequence of this is categorical perception (Moore, 1997). The main point in this area is that the listener can only correctly distinguish speech sounds to the extent that they are identified as different phonemes. Small changes to the acoustical signal may make little difference to the way the sound is perceived, yet other changes which are equally as small may produce a distinct change, altering the phoneme identity. People do not hear changes within one phoneme category. Only changes from one phoneme to another phoneme are detected (Lobacz, 1984). This sound is derived from combining the consonant that they saw and the vowel that they heard. This result is typically experienced as slightly imperfect by comparison with the normal case in which acoustical and optical stimuli are in agreement. The observers cannot tell what the nature of the imperfection is. They are not able to say that it is because they heard one thing and saw something else being said. The conclusion is the McGurk effect. Although categorical perception generally is considered to reflect the operation of a special speech decoder, there is a strong indication that categorical perception can also occur in non-speech signals. Musicians are a very good example of this. The discrimination performance of musicians was better for frequency changes that revised the identity of the chord than for changes that did not alter the identity (Moore, 1997).
Models of Speech Perception
There are many models of speech perception. There is not one specific model that is generally accepted. Three influential models being discussed are the motor theory, the cued based approach, and the TRACE model. Motor Theory In the motor theory the objects of speech perception are the intended phonetic gestures of the speaker. According to Liberman (1996), “they are represented in the brain as motor commands that call for movements of the articulators through certain linguistically significant configurations.” The listener perceives the articulatory gesture the speaker is intending to make when producing the word or utterance. In the motor theory, speech perception and speech production are closely linked and innately specified. This model accounts for many speech perception characteristics. However, the model does not specify how the translation from the signal to the perceived gesture is accomplished, thus making the model incomplete (Liberman, 1996). The motor theory is in two ways motor. First, it is considered motor because it takes the proper object of phonetic perception to be a motor event. Secondly, it assumes that adaptations of the motor system for controlling the organs of the vocal tract took precedence in the evolution of speech (Liberman and Mattingly, 1985).
Information Provided by the Face
Audio and visual information are no longer treated as separate phenomena. Instead, speech perception is now discussed in terms of the binding of auditory and visual percepts. McGurk & MacDonald (1976) found that when a visually presented syllable such as /ga/ and the auditory stimulus /ba/ were dubbed together, 98% of the participants perceived the syllable as /da/. Surprisingly, participants perceived a syllable that differed from both the visual and auditory input when both were given at the same time. The same effect was also found in other syllable pairs; e.g., the visual /ka/ dubbed into an audio /pa/ created a /ta/ perception. McGurk fusion effect occurs between speech sounds differing in places-of-articulation (POA); i.e., places in the mouth where articulators such as the tongue and mandible move relative to the lips or roof of the mouth. For example, the misperception of an alveolar constant /d/ occurs when an audio signal of a bilabial stop-consonant speech sound /b/ is dubbed onto the visual velar /g/. However, when the audio and visual information are combined in an opposite way with the visual bilabial dubbed onto an audio velar, the fused perception does not "occur. Instead, a McGurk combination is perceived in which participants report a combined syllable such as /daba/ (McGurk and MacDonald 1976; MacDonald and McGurk 1978). This sound is derived from combining the consonant that they saw and the vowel that they heard. This result is typically experienced as slightly imperfect by comparison with the normal case in which acoustical and optical stimuli are in agreement. The observers cannot tell what the nature of the imperfection is. They are not able to say that it is because they heard one thing and saw something else being said. The conclusion is the McGurk effect. The McGurk effect is observed in both English speaking children and adults (McGurk and MacDonald 1976; Rosenblum, Schmuckler et al. 1997; Cienkowski and Carney 2002). Facial movements are a powerful component of communication is that accompanies speech production because of its influence on auditory speech perception. In particular, talking face to face can improve intelligibility of speech sounds in a noisy environment (Erber, 1969; MacLeod & Summerfield, 1987, 1990; Middleweerd & Plomp, 1987; Sumby & Pollack, 1954). In addition, seeing the face of a talker can help speech understanding even when the auditory signal is clear (Arnold & Hill, 2001; Reisberg, McLean, & Goldfield, 1987). The power of visual speech further indicates the influence on auditory perception occurs with the McGurk effect (see, e.g., McGurk & MacDonald, 1976), where different auditory and visual signals combine to form a new percept that was not presented in either modality alone. A typical example is when an auditory syllable (e.g., /ba/) is presented in synchrony with a visual image of a talker articulating a different syllable (e.g., /ga/). Here, observers generally perceive a syllable (e.g., /da/) that has not been presented either visually or auditorily but that represents a combination of both sources of information. Consonants are produced by constricting the vocal tract at some point along its length. These sounds are classified into different types according to the degree and nature of the constriction. The types are stops, affricates, fricatives, nasals, and approximants. Vowels are usually voiced and are relatively stable over time Moore, 1997).
Cultural Differences
Studies of cultural differences indicate that people with different language backgrounds exhibit different levels of the McGurk effect. Some languages are not sufficiently ambiguous to clearly show the McGurk effect (Sekiyama and Tohkura 1993; Calvert, Bullmore et al. 1997; Sekiyama 1997; Gentilucci and Cattaneo 2005; Sekiyama and Burnham 2008). In these cases, visual input has little effect on the perception of speech sounds. It is hypothesized that different levels of language experience require the activation of different parts of the brain (Sekiyama and Tohkura 1993; Sekiyama and Burnham 2008). Japanese people area also susceptible to the effect but, as comparative research (Sekiyama & Tohkura, 1993; Sekiyama, 1997) has shown, Japanese and Chinese are more auditory and less visual than Americans! The Nacalian theory - that Japanese are visio-occular-centric as opposed to logo-phono-centric - is thus disproved, game, set, match? Very possibly. Some possible explanations for the weaker McGurk effect among Japanese include 1) The Japanese are more auditory than Westerners (c.f. Ishii, Reyes, & Kitayama, 2003) 2) The Japanese move their face less when speaking due to the nature of their language (unlikely due to experiments comparing Spanish and Japanese) 3) The language has more tones (Sekiyama, 1997) 4) The Japanese move their faces less during speaking for cultural reasons. 5) The Japanese watch faces less, or in other words they are less "face-aversive" (Sekiyama, 1997) for cultural reasons. Differences in auditory-visual speech perception between Japanese and Americans: McGurk effect as a function of incompatibility. The latter two explanations - both of which seem true - but neither bode well for Nacalian interpretations of Japanese culture. Why do the Japanese move their faces less, and look at others faces less if they are really a visually sensitive culture? Possible reasons for the apparent lack of attention to, and lack of facial information include 1) That the Japanese are paying attention to a different part of the face - eyes - rather than mouths, as demonstrated by Yuki's research (Yuki, Maddux, & Masuda, 2007). 2) Relatedly, that the Japanese do communicate with their face but not linguistically, and that their face is reserved for displaying purely visual, not language-supplemental, information. Consider for instance the close ups of people eating food shown so often on Japanese TV, or the insets of Japanese television personalities faces showing reacting non-verbally to the image on the screen. 3) That the Japanese identify more strongly with their faces and wish to maintain an unchanging persona or mask (Watsuji, 1937) for the sake of visual self-consistency.
Conclusion When discussing the McGurk Effect and speech perception, one is seldom really concerned about either of them alone, but in fact about essential aspects of language. Speech is a complex stimulus varying in both frequency and time. A basic problem in the study of speech perception is to relate speech wave properties to specific linguistic units. A second problem is finding cues in the acoustic waveform that clearly indicates a particular linguistic unit. Often times, a phoneme will only correctly be identified if information obtained from a word or syllable is utilized. Speech is perceived and processed in a different ways from non-speech stimuli to no-verbal cues. Speech intelligibility is relatively unaffected by severe distortions of the signal. Speech is an effective method of communication, which remains reliable under difficult conditions (Moore, 1997).

Works Cited
Fant, G. (1973). Speech Sounds and Features. Cambridge, MA: The MIT Press.
Goldstein, E.B. (2010) Sensation and Perception. 8th edition. California: Cengage.
Lobacz, P. (1984). Processing and Decoding the Signal in Speech Perception. Helmut Buske Verlag Hamburg.
MacDonald, J., Andersen, S., & Bachmann, T. (2000). Hearing by eye: how much spatial degradation can be tolerated? Perception, 29(10), 1155-1168.
McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746-748.
Moore, B.C.J. (1997). An Introduction to the Psychology of Hearing. (4th ed.) San Diego, CA: Academic Press.
Rosenblum, L. D., Schmuckler, M. A., & Johnson, J. A. (1997). The McGurk effect in infants. Percept Psychophys, 59(3), 347-357.
Sumby, W. H., & Pollack, I. (1954). Visual Contribution to Speech Intelligibility in Noise. The Journal of the Acoustical Society of America, 26(2), 4.
Thomas, S. M., & Jordan, T. R. (2004). Contributions of oral and extraoral facial movement to visual and audiovisual speech perception. Journal of Experimental Psychology: Human Perception and Performance, 30(5), 873-888. doi:http://dx.doi.org/10.1037/0096- 1523.30.5.873
Yuki, M., Maddux, W., & Masuda, T. (2007). Are the windows to the soul the same in the East and West? Cultural differences in using the eyes and mouth as cues to recognize emotions in Japan and the United States. Journal of Experimental Social Psychology.