Do listeners use speakers’ iconic gestures to predict upcoming words?
Marlijn ter Bekke1,2, Linda Drijvers1,2 & Judith Holler1,2
1 Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, the Netherlands, 2 Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands
During face-to-face conversation, people rapidly take turns talking and use hand gestures that depict semantic meaning in addition to speech (Holler & Levinson, 2019). In this pre-registered EEG study, we investigated whether listeners use speakers’ hand gestures to predict upcoming words. Participants listened to questions asked by a virtual avatar. Each question was accompanied by an iconic gesture (or control self-adaptor movement) that preceded a short silent pause and a target word. During the pause, participants showed stronger alpha and beta desynchronization in the Gesture versus the Adaptor condition, which have been reported as markers of anticipation (Prystauka & Lewis, 2019). Moreover, gestures facilitated semantic processing of target words, as shown by less negative N400 amplitudes. A Cloze test with separate participants showed that seeing the gestures improved explicit predictions of the target words. However, how much each gesture improved predictions in the Cloze test was not related to the alpha and beta desynchronization in the EEG experiment. Altogether, these results are in line with the idea that listeners can use speakers’ iconic gesture to predict upcoming words, which may facilitate coordination during conversational turn-taking by enabling earlier response planning (Holler & Levinson, 2019). However, it is unclear to what extent pre-stimulus alpha and beta desynchronization reflect predictive processing.
References
- Holler, J., & Levinson, S. C. (2019). Multimodal language processing in human communication. Trends in Cognitive Sciences, 23(8), 639–652. https://doi.org/10.1016/j.tics.2019.05.006
- Prystauka, Y., & Lewis, A. G. (2019). The power of neural oscillations to inform sentence comprehension: A linguistic perspective. Language and Linguistics Compass, 13(9), e12347. https://doi.org/10.1111/lnc3.12347