Most of us assume we are pretty good at reading how someone feels. We watch their face, listen to their voice, and trust that the overall picture adds up. But new research suggests that when those signals conflict, the words people speak can be the most confusing cue of all.
A study published in Psychology International examined what happens when facial expressions, tone of voice, and spoken content each convey a different emotion at the same time. Researchers at the University of Foggia and Pegaso University in Italy recruited 47 undergraduate students and showed them short videos in which a single actor deliberately expressed conflicting emotional information across all three channels. Participants then indicated which emotion they perceived overall, selecting their response by fixing their gaze on one of seven emoji displayed on a screen.
The use of eye-tracking technology as a response method was itself a notable feature of the study. Rather than asking participants to press a button or click a mouse, the researchers recorded where each person looked to register their answer. This gaze-based approach offered a way to capture integrated emotional judgements without relying on manual responses.
When all three channels conveyed the same emotion, recognition was consistently accurate. But when the channels conflicted, performance dropped sharply across the board. The steepest decline occurred for semantic content, meaning the actual words spoken. Facial expression produced a moderate drop in accuracy, while vocal tone proved the most resilient of the three.
This finding is counterintuitive in some respects. Many people might expect facial expressions to dominate emotional perception, as eye contact and facial reading are widely considered central to social communication. The results suggest instead that when emotional meaning is ambiguous, people tend to rely more on how something is said than on what the face is doing, and that the verbal content of speech is particularly susceptible to being overridden by the other channels.
The researchers also found that when participants had to follow just one emotional cue under conditions of conflict, they most often aligned with vocal tone, followed by facial expression, and least often with the spoken words. This pattern held up across different analytical approaches, including statistical models that accounted for individual differences between participants.
The study does carry limitations worth noting. The stimulus set was relatively small and not fully balanced across emotions and conflict types, and all the actors and spoken content were Italian. These factors may limit how broadly the findings apply to other languages and cultural contexts. The researchers acknowledged that the stimuli were constructed under controlled conditions and may not fully capture the richness of real-world emotional exchanges.
Future work is expected to explore larger and more naturalistic datasets, and to examine whether the same pattern emerges in virtual environments or during live interaction. The team also highlighted the potential value of combining eye-tracking data with computational methods for analysing facial and vocal signals.
The research adds to a growing body of work on how the brain integrates conflicting emotional information in everyday life.
