Multimodal and Temporal Perception of Audio-visual Cues for Emotion Recognition

Esam Ghaleb*, Mirela Popa, Stelios Asteriadis

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review

2 Citations (Web of Science)


In Audio-Video Emotion Recognition (AVER), the idea is to have a human-level understanding of emotions from video clips. There is a need to bring these two modalities into a unified framework, to effectively learn multimodal fusion for AVER. In addition, literature studies lack in-depth analysis and utilization of how emotions vary as a function of time. Psychological and neurological studies show that negative and positive emotions are not recognized at the same speed. In this paper, we propose a novel multimodal temporal deep network framework that embeds video clips using their audio-visual content, onto a metric space, where their gap is reduced and their complementary and supplementary information is explored. We address two research questions, (1) how audio-visual cues contribute to emotion recognition and (2) how temporal information impacts the recognition rate and speed of emotions. The proposed method is evaluated on two datasets, CREMA-D and RAVDESS. The study findings are promising, achieving the state-of-the-art performance on both datasets, and showing a significant impact of multimodal and temporal emotion perception.

Original languageEnglish
Title of host publication8th International Conference on Affective Computing & Intelligent Interaction (ACII 2019), Cambridge, United Kingdom
Number of pages7
Publication statusPublished - 2019


  • audio-video emotion recognition
  • deep metric learning
  • multimodal and incremental learning

Cite this