In the middle of applying to universities? | SHARE YOUR EXPERIENCE In the middle of applying to universities? | SHARE YOUR EXPERIENCE

Deep Learning Approaches for Multimodal Emotion Recognition

   Faculty of Computing, Engineering and the Built Environment

This project is no longer listed on and may not be available.

Click here to search for PhD studentship opportunities
  Dr George Moore  No more applications being accepted  Competition Funded PhD Project (Students Worldwide)

About the Project

Emotion recognition is the task of identifying human emotion, and is an increasingly important factor in the field of human computer interaction (HCI), with applications in healthcare, well-being, gaming, safety and security. Emotion recognition can be achieved using a variety of input modalities, ranging from video, images, voice, and text. The combination of these different input modalities such as vocal signal, vision, and verbal content have been shown to have great potential in representing a wide variety of different emotions. Traditional approaches have used feature-level fusion in combination with machine learning algorithms to achieve emotion recognition with some success, however in recent years deep learning has shown remarkable capabilities in learning detailed representations of high-dimensional image and video data, as well as audio spectral features, to achieve better classification performance. Unlike tasks that focus on a single input modality to determine emotion recognition, multimodal input techniques take into consideration a range of different input data types, such as visual and audio, without the need for more intrusive physiological data signal recordings (electrocardiogram (ECG), galvanic skin response (GSR), etc). Recent studies utilising multimodal deep learning approaches have shown promising results, however due to the complex nature of multimodal emotion recognition further challenges remain.

This project aims to explore deep learning architectures and to propose, design, and develop a multimodal deep learning pipeline for emotion recognition. This project will address the following research questions:

What is the state-of-the-art deep learning models in image and audio emotion recognition?

How can we utilise audio representative learning using deep learning models.

What deep learning models can be used to extract features from high-dimensionality data such as video and image data to generalise emotional states?

What model-level fusion strategies can be used with image and audio feature representations to achieve accurate emotion recognition?

How can we incorporate representative data sources in the development and evaluation of deep learning emotion recognition models?

​In answering these research challenges, the aim of the project is to advance the state-of-the-art in deep learning models for multimodal emotion recognition. Outputs from this research project could have potential impact in the application of emotion recognition within different fields, including healthcare, mental health and well-being.


W. Liu, J. -L. Qiu, W. -L. Zheng and B. -L. Lu, "Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition," in IEEE Transactions on Cognitive and Developmental Systems, vol. 14, no. 2, pp. 715-729, June 2022, doi: 10.1109/TCDS.2021.3071170.
Summaira, J., Li, X., Shoib, A. M., Li, S., & Abdul, J. (2021). Recent Advances and Trends in Multimodal Deep Learning: A Review. arXiv preprint arXiv:2105.11087.
L. Schoneveld, A. Othmani, and H. Abdelkawy, “Leveraging recent advances in deep learning for audio-visual emotion recognition,” Pattern Recognition Letters, 2021.
Koromilas, P.; Giannakopoulos, T. Deep Multimodal Emotion Recognition on Human Speech: A Review. Appl. Sci. 2021, 11, 7962.

How good is research at Ulster University - Belfast Campus in Computer Science and Informatics?

Research output data provided by the Research Excellence Framework (REF)

Click here to see the results for all UK universities
Search Suggestions
Search suggestions

Based on your current searches we recommend the following search filters.

PhD saved successfully
View saved PhDs