In The Name of God
Master Thesis Defense Session
​
Computer Engineering, Artificial Intelligence Engineering​
Supervisor:
Dr. Hossein Karshenas Najafabadi
Advisor:
Dr. Hamid Reza Baradaran Kashani
Internal Reviewer:
Dr. Peyman Adibi
External Reviewer:
Dr. Marjan Kaedi
Researcher:
Sara Azima Haghighi
Date: 18 September 2022
Time: 11:00 AM
Location:
Ansari building, Third floor, Dr. Braani Hall
Topic:
Deep Learning for Multi-modal Multi-label Emotion Recognition
The importance of mental health in improving the quality of life is not deniable, and therefore it has received significant attention in human societies. Affective computing, as an interdisciplinary field between computer science and psychology, deals with the analysis and investigation of various aspects of human’s mental and psychological states, using computer algorithms. One of the important applications of this field is the recognition of affections and emotions. In the real world, recognition and understanding of emotions is done by using different forms of information such as image, sound and text. The simultaneous occurrence of different modalities allows humans to identify the expressed emotion more accurately. Additionally, the behavior and expression of human emotions have a high complexity that cannot be easily described and distinguished from each other with a simple description emotion. In this research, a deep learning approach is presented for effectively fusing different modalities to identify emotions as a multi-label classification problem. With this purpose, after encoding the information obtained from different modalities using convolutional and memory structures in the deep neural network, an attention mechanism has been used to integrate multimodal information and represent information in a common hidden space. The datasets examined in this study have consist of video films containing subtitles, which include three modalities of image, speech, and text. One of the main challenges in this type of datasets is the complexity of classification due to the imbalance of the data from different labels. To extract different emotions from such dataset with different modalities, a multi-label multimodal cost function is presented, which improves the performance of model learning process. The results obtained from the experiments show that the proposed model has a promising performance compared to many other state-of-the-art methods for emotion recognition.