Towards a Comprehensive Approach to Complex Emotion Detection: Utilizing Facial and Speech Inputs in a 2D Matrix

This study presents a novel approach to detecting complex emotions by integrating facial and speech cues using a hierarchical, rule-based system. Facial expressions have been evaluated using a Convolutional Neural Network (CNN) trained on the FER2013 dataset, whereas speech cues were processed using a Multi-Layer Perceptron (MLP) trained on the RAVDESS and TESS datasets. The integration mechanism employs a predefined 2D Emotion Matrix, mapping combinations of basic emotions to complex emotions. Phase 1 demonstrates the system's capability to detect and integrate emotions effectively, with Phase 2 focusing on validation and dataset expansion using participant feedback and generative AI. Validation with IEMOCAP and experimental datasets highlights the system’s robustness in recognizing complex emotional states. This research aims to address the limitations of existing emotion detection models by contributing to the creation of comprehensive datasets and systems for complex emotion recognition.