A Lightweight Dual-Channel Multimodal Emotion Recognition Network Using Facial Expressions and Eye Movements

Emotional understanding plays a crucial role in various fields related to human–computer interaction, emotional computing, and human behavior anal-ysis. However, traditional single-modal methods often struggle to capture the complexity and subtleties of emotional states. With the advances in eye-tracking technology and facial expression recognition technology, eye-tracking and facial expressions provide complementary insight. We combine eye-tracking and facial expressions to conduct emotional research. Combining these two types of infor-mation more comprehensively and accurately describes the emotional experience of individuals and improves upon methods using a single mode. Because human emotional changes require event induction, the events and methods of emotion induction are extremely important. We also present a data collection experiment using emotion theory in psychology. We selected three types of emotion-activat-ing images (positive, neutral, and negative) from the Chinese Affective Picture System (CAPS). We design a system to extract features from the collected data, fusing the multi-modal eye tracking and facial expressions. This system is our proposed dual-channel multi-modal emotion recognition lightweight network VGG-inspired LightNet using a convolutional neural network (CNN). This model achieved an accuracy rate of 96.25% in tests using our gathered data.