Authors - Deepak Mane, Ashwanth Nair, Nihar Gundale, Om Khamkar, Tanmay Kulkarni, Ranjeet Bidwe, Amol Kamble, Suraj Sawant Abstract - Accurate emotion recognition remains a significant challenge in affective computing, particularly when relying on unimodal approaches such as facial expression analysis. These systems are inherently limited because individuals can deliberately mask their emotions, and visually similar expressions such as fear and surprise often lead to misclassification. Such limitations highlight the need for more robust methods that incorporate complementary sources of information. The proposed system uses a multimodal framework which combines the Circumplex Model of Affect through its visual and physiological cues to achieve better reliability. The FER-2013 dataset provides data for a Convolutional Neural Network which estimates emotional valence based on facial expressions captured through standard camera systems. The MAX30102 photoplethysmography sensor measures heart rate and heart rate variability through its connection with an Arduino to determine emotional arousal. The rule-based fusion engine combines these modalities to determine the final emotional state which it then categorizes into joy, stress, anxiety, and calmness. The system uses physiological data to clarify between emotional states which appear similar and it also identifies hidden emotional states which facial expressions cannot express. The system offers health monitoring, human computer interaction, and psychological assessment fields a dependable and efficient solution.
Open Zoom