Real-time Facial Expression Recognition (FER) system for virtual meetings using joint learning method

Abstract

Ever since the whole world was being hit by the global pandemic, the lifestyle of the people has been drastically impacted. Virtual meetings, seminars and online lessons have started to become the new norm since due to the social distancing measures being implemented as well as the convenience it brings. The pandemic has made people realized that having virtual meetings not only reduces the risk of being exposed to an airborne disease, it also saves cost and time. However, the down side to virtual meetings is that speakers and audience tends to have lesser dynamics and speakers often felt difficult to get a grip of what their audiences’ reaction are, even having all their faces displayed on the screen. This is where facial expression recognition would come in place. Facial Expression Recognition (or known as FER) is a field where algorithms would help automatically recognizes the expression/emotions of people based on their facial features. FER using computer vision in particular is not a new topic as there has been plenty of studies being conducted throughout recent years. However, many has figured that exist challenges such as for a computer to accurately recognize a person’s expression through its facial features as every person express their emotions on their face differently due to their unique biometric features. Therefore, this project introduces a real-time facial expression recognition system where it would be able to accurately classify them into the 7 basic expressions which includes neutral, happy sad, angry, fear, disgust and surprise. It will be done by using the concatenation of facial identity and expression recognition pre-trained model. This project was able to improve the overall classification accuracy of the automatic recognition of facial expression, using facial identity parameters as an additional feature whereby a 7-class classification accuracy of 97.10% is being achieved on CK+ dataset while 76.72% is obtained on the FER2013. A real-time application of the system is also being demonstrated.