FER-CHC: Facial expression recognition with cross-hierarchy contrast

Abstract

Facial expression recognition (FER) tasks with convolutional neural networks (CNNs) have seen remarkable progress. However, these CNN-based approaches do not well capture detailed and crucial features that can distinguish different facial expressions from a global perspective. There is still much room for improvement in the performance of existing CNN-based models for FER. To address this, we propose a novel cross-hierarchy contrast (CHC) framework called FER-CHC for FER tasks. FER-CHC employs a contrastive learning mechanism to utilize these crucial features in improving the performance of CNN-based models for FER. Specifically, FER-CHC utilizes CHC to regularize the feature learning of the backbone network and enhance global representations of facial expressions. The CHC captures common and differential features from different facial expressions with a cross-hierarchy contrast mechanism. Furthermore, a fusion network globally integrates the features learned from both the backbone network and CHC to learn a more robust feature representation. We conducted comprehensive experiments on six popular datasets: CK+, FER2013, FER+, RAF-DB, AffectNet, and JAFFE. The results show that our proposed FER-CHC achieves state-of-the-art performances on these datasets. Additionally, an ablation study was conducted to demonstrate the effectiveness of the proposed components in FER-CHC.