Pre-trained language model with feature reduction and no fine-tuning

Abstract

Pre-trained language models were proven to achieve excellent results in Natural Language Processing tasks such as Sentiment Analysis. However, the number of sentence embeddings from the base model of Bidirectional Encoder from Transformer (BERT) is 768 for a sentence, and there will be more than millions of unique numbers when the dataset is huge, leading to the increasing complexity of the system. Thus, this paper presents the feature reduction of the sentence embeddings classification with BERT to decrease the number of features and complexity by using feature reduction algorithm. With 50% fewer features, the experimental results show that the proposed system improves the accuracy by 1%−2% with 89% lesser GPU memory usage.