Enhanced text stemmer with noisy text normalization for Malay texts

Abstract

In general, the current text stemmers for Malay texts were not developed for text stemming against social media texts. Therefore, there is a need to develop an enhanced text stemmer that is able to map morphological variants based on the characteristics of non-standard derived word patterns on social media platforms. It deals with noncompliance word patterns (also called noisy texts or micro text) such as misspelled word and texting language which are often being used as informal conversation. This paper proposes an enhanced text stemmer to perform text stemming against social media texts. The investigation focuses on different patterns of non-standard, non-derived words (mechanics, non-standard word formation, code-switching, and slang words) and also non-standard derived words. The experimental results show that the performance of the proposed text stemmer depends on how much “noise” is in social media texts.