Developing a Hybrid Word Embedding Model Using Contextual Information and Classification Algorithms

Abstract

Natural language processing (NLP) has evolved considerably due to the fact that word embeddings allow presenting the words in dense vectors that can capture the semantic meaning. Common models such as Word2Vec and GloVe, however, use context free embeddings and are unable to differentiate polysemous words in separate contexts. Contextual models such as ELMo and BERT are helpful in this regard, building contextual embedding. These models however have high computing power requirements and do not necessarily work well with classification algorithms. To avoid such issues, this paper proposes a hybrid word embedding model which consists of transformer-based contextual word embedding (BERT) and robust classification algorithms (Random Forest, Logistic Regression, and SVM). The current models divide the embeddings and classifiers. We have an approach that is a combination of both that is, word sense disambiguation (WSD) and text categorization is more precise. The model was tested on such benchmark datasets as the 20 Newsgroups dataset and the IMDb movie reviews dataset. These datasets included such activities as topic classification and sentiment analysis. Its predictive accuracy, accuracy, recall, and F1-score were 86, 86, 87, 85 respectively as regards WSD. Compared to other techniques such as Word2Vec + TF-IDF (84% accuracy) and Logistic Regression with TF-IDF (68% accuracy), this was superior. These findings indicate that the hybrid method does not only capture contextual semantics more effectively, but it is also better in classification. This renders it a solution that can be applicable in a great number of various NLP situations.