A calculation cost reduction method for a log-likelihood maximization in word2vec

Word2vec models learn text data and provide distributed representations to words. The distributed representations use vectors which show the meaning of the words. Thus the word2vec models are useful for Natural Language Processing (NLP). However, it is difficult to update the models for new data addition because it takes a long time to generate the word2vec model. This calculation time has become an impediment to analize text data which contains a lot of unknown words. This is caused by computational time in the calculation of the likelihood function. The purpose of this study was to speed up the training of Continuous Bag-of-Word Model(CBOW), which is one of the word2vec models, by reducing the calculation cost of the likelihood function. The likelihood function in CBOW has been expressed by the use of a softmax function and has a huge amount of computational time. In this paper, a sigmoid function replaces the softmax function as the approximated likelihood function, because the sigmoid function can reproduce the charactaristic change of the likelihood function in CBOW.