Word2vec Skip-Gram - Overfitting

Question

i am currently training a skip-gram model on my own dataset. After each run i compare the cosine-similarity between all the vectors and get the following diagramm: So my model creates each run nearly the same similarities between the vectors and does not vary much.

My loss function looks like this and around 10.000 steps it approches the value 2 asymptotically;

i have no analogy dataset to get the accuracy of my current word embeddings and so far i don't know how to evaluate this model further.

My question is, can you overfit a word2vec model and are there more options to get the accuracy of the model?

Answer 1

I have loss history graphs that look exactly like yours from some of my early experiments with word embeddings. When I was getting started, I just wanted to make sure that I had implemented word2vec correctly, so I kept my vocabulary and my corpus small. (FYI, I'm not working with human language data, I'm working with protein sequences; but the problems are similar.)

I could achieve loss values that were trivially close to zero with a very small corpus. With a somewhat larger corpus, I would achieve non-zero but still very small loss values after a few epochs, then the loss history would go absolutely flat. Learning stopped.

Here's what I think was happening in my experiments: with very small samples, it is possible for your Embedding layer plus your decoder to memorize your corpus. If a word appears in multiple contexts, once all the context word vectors are in their optimal positions, you can't do any better. In support of my theory, when I enlarged my vocabulary and corpus, this behavior disappeared.

I'm going to offer an opinion, which I hope someone will correct if I am wrong: when you create an embedding, you are performing unsupervised learning. I don't think that the idea of overfitting exists or matters when you are creating an embedding. If perfect embedding representation of a large body of input data is achieved in a relatively small number of dimensions, that could be a good thing. Eventually the trained embedding will be used as a component of supervised models, and in that situation the normal overfitting issues will definitely be relevant.