1
$\begingroup$

i am currently training a skip-gram model on my own dataset. After each run i compare the cosine-similarity between all the vectors and get the following diagramm: cosine-similarity So my model creates each run nearly the same similarities between the vectors and does not vary much.

My loss function looks like this and around 10.000 steps it approches the value 2 asymptotically; loss function

i have no analogy dataset to get the accuracy of my current word embeddings and so far i don't know how to evaluate this model further.

My question is, can you overfit a word2vec model and are there more options to get the accuracy of the model?

$\endgroup$
0
$\begingroup$

I have loss history graphs that look exactly like yours from some of my early experiments with word embeddings. When I was getting started, I just wanted to make sure that I had implemented word2vec correctly, so I kept my vocabulary and my corpus small. (FYI, I'm not working with human language data, I'm working with protein sequences; but the problems are similar.)

I could achieve loss values that were trivially close to zero with a very small corpus. With a somewhat larger corpus, I would achieve non-zero but still very small loss values after a few epochs, then the loss history would go absolutely flat. Learning stopped.

Here's what I think was happening in my experiments: with very small samples, it is possible for your Embedding layer plus your decoder to memorize your corpus. If a word appears in multiple contexts, once all the context word vectors are in their optimal positions, you can't do any better. In support of my theory, when I enlarged my vocabulary and corpus, this behavior disappeared.

I'm going to offer an opinion, which I hope someone will correct if I am wrong: when you create an embedding, you are performing unsupervised learning. I don't think that the idea of overfitting exists or matters when you are creating an embedding. If perfect embedding representation of a large body of input data is achieved in a relatively small number of dimensions, that could be a good thing. Eventually the trained embedding will be used as a component of supervised models, and in that situation the normal overfitting issues will definitely be relevant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.