Word2vec Skip-Gram - Overfitting i am currently training a skip-gram model on my own dataset. After each run i compare the cosine-similarity between all the vectors and get the following diagramm:

So my model creates each run nearly the same similarities between the vectors and does not vary much.
My loss function looks like this and around 10.000 steps it approches the value 2 asymptotically;

i have no analogy dataset to get the accuracy of my current word embeddings and so far i don't know how to evaluate this model further.
My question is, can you overfit a word2vec model and are there more options to get the accuracy of the model?
 A: I have loss history graphs that look exactly like yours from some of my early experiments with word embeddings.  When I was getting started, I just wanted to make sure that I had implemented word2vec correctly, so I kept my vocabulary and my corpus small.  (FYI, I'm not working with human language data, I'm working with protein sequences; but the problems are similar.)
I could achieve loss values that were trivially close to zero with a very small corpus.  With a somewhat larger corpus, I would achieve non-zero but still very small loss values after a few epochs, then the loss history would go absolutely flat.  Learning stopped.
Here's what I think was happening in my experiments: with very small samples, it is possible for your Embedding layer plus your decoder to memorize your corpus.  If a word appears in multiple contexts, once all the context word vectors are in their optimal positions, you can't do any better.  In support of my theory, when I enlarged my vocabulary and corpus, this behavior disappeared.
I'm going to offer an opinion, which I hope someone will correct if I am wrong: when you create an embedding, you are performing unsupervised learning.  I don't think that the idea of overfitting exists or matters when you are creating an embedding.  If perfect embedding representation of a large body of input data is achieved in a relatively small number of dimensions, that could be a good thing.  Eventually the trained embedding will be used as a component of supervised models, and in that situation the normal overfitting issues will definitely be relevant.
