Training embedding model on entire corpus then classifying documents from that corpus using trained embeddings

Question

Let's say I have a large corpus of documents. Instead of using a pretrained embedding model, I train my own non-contextual embedding model like w2v/fasttext from scratch on all the documents and save the model. Now I take a subset of my corpus for training a classification task with a sophisticated neural network. I tokenize every document and substitute the word vectors from my embedding model and train on this.

Since the word vectors for both train and test sets come from the same embedding model, can we say that the neural network has already somewhat seen the test set? Would it be cheating? Or is it that just the co-occurrence statistics used for learning the embeddings aren't complex enough for the model to be able to cheat at the specific classification task?

Answer 1

It depends on what type of use of the model you want to simulate with your train-test split.

• If you want to simulate on-line use of the classifier, i.e., you train the classifier once, and that it is used for data instances that you never saw before. (E.g., you train a machine translation model and deploy it in a service like Google Translation). It that case, what you suggest is indeed cheating because it does not stimulate the use case and thus the results on the test set are not realistic estimation of the future performance.

• If you want to simulate an off-line use, this is probably a good thing to do. (E.g., you want to machine-translate a book and you already have the book in the source language and you should deliver the best translation possible.)