I have high-dimensional events that I want to feed into an LSTM.
I was planning to pre-learn a context embedding (like word2vec), as opposed to learning an embedding as the first layer in the network, so that I can use the same embedding for multiple different prediction tasks, but I don't know if this is the best idea. If I end up doing it this way, is it best to separate out a portion of the training set to learn the embeddings, or can I learn the embeddings over the whole training set before re-using the training set to train the whole network?
I am leaning towards using the whole training set to learn the embeddings, as it feels analogous to the typical setup of adding a single embedding layer to the whole model, but I'm not confident in my assumptions.
