I have high-dimensional events that I want to feed into an LSTM.

I was planning to pre-learn a context embedding (like word2vec), as opposed to learning an embedding as the first layer in the network, so that I can use the same embedding for multiple different prediction tasks, but I don't know if this is the best idea. If I end up doing it this way, is it best to separate out a portion of the training set to learn the embeddings, or can I learn the embeddings over the whole training set before re-using the training set to train the whole network?

I am leaning towards using the whole training set to learn the embeddings, as it feels analogous to the typical setup of adding a single embedding layer to the whole model, but I'm not confident in my assumptions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.