Losing Order with Word Embeddings?

Question

I'm reading up on word embeddings and am a bit confused. It seems there are a couple approaches:

1) Use an unsupervised approach to generate word embeddings (basically predicting the probability of a word A being in the neighborhood of a word B). Then you take the weights matrix and each row corresponds to a word embedding.

2) Add an embedding layer at the start of your neural network. It takes in a one-hot encoded document (suppose) and this weights matrix learns word embeddings that are useful for the task at hand. Maybe classifying documents by author uses word embeddings that are pretty different from approach #1.

Here's my confusion: In approach #1, I can maintain the order of words in my document by just going through and replacing each word by its word embedding.

How does approach #2 allow me to do this? I'd like to learn the word embeddings at train time but also take advantage of the order in which the words occur. If I simply one-hot encode first, I lose out on the order-based information.

Any ideas how I can both learn embeddings during training and take advantage of the words' ordering?

Thanks!

Answer 1

To keep the word order, one way is to use LSTM. The input of LSTM is a sentence, in which each word is represented as a word embedding. You feed one-hot encoding as the input of the word embedding layer. The LSTM "reads" the sentence from the first word of the sentence to the last word. The output at the last word is the target of your task (say, sentence labels).

Answer 2

The embedding layer that you add at the start of your neural network in solution #2 has to be a time-distributed layer. This means that the layer must process each time slice independently and then feed the output into your neural network in a time-series. Generally you will then use the output of the Embedding layer in a recurrent neural network of some sort.

As a concrete example, suppose you have inputs of shape (32,128,100000) where the dimensions of this tensor is (batch_size,time,one_hot) - i.e. you have 32 sentences in a batch, and each sentence is 128 words long and each word in the sentence is a 100000 dimensional one-hot vector. The output of an Embedding layer should be (32,128,300) where the last dimension is now the size of the embeddings. Notice that the embedding layer does not modify the time-dimension, hence it is time-distributed. For each time segment, you will embed the word at that time segment from a 100000 dimensional (one-hot) space to a 300 dimensional space.