Say that we are trying to train a language model with an RNN/LSTM i.e. the inputs are words in a sentence and the outputs are the same words shifted by one such that for each input word the output is the next word. Also let's say that instead of using one-hot encoded inputs we are using word vectors. My question is that even if the inputs are word vectors, are the output labels still one-hot encoded vectors? If so, has anyone ever tried making the output labels word vectors also?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Yes that has been tried. The outputs are probability distributions over all tokens in the vocab. Each distribution is the result of a softmax on the decoder output. Argmax to get the predicted token.
