1
$\begingroup$

Say that we are trying to train a language model with an RNN/LSTM i.e. the inputs are words in a sentence and the outputs are the same words shifted by one such that for each input word the output is the next word. Also let's say that instead of using one-hot encoded inputs we are using word vectors. My question is that even if the inputs are word vectors, are the output labels still one-hot encoded vectors? If so, has anyone ever tried making the output labels word vectors also?

$\endgroup$

1 Answer 1

0
$\begingroup$

Yes that has been tried. The outputs are probability distributions over all tokens in the vocab. Each distribution is the result of a softmax on the decoder output. Argmax to get the predicted token.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.