How to make a LSTM to predict next word

Question

I have made a LSTM network (hidden size 16) where I give a sequence of 10 numbers as the input and feed its output to a fully connected layer.
The numbers in the sequence are corresponding to the words in the dictionary that I created (Eg: number 12 in the sequence corresponds to the 12th word in the dictionary)

What I want is to predict the next number (which is the next word) using that fully connected layer and I'm stuck at that place.

One method I tried is to create the FC layer with one neuron in the output with relu activation and cast it's output a to integer and get the word corresponding to that number. It's accuracy is too low.

Another method that came into my mind is to create FC with multiple outputs and connect it to a softmax layer. Then get its output and convert the output to a binary number as probability < 0.5 as binary 0 and others binary 1. And then get the corresponding word. I haven't tried this method yet.

What I want to ask is whether there is a better way to do this..? What is the general solution for these kind of problems..? (A different architecture may be)

Answer 1

The most widely-used method for predicting the probability of a categorical output is to use softmax activation in the final layer and cross-entropy loss for training. This is because you're modeling the probability that each word in your dictionary comes next. Full softmax can be expensive, though, so sampled softmax and its variants can speed things up.

Another option is to observe that it's generally expensive to work with words, because there are so many of them: you'll have a large embedding layer, and a large projection layer -- that's a lot of data to store. Instead, working with characters can work very well -- there are relatively fewer English characters, so full softmax works fine, and LSTMs can essentially "learn" English from nothing.

To generate the next word, perhaps because you want your network write a new sonnet, take the output of the network as a probability vector. Then sample from a multinomial distribution. The sampled element is used as the next input to the network, and so on. This works because the output of the softmax layer is a probability vector, and you can write your sonnet word-by-word, or character-by-character.

Using ReLU as the output layer doesn't make much sense. Your outcome is categorical so you need a probability distribution over the categorical output. ReLUs output unbounded values, so it's not a probability distribution.

This article is a good place to start: Andrej Karpathy, "The Unreasonable Effectiveness of Recurrent Neural Networks"