multiple likely ys for one instance of x: word prediction with LSTM

Question

I have a ML project that is about predicting (suggesting) the next word based on the last n words, using LSTM. The output is a softmax dense layer the size of the vocabulary that shows the probability of the next word being that word. In the end, I want to be able to suggest the most likely words so users can choose from them.

However is this model really capable of capturing the nature of the training dataset if I assume there could be multiple right answers for one input?

Let's say for example that in my data there are almost equal number of instances (chances) of having sky and sea after the words blah-blah-blah-blue. Is my LSTM model going to accurately predict that "the most likely words are sky and blue"?

To be more specific, there are 3 questions regarding this issue:

1. I know traditional neural networks try to make point estimates so they punish ambiguous predictions more to improve confidence. My understanding is that if one of the similar 2 options are slightly more likely- say sky appears slightly more often than sea in the example above. Then, the network tries to come up with weights that increase probability of sky and decrease sea rather than spitting out something like 0.51 and 0.49, which is the actual distribution in the dataset. Is this correct? And if so, does this mean that the "rank" of the most likely predictions is preserved despite the actual percentage values?

2. I have read that if I were to expect multiple correct y for one given x, I should aim at modeling the probability distribution instead, like mixture density networks. However if my idea of rank preservation in question 1 is correct, will both methods show the same results in terms of word prediction? What is the difference in the output?

3. If RNNs are not capable of displaying multiple likely answers, what should I aim to make?

Answer 1

The answer depends on how exactly does your training data looks.

• If you are predicting the next word and you have only a single word in your training data, then the data doesn't have explicit ambiguities. If you have standard NLP data, just train it as usual. From the ambiguous examples in your data (e.g. you have both "I have an animal, the animal is a dog", and "I have an animal, the animal is a cat" sentences in your data), it will learn by itself ambiguities that exist in the natural language. In prediction time, don't take the word with the highest probability as your prediction, but pick $k$ words with the highest probabilities. Alternatively, you could come up with some better rules, e.g. that the probabilities need to be higher than some $\varepsilon$, or not drastically smaller than the highest probability, etc.
• If your data is tagged in a way that already allows for the alternatives, e.g. "I have an animal, the animal is a (dog|cat)", treat it as a multi-label problem. In multi-class classification, your target words would be a one-hot vector, with only a single "correct" word. In the multi-label scenario, you have a vector of zeroes and ones. What follows, you don't use softmax as an activation but need to use sigmoids and a binary loss function. This StackOverflow.com thread describes how it is done in Keras. In such a case, for each word, it would independently make a classification.