I've successfully trained a language model using LSTMs. But I have a confusion about sampling.

On sampling, we generate a probability distribution at each time step. It will be of length vocabulary size.

Now how to select a word from it?

Take the highest probable word (maximum probability) in this distribution and pass it as the next input? On implementation, I found it a cyclic prediction nature using this approach.

Or, say select top 5-10 most probable words and select one from them?

Or, randomly select any one word from the distribution and pass that one? In this case, what is the point of language modeling at all?

How to sample a language model?

  • $\begingroup$ It's the third option. But what do you mean by "what is the point"? $\endgroup$
    – shimao
    Aug 24, 2018 at 2:41
  • $\begingroup$ My understanding is that we get a distribution with maximum probability to minimum probability among the vocabulary. From this distribution, if we sample randomly, at the end of the day, what is really happening is a random selection of word. We don't really care about the maximum probable word. $\endgroup$ Aug 24, 2018 at 16:41

1 Answer 1


Sampling a language model is a good way of understanding what the model has learnt.

You're right there's no point randomly sampling(with equal probability of each word at each time step t), then why have you even trained your model!?

We rather sample based on the probability distribution of $\hat{y^{t}}$. Eg: say probability of $\hat{y_{i}^{t}}=0.16$ (where $i^{th}$ index in probability distribution of our prediction for time step t) means that we select word at $i^{th}$ index with the probability 0.16. You do that in numpy using np.random.choice.

Also if you select the word with the highest probability you'll end up with the same sentence given the same initial word.

This is a great link to understand novel sampling.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.