I've successfully trained a language model using LSTMs. But I have a confusion about sampling.

On sampling, we generate a probability distribution at each time step. It will be of length vocabulary size.

Now how to select a word from it?

Take the highest probable word (maximum probability) in this distribution and pass it as the next input? On implementation, I found it a cyclic prediction nature using this approach.

Or, say select top 5-10 most probable words and select one from them?

Or, randomly select any one word from the distribution and pass that one? In this case, what is the point of language modeling at all?

How to sample a language model?

  • $\begingroup$ It's the third option. But what do you mean by "what is the point"? $\endgroup$ – shimao Aug 24 '18 at 2:41
  • $\begingroup$ My understanding is that we get a distribution with maximum probability to minimum probability among the vocabulary. From this distribution, if we sample randomly, at the end of the day, what is really happening is a random selection of word. We don't really care about the maximum probable word. $\endgroup$ – Sleeba Paul Aug 24 '18 at 16:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.