I have different N gram language models trained on a corpus. I want to show that increasing N really leads to an improvement in modeling the training set.

I don't understand how to generate random sentences using them?

Please explain mathematically using probability terms and if possible any directions to implement. I assume this sampling from the learnt distribution/ model problem must be well studied, but I am not finding any resources, just a huge number of demonstrations of randomly generated data.

I know that once I start generating words, I can stop until a stop token such as ? or . is generated.

Same problem I have while generating random sentences using character level rnns trained on the same corpus.



1 Answer 1


First, figure out a way to start, which means have a way to randomly generate $N-1$ words.

Then, at each step of the generation:

  • Pick a word, uniformly at random.
  • Form an N-gram out of the picked word and the last $N-1$ words.
  • Look up the probability of that particular $N$-gram.
  • Generate a uniform random number between 0 and 1. If that number is smaller than the probability of your $N$-gram, "accept" the new word. Otherwise, go back to the start.\

Some details left as an exercise ;)

  • $\begingroup$ any standard algorithms/variants of this, what about temperature sampling etc ? $\endgroup$
    – Rafael
    Jan 17, 2017 at 3:40
  • $\begingroup$ Not sure what you're looking for there... $\endgroup$
    – Lagerbaer
    Jan 17, 2017 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.