2
$\begingroup$

I have been working on an assignment where I train upon 3 corpora in 3 separate languages, and then I read in a set of sentences and use a number of models to determine the most likely language for each sentence. That part I have done. (for reference: the models I implemented were a Bigram Letter model, a Laplace smoothing model, a Good Turing smoothing model, and a Katz back-off model).

Now, I am tasked with trying to find the perplexity of the test data (the sentences for which I am predicting the language) against each language model. I have read the relevant section in "Speech and Language Processing" by Jurafsky and Martin, as well as scoured the internet to try to figure out what it means to take the perplexity in the manner above.

The examples I consistently see are not helpful at all. I have zero understanding as to what I need to be doing here. What does it mean to take the perplexity of test data? How do the language models play into it? How do the separate languages themselves factor into things?

Simply knowing an equation does me no good if I don't understand how to implement it.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Your language models can be used to estimate the probability of observing each token in the test data. If you average over all the tokens then you will have the entropy of the test data. The perplexity is just $2^\text{entropy}$

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you. I believe my problem was that my models were rather inefficient and I thought my answers must be wrong. Turns out to not be the case. As an aside: why is perplexity given more weight over entropy in determining whether or not a model is "good"? $\endgroup$
    – basil
    Commented Oct 24, 2016 at 14:09
  • 2
    $\begingroup$ Reporting improvements in perplexity makes people feel better because the larger scale makes the improvement seem more dramatic. $\endgroup$
    – Aaron
    Commented Oct 25, 2016 at 20:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.