I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is what suggested also in this paper https://link.springer.com/chapter/10.1007/978-3-642-24809-2_15 Alternatively, I have just found out a more complex approach to perplexity calculation (https://arxiv.org/abs/1703.08864), which is enter image description here with hidden state being fed into max-entropy classifier enter image description here, optimized with respect to the sequence negative log likelihood enter image description here. The paper compares perplexities to the SRILM toolkit, which produces similar results and gives more confidence in their approach to perplexity calculation.

I believe the second formula is similar to that of PPL = exp(NLL) as in (https://dp.readthedocs.io/en/latest/languagemodeltutorial/index.html)

The question is, why would I choose one formula over another? I suppose there are arguments out there that Log Loss and Cross Entropy are similar (https://jamesmccaffrey.wordpress.com/2016/09/25/log-loss-and-cross-entropy-are-almost-the-same/), but they produce very different perplexity results in language modelling from my experience.


Negative log-likelihood and negative likelihood both have minima in the same location because the logarithm is a monotonic injective transformation. In terms of optimization, they are the same. In terms of numerics, one generally prefers to avoid exponentiation since this can cause overflow errors.

Note that we can use algebra to re-arrange from one expression to the other. $$ \begin{align} PPL &= \exp\left(-\frac{1}{N}\sum_{i=1}^N\sum_{t=1}^T \log\left(P_\Theta(w_t | h)\right) \right) \\ N \log(PPL) &= - \sum_{i=1}^N\sum_{t=1}^T \log\left(P_\Theta(w_t | h)\right) \end{align} $$

One reason to prefer one presentation over another is if you're reporting results and convention prefers reporting in one format or another in your subject.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.