Perplexity calculation with neural nets

Question

I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is what suggested also in this paper https://link.springer.com/chapter/10.1007/978-3-642-24809-2_15 Alternatively, I have just found out a more complex approach to perplexity calculation (https://arxiv.org/abs/1703.08864), which is with hidden state being fed into max-entropy classifier , optimized with respect to the sequence negative log likelihood . The paper compares perplexities to the SRILM toolkit, which produces similar results and gives more confidence in their approach to perplexity calculation.

I believe the second formula is similar to that of PPL = exp(NLL) as in (https://dp.readthedocs.io/en/latest/languagemodeltutorial/index.html)

The question is, why would I choose one formula over another? I suppose there are arguments out there that Log Loss and Cross Entropy are similar (https://jamesmccaffrey.wordpress.com/2016/09/25/log-loss-and-cross-entropy-are-almost-the-same/), but they produce very different perplexity results in language modelling from my experience.

Answer 1

Negative log-likelihood and negative likelihood both have minima in the same location because the logarithm is a monotonic injective transformation. In terms of optimization, they are the same. In terms of numerics, one generally prefers to avoid exponentiation since this can cause overflow errors.

Note that we can use algebra to re-arrange from one expression to the other. $$ \begin{align} PPL &= \exp\left(-\frac{1}{N}\sum_{i=1}^N\sum_{t=1}^T \log\left(P_\Theta(w_t | h)\right) \right) \\ N \log(PPL) &= - \sum_{i=1}^N\sum_{t=1}^T \log\left(P_\Theta(w_t | h)\right) \end{align} $$

One reason to prefer one presentation over another is if you're reporting results and convention prefers reporting in one format or another in your subject.