# Perplexity calculation with neural nets

I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is what suggested also in this paper https://link.springer.com/chapter/10.1007/978-3-642-24809-2_15 Alternatively, I have just found out a more complex approach to perplexity calculation (https://arxiv.org/abs/1703.08864), which is with hidden state being fed into max-entropy classifier , optimized with respect to the sequence negative log likelihood . The paper compares perplexities to the SRILM toolkit, which produces similar results and gives more confidence in their approach to perplexity calculation.

I believe the second formula is similar to that of PPL = exp(NLL) as in (https://dp.readthedocs.io/en/latest/languagemodeltutorial/index.html)

The question is, why would I choose one formula over another? I suppose there are arguments out there that Log Loss and Cross Entropy are similar (https://jamesmccaffrey.wordpress.com/2016/09/25/log-loss-and-cross-entropy-are-almost-the-same/), but they produce very different perplexity results in language modelling from my experience.

Note that we can use algebra to re-arrange from one expression to the other. \begin{align} PPL &= \exp\left(-\frac{1}{N}\sum_{i=1}^N\sum_{t=1}^T \log\left(P_\Theta(w_t | h)\right) \right) \\ N \log(PPL) &= - \sum_{i=1}^N\sum_{t=1}^T \log\left(P_\Theta(w_t | h)\right) \end{align}