I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is what suggested also in this paper https://link.springer.com/chapter/10.1007/978-3-642-24809-2_15
Alternatively, I have just found out a more complex approach to perplexity calculation (https://arxiv.org/abs/1703.08864), which is with hidden state being fed into max-entropy classifier
, optimized with respect to the sequence
negative log likelihood
. The paper compares perplexities to the SRILM toolkit, which produces similar results and gives more confidence in their approach to perplexity calculation.
I believe the second formula is similar to that of PPL = exp(NLL) as in (https://dp.readthedocs.io/en/latest/languagemodeltutorial/index.html)
The question is, why would I choose one formula over another? I suppose there are arguments out there that Log Loss and Cross Entropy are similar (https://jamesmccaffrey.wordpress.com/2016/09/25/log-loss-and-cross-entropy-are-almost-the-same/), but they produce very different perplexity results in language modelling from my experience.