# Perplexity of a Non-Statistical Language Model

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception that the candidate output words are simply an ordered list and there are no probabilities associated with them.

This (https://web.stanford.edu/class/cs124/lec/languagemodeling.pdf) does a nice job of explaining perplexity and from the description of "intuitive perplexity" I thought I may be able to estimate perplexity by simply averaging the correct location of the output words. ie.. If the correct words, for 3 different phrases, shows up on the list at positions 40,50,60, then the perplexity would be 50. Trying this with a standard statistical LM gave quite a bit higher (worse) results than the normal perplexity calculation so I suspect this isn't correct.

A second way I thought might work was to try to assign a distribution to the words based on their rank on the list but I'm unsure of the exact math here. Experimentally assigning a probability to each predicted word of 1/rank, normalized by the sum all ranks to 1.0, gives numbers that are ballpark of what I'd expect but I wonder if this is mathematically valid.

Is there a correct way to get an estimate of perplexity (even a pseudo estimate just for rough comparison purposes) of ranked list of words like this?

• Perplexity is usually defined in terms of the probability of the predicted words. So it seems that you either have to (1) go from a ranked list to a probability or (2) construct a new definition of perplexity that only takes a ranked list, not a probability. In either case, there are any number of alternative strategies that one might employ, but I don't think there's an obviously best strategy. Particularly, I don't think that there's a good solution to go from a ranked list to probabilities, since ranks have discarded all of the fine-grained, continuous-valued information. – Sycorax Aug 18 '18 at 16:43