I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception that the candidate output words are simply an ordered list and there are no probabilities associated with them.
This (https://web.stanford.edu/class/cs124/lec/languagemodeling.pdf) does a nice job of explaining perplexity and from the description of "intuitive perplexity" I thought I may be able to estimate perplexity by simply averaging the correct location of the output words. ie.. If the correct words, for 3 different phrases, shows up on the list at positions 40,50,60, then the perplexity would be 50. Trying this with a standard statistical LM gave quite a bit higher (worse) results than the normal perplexity calculation so I suspect this isn't correct.
A second way I thought might work was to try to assign a distribution to the words based on their rank on the list but I'm unsure of the exact math here. Experimentally assigning a probability to each predicted word of 1/rank, normalized by the sum all ranks to 1.0, gives numbers that are ballpark of what I'd expect but I wonder if this is mathematically valid.
Is there a correct way to get an estimate of perplexity (even a pseudo estimate just for rough comparison purposes) of ranked list of words like this?