0
$\begingroup$

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception that the candidate output words are simply an ordered list and there are no probabilities associated with them.

This (https://web.stanford.edu/class/cs124/lec/languagemodeling.pdf) does a nice job of explaining perplexity and from the description of "intuitive perplexity" I thought I may be able to estimate perplexity by simply averaging the correct location of the output words. ie.. If the correct words, for 3 different phrases, shows up on the list at positions 40,50,60, then the perplexity would be 50. Trying this with a standard statistical LM gave quite a bit higher (worse) results than the normal perplexity calculation so I suspect this isn't correct.

A second way I thought might work was to try to assign a distribution to the words based on their rank on the list but I'm unsure of the exact math here. Experimentally assigning a probability to each predicted word of 1/rank, normalized by the sum all ranks to 1.0, gives numbers that are ballpark of what I'd expect but I wonder if this is mathematically valid.

Is there a correct way to get an estimate of perplexity (even a pseudo estimate just for rough comparison purposes) of ranked list of words like this?

$\endgroup$
  • $\begingroup$ Perplexity is usually defined in terms of the probability of the predicted words. So it seems that you either have to (1) go from a ranked list to a probability or (2) construct a new definition of perplexity that only takes a ranked list, not a probability. In either case, there are any number of alternative strategies that one might employ, but I don't think there's an obviously best strategy. Particularly, I don't think that there's a good solution to go from a ranked list to probabilities, since ranks have discarded all of the fine-grained, continuous-valued information. $\endgroup$ – Sycorax says Reinstate Monica Aug 18 '18 at 16:43
0
$\begingroup$

After a bit of research it appears there is no "right" way to do this.

You can get close by assigning probabilities to the word's rank on a list but the two methods I tried were not accurate enough for a good comparison.

  1. Use 1/rank normalized to 1.0. Est perplexity = 132. Actual = 100
  2. Use the unigram prob curve. Est perplexity = 156. Actual = 100

There is a paper Estimation of gap between current language models and human performance where they ran into a similar need. The authors used 2 different measurement criteria, mean-log-rank (Rank) and percent-words-correct (Top1). In the paper, they ran these metrics against a number of different language models and calculated perplexity too. Table #2 of the paper is a few dozen models with perplexity measured along with the two new metrics. There is a reasonably good correlation between all three metrics so, using the table, you can estimate perplexity (PPL), given "Rank" or "Top1".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.