# Model Perplexity explanation

Not sure if this is the place to ask about model perplexity.

I have googled some explanations but it does not seem to yield much results besides the wikipedia explanation.

Can someone explain to me what is this exactly all about preferably with an example

It's a measure of how "surprised" a model is by some test data, namely $$\mathbb{P}_\textrm{model}(d_1,\ldots,d_n)^{-1/n}$$, call it $$x$$. Equivalently, $$\mathbb{P}_\textrm{model}(d_1,\ldots,d_n) = (1/x)^n$$ . Low $$x$$ is good, because it means that the test data are highly probable under your model.

Imagine your model is trying to guess the test data one item (character, say) at a time. Then $$x$$ is effectively the "average-case" number of possibilities for each new character given all the previous ones. Low $$x$$ means the model has few options for the next character, which is good.

Here's an example. Suppose we have a data $$ABABBBABABBA\ldots$$ in which there are twice as many B's as A's. Now:

1. Model 1 says that A's and B's are independent with $$\mathbb{P}(A)=\frac{1}{3}, \mathbb{P}(B)=\frac{2}{3}$$. Test data of size $$n$$ will have probability under model 1 of about $$\frac{1}{3}^{n/3}\frac{2}{3}^{2n/3}$$, thus perplexity $$3\times 2^{-2/3}\approx 1.89$$.

2. Ah, but look closer: $$AA$$ never occurs. This suggests model 2: $$\mathbb{P}(A\rightarrow B)=1$$ and $$\mathbb{P}(B\rightarrow A)=\mathbb{P}(B\rightarrow B)=\frac{1}{2}$$. Under that model, test data of size $$n$$ will have probability about $$1^{n/3} \frac{1}{2}^{2n/3}$$, thus perplexity $$2^{2/3} \approx 1.59$$.

So in this example model 2, which picked up on a feature of the data that model 1 had missed, had the lower (better) perplexity, and would be able to predict a length-$$n$$ sequence from nothing with probability about $$1.59^{-n}$$.

• 2 parts I would like to clarify. What did you mean by highly probable under the test data? And the next part would be how did you perform the calculation for complexity Sep 6, 2015 at 3:17
• The model should be able to "look" at any data and assign it a probability. Under my model 1, for example, $ABABBBA$ would have probability $\frac{1}{3}^3\frac{2}{3}^4$ because there are 3 A's and four B's. When I said that "low" $x$ means "highly" probable, I really just meant that if you have a bunch of potential models for your data then those that have lower perplexity on a brand new chunk of ("test") data, equivalently that assign higher probability to the test data, are better in general. Here, "low" perplexity is a relative notion; there's no absolute scale of good<-->bad perplexity. Sep 6, 2015 at 6:02
• For the particular perplexity calculations shown in my answer, well, let's take the harder model 2 for example. If the data have twice as many B's as A's and no AA's then there'll be about $n/3$ lots of $A\rightarrow B$ (because there are $n/3$ A's), about $n/3$ lots of $B\rightarrow A$ and about $n/3$ lots of $B\rightarrow B$. Those transitions have probabilities $1, \frac{1}{2}, \frac{1}{2}$ respectively, so the total probability is $1^{n/3}\frac{1}{2}^{n/3}\frac{1}{2}^{n/3}$. Now raise to the power $-1/n$ to convert probability to perplexity: $1^3 2^{1/3} 2^{1/3} = 2^{2/3} \approx 1.59$. Sep 6, 2015 at 6:08
• @Creosote Sorry for the late comment, but your last sentence makes no sense to me. You can't have probability 1.59 of predicting the next term. Are you missing a minus sign in the exponent? Oct 13, 2017 at 19:38
• @stella-biderman ... well spotted, now fixed. Oct 27, 2018 at 20:47