Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Not sure if this is the place to ask about model perplexity.
I have googled some explanations but it does not seem to yield much results besides the wikipedia explanation.
Can someone explain to me what is this exactly all about preferably with an example
It's a measure of how "surprised" a model is by some test data, namely $\mathbb{P}_\textrm{model}(d_1,\ldots,d_n)^{-1/n}$, call it $x$. Equivalently, $\mathbb{P}_\textrm{model}(d_1,\ldots,d_n) = (1/x)^n$ . Low $x$ is good, because it means that the test data are highly probable under your model.
Imagine your model is trying to guess the test data one item (character, say) at a time. Then $x$ is effectively the "average-case" number of possibilities for each new character given all the previous ones. Low $x$ means the model has few options for the next character, which is good.
Here's an example. Suppose we have a data $ABABBBABABBA\ldots$ in which there are twice as many B's as A's. Now:
Model 1 says that A's and B's are independent with $\mathbb{P}(A)=\frac{1}{3}, \mathbb{P}(B)=\frac{2}{3}$. Test data of size $n$ will have probability under model 1 of about $\frac{1}{3}^{n/3}\frac{2}{3}^{2n/3}$, thus perplexity $3\times 2^{-2/3}\approx 1.89$.
Ah, but look closer: $AA$ never occurs. This suggests model 2: $\mathbb{P}(A\rightarrow B)=1$ and $\mathbb{P}(B\rightarrow A)=\mathbb{P}(B\rightarrow B)=\frac{1}{2}$. Under that model, test data of size $n$ will have probability about $1^{n/3} \frac{1}{2}^{2n/3}$, thus perplexity $2^{2/3} \approx 1.59$.
So in this example model 2, which picked up on a feature of the data that model 1 had missed, had the lower (better) perplexity, and would be able to predict a length-$n$ sequence from nothing with probability about $1.59^{-n}$.
