# Tag Info

27

I found this rather intuitive: The perplexity of whatever you're evaluating, on the data you're evaluating it on, sort of tells you "this thing is right about as often as an x-sided die would be." http://planspace.org/2013/09/23/perplexity-what-it-is-and-what-yours-is/

12

I've wondered this too. The first explanation isn't bad, but here are my 2 nats for whatever that's worth. First of all, perplexity has nothing to do with characterizing how often you guess something right. It has more to do with characterizing the complexity of a stochastic sequence. We're looking at a quantity, $$2^{-\sum_x p(x)\log_2 p(x)}$$ Let's ...

9

Yes, the perplexity is always equal to two to the power of the entropy. It doesn't matter what type of model you have, n-gram, unigram, or neural network. There are a few reasons why language modeling people like perplexity instead of just using entropy. One is that, because of the exponent, improvements in perplexity "feel" like they are more substantial ...

5

There is actually a clear connection between perplexity and the odds of correctly guessing a value from a distribution, given by Cover's Elements of Information Theory 2ed (2.146): If $X$ and $X'$ are iid variables, then $P(X=X') \ge 2^{-H(X)} = \frac{1}{2^{H(X)}} = \frac{1}{\text{perplexity}}$ (1) To explain, perplexity of a uniform distribution X is ...

5

As you noticed, it's good idea to have some kind of averaging. Since in LM probabilities get multiplied, geometric average seems like a good fit. From Speech and Language Processing In practice we don’t use raw probability as our metric for evaluating language models, but a variant called perplexity. The perplexity (sometimes called PP for short) of a ...

4

Counter intuitively, it appears that the log_perplexity function doesn't output a $perplexity$ after all (the documentation of the function wasn't clear enough for me personally), but a likelihood $bound$ which must be utilised in the perplexity's lower bound equation thus (Taken from this paper - Online Learning for Latent Dirichlet Allocation by Hoffman, ...

3

The larger the perplexity, the more non-local information will be retained in the dimensionality reduction result. Yes, I believe that this is a correct intuition. The way I think about perplexity parameter in t-SNE is that it sets the effective number of neighbours that each point is attracted to. In t-SNE optimisation, all pairs of points are repulsed ...

2

One cannot have perplexity values larger than sample size. [I don't have time right now, but I will try to provide a brief mathematical explanation of this later.] A popular t-SNE tutorial https://distill.pub/2016/misread-tsne/ says The image for perplexity 100, with merged clusters, illustrates a pitfall: for the algorithm to operate properly, the ...

1

I believe this is because of this mismatch in t-SNE between the input (Gaussian) and output (student-t) distributions. It is beneficial to make such blobs in order to separate from everything else as required by the long tail of the t-distribution. The repulsive forces dominate. In such cases SNE may work better. Have you tried?

1

For a given $i$, the similarities $p_{j|i}$ are obtained from Euclidean distances $d_{ij}$ via a Gaussian similarity kernel and then normalized to sum to one: $$p_{j|i} = \frac{\exp(-\beta d_{ij})}{\sum_k \exp(-\beta d_{ik})}.$$ Plugging this into the entropy formula, we obtain: \begin{align} H(p_{\cdot|i}) &= -\sum_j p_{j|i} \log p_{j|i}= -\sum_j \frac{\...

1

Negative log-likelihood and negative likelihood both have minima in the same location because the logarithm is a monotonic injective transformation. In terms of optimization, they are the same. In terms of numerics, one generally prefers to avoid exponentiation since this can cause overflow errors. Note that we can use algebra to re-arrange from one ...

1

The original paper on LDA gives some insights into this: In particular, we computed the perplexity of a held-out test set to evaluate the models. The perplexity, used by convention in language modeling, is monotonically decreasing in the likelihood of the test data, and is algebraicly equivalent to the inverse of the geometric mean per-word likelihood. A ...

1

Agreed with the @Aaron answer with a slight modification: It's not always equal to two to the power of the entropy. Actually, It will be (base for log) to the power of entropy. If you have used e as your base then it would be e^entropy.

1

Perplexity is essentially a geometric average of inverse probabilities. So for your case, a natural interpretation would be to compute the sample average over the discrete points, i.e. $$\exp\left[-\tfrac{1}{N}\sum_{i=1}^N\log\left[f\left[x_i\right]\right]\right]$$ So log perplexity would be the average of negative log likelihood over the data points. (How ...

Only top voted, non community-wiki answers of a minimum length are eligible