Questions tagged [perplexity]

Perplexity is a measure uses to evaluate how well a probability model predicts a given test set. It is closely related to cross-entropy.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Likelihood ratio test for language modeling

I am trying to use a likelihood ratio test to evaluate whether one language model is significantly better than another. (Note: In the example below, the language models should not be very different, ...
0
votes
0answers
9 views

Normalizing by root N in perplexity

In Chapter 3 N-gram Language Models by Jurafsky & Martin (pg 8), perplexity is defined as: The perplexity (sometimes called PP for short) of a language model on a test set is the inverse ...
0
votes
0answers
43 views

Perplexity for short sentences

I have a model that outputs short sentences and want to compare the quality of its outputs for different configurations by computing their perplexities using another model. I tried to use the gpt-2 ...
0
votes
0answers
17 views

Metrics used in evaluation of topic models

I know that perplexity requires a held out corpus. I am new to coherence measures. What are intrinsic and extrinsic coherence measures? Which one of them require an external corpus like Wikipedia ...
1
vote
0answers
47 views

Why does perplexity change with different ranges of k?

I ran a 5-fold cross-validation in R to calculate LDA perplexity for k = 2:9 using a 10% sample of my data. The output was: ...
2
votes
2answers
523 views

Why does larger perplexity tend to produce clearer clusters in t-SNE?

Why does larger perplexity tend to produce clearer clusters in t-SNE? By reading the original paper, I learned that the perplexity in t-SNE is $2$ to the power of Shannon entropy of the conditional ...
2
votes
1answer
81 views

Perplexity formula in the t-SNE paper vs. in the implementation

The perplexity formula in the official paper of t-SNE IS NOT the same as in its implementation. In the implementation (MATLAB): ...
0
votes
0answers
101 views

Perplexity and accuracy

I am a little bit unsure of the results I am getting from my skip-gram model, as it says that I have a very high perplexity (>100k) but that the accuracy is also more than 50%. It seems a bit ...
4
votes
0answers
284 views

Intuition behind perplexity parameter in t-SNE

While reading Laurens van der Maaten's paper about t-SNE we can encounter the following statement about perplexity: The perplexity can be interpreted as a smooth measure of the effective number of ...
1
vote
1answer
2k views

LDA and test data perplexity [closed]

I've performed Latent Dirichlet Analysis on a training set of documents. At the ideal number of topics I would expect a minimum of perplexity for the test dataset. However, I find that the ...
0
votes
1answer
234 views

Perplexity of a Non-Statistical Language Model

I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
0
votes
1answer
595 views

Perplexity calculation with neural nets

I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is ...
2
votes
1answer
448 views

Why do I get weird results when using high perpexity in t-SNE?

I played around with the t-SNE implementation in scikit-learn and found that increasing perplexity seemed to always result in a torus/circle. I couldn't find any ...
0
votes
1answer
5k views

Inferring the number of topics for gensim's LDA - perplexity, CM, AIC, and BIC

I am confused as to how to interpret the LDA's perplexity fluctuations with different numbers of topics, in the endeavour of determining the best number of topics. Additionally, I would like to know ...
2
votes
1answer
2k views

How should perplexity of LDA behave as value of the latent variable k increases?

When increasing the value of the latent variable k for LDA (latent Dirichlet allocation), how should perplexity behave: On the training set? On the testing set?
4
votes
1answer
1k views

Language Model compare probability scores between Length varying sentence

My question is : How can I compare Language Model(LM) score for two sentences with different lengths ? Probabilities are < 1...
9
votes
2answers
5k views

Perplexity and cross-entropy for n-gram models

Trying to understand the relationship between cross-entropy and perplexity. In general for a model M, Perplexity(M)=2^entropy(M) . Does this relationship hold for all different n-grams, i.e. unigram, ...
1
vote
1answer
260 views

What is the equivalent of perplexity for a continuous distribution?

Lets say I have, though some unspecified means, created a model that gives me a continuous distribution with a p.d.f $f_X$, for some variable $X$ I am modelling. And let us say that I have a set of ...
43
votes
4answers
31k views

What is perplexity?

I came across term perplexity which refers to the log-averaged inverse probability on unseen data. Wikipedia article on perplexity does not give an intuitive meaning for the same. This perplexity ...