Questions tagged [perplexity]
Perplexity is a measure uses to evaluate how well a probability model predicts a given test set. It is closely related to cross-entropy.
26 questions
0
votes
1
answer
347
views
Measuring perplexity over a limited domain in an LLM
Are there papers/a literature on measuring perplexity in using a Large Language Model such as ChatGPT/Flan over a limited domain?
I want to prompt an LLM to do movie recommendations/next job ...
- 113
1
vote
0
answers
16
views
Accounting for edge cases without training on the test set
I'm fine-tuning a large language model to predict binary sentiment, where a false negative is far more costly for my use case than a false positive. I've used weighted cross-entropy to account for ...
0
votes
1
answer
2k
views
Codebook Perplexity in VQ-VAE
I am working on VQ-VAE experiments, and I have noticed that perplexity has been used as an evaluation measure for VQ codebook. Also, most of the work including codebook perplexity as a evaluation ...
- 97
0
votes
1
answer
484
views
Perplexity for different n-gram models
I'm training a Lidstone Model with different n-gram sequences to see witch one is the best (2-gram, 3-gram, 4-gram, etc) in the same text database.
When I give all these models an unseen text sample ...
- 3
1
vote
1
answer
441
views
How to avoid underflow of the probability of sentence in calculating the perplexity of corpus
I am looking at this post How to find the perplexity of a corpus. I understand the whole post, but
the probability of a sentence appear in a corpus, in a unigram model,
is given by p(s)=∏ni=1p(wi), ...
- 13
2
votes
2
answers
1k
views
Calculating perplexity with smoothing techniques (NLP)
This question is about smoothed n-gram language models. When we use additive smoothing on the train set to determine the conditional probabilities, and calculate the perplexity of train data, where ...
- 23
3
votes
1
answer
187
views
Breaking substitution cipher with language model
Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
0
votes
1
answer
3k
views
Finding the perplexity of multiple examples
I am trying to find a way to calculate perplexity of a language model of multiple 3-word examples from my test set, or perplexity of the corpus of the test set. As the test set, I have a paragraph ...
0
votes
0
answers
324
views
Normalization of corpus to find perplexity
In the formula of finding the perplexity of a corpus, why is it normalized based on the total number of words?
Why shouldn't be normalized based on number of sentences? If # of sentences is used for ...
1
vote
0
answers
516
views
Perplexity for short sentences
I have a model that outputs short sentences and want to compare the quality of its outputs for different configurations by computing their perplexities using another model.
I tried to use the gpt-2 ...
- 11
1
vote
0
answers
240
views
Why does perplexity change with different ranges of k?
I ran a 5-fold cross-validation in R to calculate LDA perplexity for k = 2:9 using a 10% sample of my data. The output was:
...
2
votes
2
answers
587
views
Word perplexity on a subword language model
Let's have corpora
$X = x_1...x_N$
in which every word can be represented using subwords (from a fixed size vocabulary of subwords)
$x_i = x_{i,0}...x_{i,M(x_i)}$
where $M(x_i)$ is number of ...
- 31
9
votes
2
answers
13k
views
Why does larger perplexity tend to produce clearer clusters in t-SNE?
Why does larger perplexity tend to produce clearer clusters in t-SNE?
By reading the original paper, I learned that the perplexity in t-SNE is $2$ to the power of Shannon entropy of the conditional ...
- 1,942
3
votes
1
answer
674
views
Perplexity formula in the t-SNE paper vs. in the implementation
The perplexity formula in the official paper of t-SNE IS NOT the same as in its implementation.
In the implementation (MATLAB):
...
- 43
7
votes
2
answers
2k
views
Intuition behind perplexity parameter in t-SNE
While reading Laurens van der Maaten's paper about t-SNE we can encounter the following statement about perplexity:
The perplexity can be interpreted as a smooth measure of the effective number of ...
- 171
1
vote
1
answer
5k
views
LDA and test data perplexity [closed]
I've performed Latent Dirichlet Analysis on a training set of documents.
At the ideal number of topics I would expect a minimum of perplexity for the test dataset. However, I find that the ...
- 151
0
votes
1
answer
519
views
Perplexity of a Non-Statistical Language Model
I have a piece of software that, given a input phrase, returns an ordered list of the next most likely words (entire vocab is ordered 1 to n). This is essentially an Language Model with the exception ...
- 101
0
votes
1
answer
1k
views
Perplexity calculation with neural nets
I am having troubles understanding which formula to use to calculate perplexity of a neural language model. Various places online on the forums people suggest using 2^(cross-entropy) measure, which is ...
- 101
2
votes
1
answer
1k
views
Why do I get weird results when using high perpexity in t-SNE?
I played around with the t-SNE implementation in scikit-learn and found that increasing perplexity seemed to always result in a torus/circle. I couldn't find any ...
- 145
6
votes
1
answer
12k
views
Inferring the number of topics for gensim's LDA - perplexity, CM, AIC, and BIC
I am confused as to how to interpret the LDA's perplexity fluctuations with different numbers of topics, in the endeavour of determining the best number of topics. Additionally, I would like to know ...
- 441
3
votes
1
answer
4k
views
How should perplexity of LDA behave as value of the latent variable k increases?
When increasing the value of the latent variable k for LDA (latent Dirichlet allocation), how should perplexity behave:
On the training set?
On the testing set?
- 90
4
votes
1
answer
3k
views
Language Model compare probability scores between Length varying sentence
My question is : How can I compare Language Model(LM) score for two sentences with different lengths ?
Probabilities are < 1...
13
votes
2
answers
14k
views
Perplexity and cross-entropy for n-gram models
Trying to understand the relationship between cross-entropy and perplexity. In general for a model M, Perplexity(M)=2^entropy(M) . Does this relationship hold for all different n-grams, i.e. unigram, ...
- 131
2
votes
1
answer
486
views
What is the equivalent of perplexity for a continuous distribution?
Lets say I have, though some unspecified means, created a model that gives me a continuous distribution with a p.d.f $f_X$, for some variable $X$ I am modelling.
And let us say that I have a set of ...
- 3,214
1
vote
0
answers
520
views
Perplexity, cross/conditional entropy and law of total variance
I was reading about the concept of Perplexity and was thinking whether there's a connexion with the law of law of total variance, but I couldn't find any reference.
The law of total variance is:
$$...
- 750
61
votes
4
answers
47k
views
What is perplexity?
I came across term perplexity which refers to the log-averaged inverse probability on unseen data. Wikipedia article on perplexity does not give an intuitive meaning for the same.
This perplexity ...
- 4,477