All Questions
Tagged with perplexity natural-language
8 questions
1
vote
0
answers
16
views
Accounting for edge cases without training on the test set
I'm fine-tuning a large language model to predict binary sentiment, where a false negative is far more costly for my use case than a false positive. I've used weighted cross-entropy to account for ...
0
votes
1
answer
484
views
Perplexity for different n-gram models
I'm training a Lidstone Model with different n-gram sequences to see witch one is the best (2-gram, 3-gram, 4-gram, etc) in the same text database.
When I give all these models an unseen text sample ...
- 3
1
vote
1
answer
441
views
How to avoid underflow of the probability of sentence in calculating the perplexity of corpus
I am looking at this post How to find the perplexity of a corpus. I understand the whole post, but
the probability of a sentence appear in a corpus, in a unigram model,
is given by p(s)=∏ni=1p(wi), ...
- 13
2
votes
2
answers
1k
views
Calculating perplexity with smoothing techniques (NLP)
This question is about smoothed n-gram language models. When we use additive smoothing on the train set to determine the conditional probabilities, and calculate the perplexity of train data, where ...
- 23
3
votes
1
answer
187
views
Breaking substitution cipher with language model
Frequency analysis is a common tool used to break substitution ciphers, but often relies on intuition and guesswork of a human. Since language models can objectively calculate perplexity (how ...
0
votes
0
answers
324
views
Normalization of corpus to find perplexity
In the formula of finding the perplexity of a corpus, why is it normalized based on the total number of words?
Why shouldn't be normalized based on number of sentences? If # of sentences is used for ...
4
votes
1
answer
3k
views
Language Model compare probability scores between Length varying sentence
My question is : How can I compare Language Model(LM) score for two sentences with different lengths ?
Probabilities are < 1...
13
votes
2
answers
14k
views
Perplexity and cross-entropy for n-gram models
Trying to understand the relationship between cross-entropy and perplexity. In general for a model M, Perplexity(M)=2^entropy(M) . Does this relationship hold for all different n-grams, i.e. unigram, ...
- 131