Questions tagged [laplace-smoothing]

Laplace smoothing (also known as additive smoothing) is a technique associated with a probability regularisation task. It ensures that certain improbable outcomes are still associated with a minimum probability of occurrence.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

How to replace empirical frequency predictions with Laplace estimates

I have frequency predictions for a discrete distribution: $p(x_1)=0$, $p(x_2)=0$, $p(x_3)=0.05$, $p(x_4)=0.95$ I need to smooth the distribution so I don't have zero values. I think the solution is ...
nafrtiti's user avatar
  • 675
1 vote
1 answer
63 views

What is the influence of initial state in sequence generated from a markov chain?

For thousands of item, I have observations about their state (a letter) for 9 timestep. From that, I build a transition matrix (RotationMatrix by couting their ...
Sarahdata's user avatar
  • 151
1 vote
1 answer
55 views

Impact of Laplace smoothing on likelihood in Naive Bayes

When 1 is added to word count in Laplace Smoothing in Naive Bayes, the new probabilities either increase or decrease as shown below. Though the problem of "zero" probability has been solved. ...
Arpit Saxena's user avatar
1 vote
0 answers
28 views

Laplace Smoothing in Naive Bayes [duplicate]

I'm reading up on Laplace Smoothing/Add-1 Smoothing in Naive Bayes and I'm given the formula $ \frac{Count(Feature=Value) + α}{N + α\cdot k} $. In reference to the image above, if we have to classify ...
Dithering's user avatar
4 votes
1 answer
2k views

What attributes does Laplace Smoothing apply on in Naive Bayes

Consider the dataset: Outlook Temperature Humidity Play Golf? Overcast Cool Low Yes Sunny Hot Low Yes Rainy Cool High No Sunny Hot High No Rainy Cool Low Yes There are 3 possible values for the ...
UnsurelyStuck's user avatar
0 votes
1 answer
3k views

How can Naive Bayes overfit the data?

I know that Laplace smoothing results in a high bias of Naive Bayes. If the value of the smoothing parameter (alpha) is large, then the probability distribution will be uniform for all the features. ...
Mauj Mishra's user avatar
1 vote
1 answer
870 views

How to compute KL-divergence when there are categories of zero counts?

I have two very large discrete frequency distributions (about 4 million items), and each contains many items with counts of 0. I want to calculate the KL divergence between them and use the empirical ...
joshisanonymous's user avatar
2 votes
2 answers
1k views

Calculating perplexity with smoothing techniques (NLP)

This question is about smoothed n-gram language models. When we use additive smoothing on the train set to determine the conditional probabilities, and calculate the perplexity of train data, where ...
Janani K's user avatar
2 votes
2 answers
574 views

Naive Bayes - having trouble coming up with a case where Laplace smoothing changes the prediction

I'm thinking through the logic of Naive Bayes and encountered a brain teaser. I know that adding smoothing (alpha) to Naive Bayes can help to increase the accuracy of the model, which implies that it ...
mythander889's user avatar
0 votes
0 answers
308 views

How should I handle Laplace smoothing in Naive Bayes in this example?

I have a toy dataset on animals, with 4 features and 2 possible classes (mammals vs non-mammals). I have summarized the dataset ...
Mehdi Saffar's user avatar
1 vote
2 answers
527 views

Why do we need to apply Laplace smoothing to all the words in Naive Bayes for text classification?

I understood that we need to apply for Laplace smoothing to the words that are not present in our training data. But then why/what is the need to do Laplace smoothing for all the words (even the words ...
Tushar Tiwari's user avatar
0 votes
1 answer
797 views

How is Laplace Smoothing used in this example of Binary classification in Naive Bayes

I am following CS229 course by Andrew Ng. On this lecture note it talks about using Laplace smoothing to bypass situations of 0-probabilities. What does not make sense is the immediate jump to the ...
Shiladitya Bose's user avatar
0 votes
1 answer
3k views

Hoes does laplace smoothing in Naive Bayes control high bias and high variance?

I'm trying to understand how laplace smoothing exactly helps to balance between overfitting and underfitting. I know that Laplace smoothing is used as a fail safe probability if there's a any ...
Jeeth's user avatar
  • 103
1 vote
1 answer
2k views

Global sensitivity of mean and variance in differential privacy?

Please explain me why global sensitivity of a mean or variance queries will be (b-a)/n and (b-a)^2/n where b is the upper ...
SteveS's user avatar
  • 111
0 votes
0 answers
127 views

Laymen's description of Laplace Smoothing

I have understood that Laplace Smoothing will provide a small non-zero value to the probability score. But still, I am missing something. Please if anyone can provide a better description, it will be ...
Anroop's user avatar
  • 1
13 votes
3 answers
307 views

Terminology for Bayesian Posterior Mean of Probability with Uniform Prior

If $p \sim$ Uniform$(0,1)$, and $X \sim$ Bin$(n, p)$, then the posterior mean of $p$ is given by $\frac{X+1}{n+2}$. Is there a common name for this estimator? I've found it solves lots of people's ...
Cliff AB's user avatar
  • 20.7k
0 votes
1 answer
1k views

What is the interpretation for the priors in the derivation of Laplace smoothing? [duplicate]

Laplace smoothing has a generalisation that can be justified with the use of Bayes formula. Let $f(x;\alpha,\beta)$ be the (non-normalised) beta distribution, i.e. $$f(x;\alpha,\beta) = x^{\alpha-1}(...
Phoenix87's user avatar
  • 101
0 votes
1 answer
1k views

R Naive Bayes and Laplace: Even turned off, works fine with unseen words in test data?

I'm trying to better understand Laplace+1 smoothing on Naive Bayes for text classification. Using the e1071 package in R, naiveBayes() function, I get some confusing results. If I fit a model using ...
userninenineninenine's user avatar
2 votes
1 answer
2k views

What attributes to apply laplace smoothing in naive bayes classifier?

I am reading naive Bayes classifier from the book "Data mining practical machine learning tools and techniques". The example of naive Bayes is given using the below dataset. As (Outlook=Overcast | ...
Jahir Islam's user avatar
5 votes
1 answer
726 views

Smoothing/shrinking the predicted probability of a classifier to reduce live logloss

Let us assume we work on a 2 -class classification problem. In my setting the sample is balanced. To be precise it is a financial markets setting where up and down have approximately 50:50 chance. The ...
Richi W's user avatar
  • 3,436
2 votes
1 answer
3k views

Why do we need Laplace smoothing in Naive Bayes while logarithm may resolve the problem?

In Naive Bayes algorithm, we use $$P(c)P(x_1|c)P(x_2|c)...p(x_n|c)\space\space (*)$$ to decide about the class of a sample $\textbf{x} =(x_1,...,x_n)$. It is possible that for a class $c$, a feature $...
Hossein's user avatar
  • 3,454
2 votes
1 answer
312 views

How to do multinomial regression when baseline choice percentage of one choice item is 0%

I am trying to find a clever way to analyze the choice ratios between a baseline condition and a different condition. Specifically, I have a baseline condition, where the choice ratio between two ...
Desperate in Statistics's user avatar
1 vote
1 answer
8k views

Laplace smoothing and naive bayes

If I want to use naive bayes with laplace smoothing and therefore add 1 to probabilities with the value of 0, what does this mean for probabilities which have the actual value of 1?
link's user avatar
  • 19
0 votes
1 answer
4k views

Naive Bayes with Laplace Smoothing Probabilities Not Adding Up

Let c refer to a class (such as Positive or Negative), and let w refer to a token or word. Define $count(w,c) = $ $counts \ w \ in\ class \ c$ $count(c) = counts \ of \ words \ in \ class \...
BayesTesting's user avatar
3 votes
2 answers
4k views

Laplace smoothing understanding implementation

Considering the data set given below Here if we have to classify new data point: D15 (O=Overcast, T=Cool, H=High, W=Strong) Then for P(No|Overcast, Cool, High, Strong) we have, (5/14) * 0 * (1/5) *...
Cybercop's user avatar
  • 151
1 vote
1 answer
4k views

A simple numerical example for Kneser-Ney Smoothing

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...
Mohammad Ali Nematollahi's user avatar
5 votes
2 answers
10k views

Understanding Add-1/Laplace smoothing with bigrams

I am working through an example of Add-1 smoothing in the context of NLP Say that there is the following corpus (start and end tokens included) ...
basil's user avatar
  • 173
2 votes
1 answer
123 views

bayesian classification unknown domain

Suppose I am building a naive Bayes model to classify text messages as either spam or legit. I am training my model using a dataset containing both classes and for which I know the domain (the number ...
Imlerith's user avatar
0 votes
1 answer
908 views

Regarding probabilites for naiveBayes algo

I have trained my data with naiveBayes algo in e1071 package. I have 6 classes in my data. I have predicted test data. the prediction returns only one class for each data point but I would like to ...
Gaurav Khatwani's user avatar
11 votes
2 answers
13k views

Calculating Emission Probability values for Hidden Markov Model (HMM)

I'm new to HMM and still learning. I'm currently using HMM to tag part-of-speech. To implement the viterbi algorithm I need transition probabilities ($ a_{i,j} \newcommand{\Count}{\text{Count}}$) and ...
Ramesh-X's user avatar
  • 373
1 vote
2 answers
2k views

Kneser-Ney for unigrams?

I was wondering if it is at all possible to use Kneser-Ney to smooth word unigram probabilites? The basic idea behind back-off is to use (n-1)-gram frequencies when an n-gram has 0 count. This is ...
twowo's user avatar
  • 202
4 votes
1 answer
6k views

Is the Laplace/Lidstone smoothing parameter (talking about Multinomial/Bernoulli Naive Bayes) related to the particular structure of the dataset?

I'm working with Multinomial and Bernoulli Naive Bayes implementation of scikit-learn (python) for text classification. I'm using the 20_newsgroups dataset. From the scikit documentation we have: <...
Trevor's user avatar
  • 41
1 vote
1 answer
735 views

Alternative Smoothing techniques for Naive Bayes?

I've always wondered why no one using smoothing techniques for Naive Bayes which implicitly test which conditional probabilities should actually be used and which ones shouldn't. The formula for ...
ALutes's user avatar
  • 11
1 vote
1 answer
667 views

Choosing smoothing parameters across multiple Naïve Bayes classifiers with different number of categories

I would like to train multiple Naïve Bayes classifiers with different number of categories, and also have a global threshold for how certain one classifier must be in order for the classification to ...
elgehelge's user avatar
  • 223
36 votes
8 answers
104k views

In Naive Bayes, why bother with Laplace smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
tumultous_rooster's user avatar
5 votes
1 answer
6k views

How to handle unseen features in a Naive Bayes classifier?

I am writing a naive bayes classifier for a text classification problem. I have a bunch of words and an associated label: ...
applecider's user avatar
  • 1,265
1 vote
1 answer
479 views

Question about Good Turing Discounting

I've got a question about Good Turing discounting. I understand the how and the why, but I'm having trouble wrapping my head around it. Say we're discounting the probability of each n-gram that ...
user avatar
12 votes
2 answers
9k views

Laplace smoothing and Dirichlet prior

On the wikipedia article of Laplace smoothing (or additive smoothing), it is said that from a Bayesian point of view, this corresponds to the expected value of the posterior distribution, using a ...
DanielX2010's user avatar
2 votes
1 answer
3k views

In what conditions does naive Bayes classifier perform poorly?

When does naive Bayes perform poorly? Can you think of any specific examples of problems in which it wouldn't work? We can ignore not having seen given data points before as that can be corrected by ...
Remertion's user avatar
2 votes
1 answer
267 views

What is the proper way to estimate the probability (proportion of time) a rare event occurs?

Often, I need to estimate the probability (proportion of time) a rare event occurs. The standard MLE estimate often gives me extreme estimates since the denominator is usually 1, and the numerator is ...
learner's user avatar
  • 21
2 votes
1 answer
659 views

How to approximate 0 in transition probability matrix without loss of generality?

In trying to implement Mixture Markov Model, (see question here), I have extreme cases ( e.g. 0's in the Transition Probability Matrix). I have approached this with replacing 0 with 1e-17. However, I ...
zima's user avatar
  • 769
5 votes
0 answers
2k views

Laplace smoothing parameter choice for Markov chain transitions

Let $Y_{t}$ be the state of the process at time $t$, ${\bf P}$ be the transition matrix then: $$ {\bf P}_{ij} = P(Y_{t} = j | Y_{t-1} = i) $$ Since this is a Markov chain, this probability depends ...
HCAI's user avatar
  • 779
2 votes
1 answer
393 views

Markov chain getting stuck due to insufficient data samples

There is a lot of theory on Markov models and output generation out there, but I cannot locate any information on models getting stuck. I'm trying to create a model of a data set using a Markov model....
rasole's user avatar
  • 33
3 votes
1 answer
335 views

Smoothing a 2-by-2 contingency table

I am trying to implement a system for automatic document categorization, where each document of a corpus belongs to some class. I define the following contingency table for every class C and every ...
DevelBD's user avatar
  • 31
8 votes
2 answers
4k views

What's a good approach to estimate the probability of word frequencies?

I have a document corpus and I want to estimate the probability of occurrence of a certain word $w$. Simply calculating the frequencies and use such a number as an estimation is not a good choice. Is ...
derekhh's user avatar
  • 205
6 votes
2 answers
2k views

What if a numerator term is zero in Naive Bayes?

I'm trying to predict the probability that a user will visit a particular website based on several factors (day of the week, time since last visit, etc). My question is what to do if one of the ...
Jeff's user avatar
  • 3,885