Questions tagged [laplace-smoothing]

Laplace smoothing (also known as additive smoothing) is a technique associated with a probability regularisation task. It ensures that certain improbable outcomes are still associated with a minimum probability of occurrence.

Filter by
Sorted by
Tagged with
0
votes
1answer
50 views

Hoes does laplace smoothing in Naive Bayes control high bias and high variance?

I'm trying to understand how laplace smoothing exactly helps to balance between overfitting and underfitting. I know that Laplace smoothing is used as a fail safe probability if there's a any ...
0
votes
1answer
25 views

Global sensitivity of mean and variance in differential privacy?

Please explain me why global sensitivity of a mean or variance queries will be (b-a)/n and (b-a)^2/n where b is the upper ...
0
votes
0answers
33 views

How to calculate Laplace Smoothing

I have this table The probability of the Gender to be a Female giving the the preferred subject is Math $ p(Female | Math) = \frac{Number of Female Prefer Math}{ Number of Math Occurance} = \frac{0}...
0
votes
0answers
25 views

Should the <s> and <e> be included in the vocabulary while calculating probability of a sentence in a Bigram model with Laplace smoothing?

I am working through an example of Add-1 smoothing in the context of NLP Say that there is the following corpus (start and end tokens included) ...
0
votes
0answers
72 views

Laymen's description of Laplace Smoothing

I have understood that Laplace Smoothing will provide a small non-zero value to the probability score. But still, I am missing something. Please if anyone can provide a better description, it will be ...
11
votes
3answers
147 views

Terminology for Bayesian Posterior Mean of Probability with Uniform Prior

If $p \sim$ Uniform$(0,1)$, and $X \sim$ Bin$(n, p)$, then the posterior mean of $p$ is given by $\frac{X+1}{n+2}$. Is there a common name for this estimator? I've found it solves lots of people's ...
0
votes
1answer
72 views

What is the interpretation for the priors in the derivation of Laplace smoothing?

Laplace smoothing has a generalisation that can be justified with the use of Bayes formula. Let $f(x;\alpha,\beta)$ be the (non-normalised) beta distribution, i.e. $$f(x;\alpha,\beta) = x^{\alpha-1}(...
0
votes
1answer
219 views

R Naive Bayes and Laplace: Even turned off, works fine with unseen words in test data?

I'm trying to better understand Laplace+1 smoothing on Naive Bayes for text classification. Using the e1071 package in R, naiveBayes() function, I get some confusing results. If I fit a model using ...
2
votes
1answer
407 views

What attributes to apply laplace smoothing in naive bayes classifier?

I am reading naive Bayes classifier from the book "Data mining practical machine learning tools and techniques". The example of naive Bayes is given using the below dataset. As (Outlook=Overcast | ...
4
votes
1answer
386 views

Smoothing/shrinking the predicted probability of a classifier to reduce live logloss

Let us assume we work on a 2 -class classification problem. In my setting the sample is balanced. To be precise it is a financial markets setting where up and down have approximately 50:50 chance. The ...
2
votes
1answer
1k views

Why do we need Laplace smoothing in Naive Bayes while logarithm may resolve the problem?

In Naive Bayes algorithm, we use $$P(c)P(x_1|c)P(x_2|c)...p(x_n|c)\space\space (*)$$ to decide about the class of a sample $\textbf{x} =(x_1,...,x_n)$. It is possible that for a class $c$, a feature $...
2
votes
1answer
179 views

How to do multinomial regression when baseline choice percentage of one choice item is 0%

I am trying to find a clever way to analyze the choice ratios between a baseline condition and a different condition. Specifically, I have a baseline condition, where the choice ratio between two ...
1
vote
1answer
5k views

Laplace smoothing and naive bayes

If I want to use naive bayes with laplace smoothing and therefore add 1 to probabilities with the value of 0, what does this mean for probabilities which have the actual value of 1?
0
votes
1answer
3k views

Naive Bayes with Laplace Smoothing Probabilities Not Adding Up

Let c refer to a class (such as Positive or Negative), and let w refer to a token or word. Define $count(w,c) = $ $counts \ w \ in\ class \ c$ $count(c) = counts \ of \ words \ in \ class \...
2
votes
2answers
2k views

Laplace smoothing understanding implementation

Considering the data set given below Here if we have to classify new data point: D15 (O=Overcast, T=Cool, H=High, W=Strong) Then for P(No|Overcast, Cool, High, Strong) we have, (5/14) * 0 * (1/5) *...
1
vote
1answer
1k views

A simple numerical example for Kneser-Ney Smoothing

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...
3
votes
1answer
3k views

Understanding Add-1/Laplace smoothing with bigrams

I am working through an example of Add-1 smoothing in the context of NLP Say that there is the following corpus (start and end tokens included) ...
2
votes
1answer
72 views

bayesian classification unknown domain

Suppose I am building a naive Bayes model to classify text messages as either spam or legit. I am training my model using a dataset containing both classes and for which I know the domain (the number ...
0
votes
1answer
734 views

Regarding probabilites for naiveBayes algo

I have trained my data with naiveBayes algo in e1071 package. I have 6 classes in my data. I have predicted test data. the prediction returns only one class for each data point but I would like to ...
7
votes
2answers
6k views

Calculating Emission Probability values for Hidden Markov Model (HMM)

I'm new to HMM and still learning. I'm currently using HMM to tag part-of-speech. To implement the viterbi algorithm I need transition probabilities ($ a_{i,j} \newcommand{\Count}{\text{Count}}$) and ...
0
votes
2answers
884 views

Kneser-Ney for unigrams?

I was wondering if it is at all possible to use Kneser-Ney to smooth word unigram probabilites? The basic idea behind back-off is to use (n-1)-gram frequencies when an n-gram has 0 count. This is ...
3
votes
1answer
2k views

Is the Laplace/Lidstone smoothing parameter (talking about Multinomial/Bernoulli Naive Bayes) related to the particular structure of the dataset?

I'm working with Multinomial and Bernoulli Naive Bayes implementation of scikit-learn (python) for text classification. I'm using the 20_newsgroups dataset. From the scikit documentation we have: <...
1
vote
1answer
315 views

Alternative Smoothing techniques for Naive Bayes?

I've always wondered why no one using smoothing techniques for Naive Bayes which implicitly test which conditional probabilities should actually be used and which ones shouldn't. The formula for ...
1
vote
1answer
360 views

Choosing smoothing parameters across multiple Naïve Bayes classifiers with different number of categories

I would like to train multiple Naïve Bayes classifiers with different number of categories, and also have a global threshold for how certain one classifier must be in order for the classification to ...
27
votes
7answers
67k views

In Naive Bayes, why bother with Laplace smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
3
votes
1answer
3k views

How to handle unseen features in a Naive Bayes classifier?

I am writing a naive bayes classifier for a text classification problem. I have a bunch of words and an associated label: ...
1
vote
1answer
225 views

Question about Good Turing Discounting

I've got a question about Good Turing discounting. I understand the how and the why, but I'm having trouble wrapping my head around it. Say we're discounting the probability of each n-gram that ...
11
votes
2answers
5k views

Laplace smoothing and Dirichlet prior

On the wikipedia article of Laplace smoothing (or additive smoothing), it is said that from a Bayesian point of view, this corresponds to the expected value of the posterior distribution, using a ...
1
vote
1answer
2k views

In what conditions does naive Bayes classifier perform poorly?

When does naive Bayes perform poorly? Can you think of any specific examples of problems in which it wouldn't work? We can ignore not having seen given data points before as that can be corrected by ...
2
votes
1answer
86 views

What is the proper way to estimate the probability (proportion of time) a rare event occurs?

Often, I need to estimate the probability (proportion of time) a rare event occurs. The standard MLE estimate often gives me extreme estimates since the denominator is usually 1, and the numerator is ...
2
votes
1answer
490 views

How to approximate 0 in transition probability matrix without loss of generality?

In trying to implement Mixture Markov Model, (see question here), I have extreme cases ( e.g. 0's in the Transition Probability Matrix). I have approached this with replacing 0 with 1e-17. However, I ...
5
votes
0answers
1k views

Laplace smoothing parameter choice for Markov chain transitions

Let $Y_{t}$ be the state of the process at time $t$, ${\bf P}$ be the transition matrix then: $$ {\bf P}_{ij} = P(Y_{t} = j | Y_{t-1} = i) $$ Since this is a Markov chain, this probability depends ...
2
votes
1answer
211 views

Markov chain getting stuck due to insufficient data samples

There is a lot of theory on Markov models and output generation out there, but I cannot locate any information on models getting stuck. I'm trying to create a model of a data set using a Markov model....
3
votes
1answer
255 views

Smoothing a 2-by-2 contingency table

I am trying to implement a system for automatic document categorization, where each document of a corpus belongs to some class. I define the following contingency table for every class C and every ...
7
votes
2answers
3k views

What's a good approach to estimate the probability of word frequencies?

I have a document corpus and I want to estimate the probability of occurrence of a certain word $w$. Simply calculating the frequencies and use such a number as an estimation is not a good choice. Is ...
6
votes
2answers
1k views

What if a numerator term is zero in Naive Bayes?

I'm trying to predict the probability that a user will visit a particular website based on several factors (day of the week, time since last visit, etc). My question is what to do if one of the ...