Questions tagged [laplace-smoothing]

Laplace smoothing (also known as additive smoothing) is a technique associated with a probability regularisation task. It ensures that certain improbable outcomes are still associated with a minimum probability of occurrence.

Filter by
Sorted by
Tagged with
0
votes
1answer
24 views

Hoes does laplace smoothing in Naive Bayes control high bias and high variance?

I'm trying to understand how laplace smoothing exactly helps to balance between overfitting and underfitting. I know that Laplace smoothing is used as a fail safe probability if there's a any ...
0
votes
0answers
13 views

Global sensitivity of mean and variance in differential privacy?

Please explain me why global sensitivity of a mean or variance queries will be (b-a)/n and (b-a)^2/n where b is the upper ...
0
votes
0answers
25 views

How to calculate Laplace Smoothing

I have this table The probability of the Gender to be a Female giving the the preferred subject is Math $ p(Female | Math) = \frac{Number of Female Prefer Math}{ Number of Math Occurance} = \frac{0}...
0
votes
0answers
23 views

Should the <s> and <e> be included in the vocabulary while calculating probability of a sentence in a Bigram model with Laplace smoothing?

I am working through an example of Add-1 smoothing in the context of NLP Say that there is the following corpus (start and end tokens included) ...
0
votes
0answers
71 views

Laymen's description of Laplace Smoothing

I have understood that Laplace Smoothing will provide a small non-zero value to the probability score. But still, I am missing something. Please if anyone can provide a better description, it will be ...
11
votes
3answers
137 views

Terminology for Bayesian Posterior Mean of Probability with Uniform Prior

If $p \sim$ Uniform$(0,1)$, and $X \sim$ Bin$(n, p)$, then the posterior mean of $p$ is given by $\frac{X+1}{n+2}$. Is there a common name for this estimator? I've found it solves lots of people's ...
0
votes
1answer
60 views

What is the interpretation for the priors in the derivation of Laplace smoothing?

Laplace smoothing has a generalisation that can be justified with the use of Bayes formula. Let $f(x;\alpha,\beta)$ be the (non-normalised) beta distribution, i.e. $$f(x;\alpha,\beta) = x^{\alpha-1}(...
0
votes
1answer
185 views

R Naive Bayes and Laplace: Even turned off, works fine with unseen words in test data?

I'm trying to better understand Laplace+1 smoothing on Naive Bayes for text classification. Using the e1071 package in R, naiveBayes() function, I get some confusing results. If I fit a model using ...
2
votes
1answer
354 views

What attributes to apply laplace smoothing in naive bayes classifier?

I am reading naive Bayes classifier from the book "Data mining practical machine learning tools and techniques". The example of naive Bayes is given using the below dataset. As (Outlook=Overcast | ...
4
votes
1answer
378 views

Smoothing/shrinking the predicted probability of a classifier to reduce live logloss

Let us assume we work on a 2 -class classification problem. In my setting the sample is balanced. To be precise it is a financial markets setting where up and down have approximately 50:50 chance. The ...
1
vote
1answer
1k views

Why do we need Laplace smoothing in Naive Bayes while logarithm may resolve the problem?

In Naive Bayes algorithm, we use $$P(c)P(x_1|c)P(x_2|c)...p(x_n|c)\space\space (*)$$ to decide about the class of a sample $\textbf{x} =(x_1,...,x_n)$. It is possible that for a class $c$, a feature $...
2
votes
1answer
176 views

How to do multinomial regression when baseline choice percentage of one choice item is 0%

I am trying to find a clever way to analyze the choice ratios between a baseline condition and a different condition. Specifically, I have a baseline condition, where the choice ratio between two ...
1
vote
1answer
5k views

Laplace smoothing and naive bayes

If I want to use naive bayes with laplace smoothing and therefore add 1 to probabilities with the value of 0, what does this mean for probabilities which have the actual value of 1?
0
votes
1answer
2k views

Naive Bayes with Laplace Smoothing Probabilities Not Adding Up

Let c refer to a class (such as Positive or Negative), and let w refer to a token or word. Define $count(w,c) = $ $counts \ w \ in\ class \ c$ $count(c) = counts \ of \ words \ in \ class \...
2
votes
2answers
2k views

Laplace smoothing understanding implementation

Considering the data set given below Here if we have to classify new data point: D15 (O=Overcast, T=Cool, H=High, W=Strong) Then for P(No|Overcast, Cool, High, Strong) we have, (5/14) * 0 * (1/5) *...
1
vote
1answer
1k views

A simple numerical example for Kneser-Ney Smoothing

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...
3
votes
1answer
2k views

Understanding Add-1/Laplace smoothing with bigrams

I am working through an example of Add-1 smoothing in the context of NLP Say that there is the following corpus (start and end tokens included) ...
2
votes
1answer
71 views

bayesian classification unknown domain

Suppose I am building a naive Bayes model to classify text messages as either spam or legit. I am training my model using a dataset containing both classes and for which I know the domain (the number ...
0
votes
1answer
714 views

Regarding probabilites for naiveBayes algo

I have trained my data with naiveBayes algo in e1071 package. I have 6 classes in my data. I have predicted test data. the prediction returns only one class for each data point but I would like to ...
7
votes
2answers
6k views

Calculating Emission Probability values for Hidden Markov Model (HMM)

I'm new to HMM and still learning. I'm currently using HMM to tag part-of-speech. To implement the viterbi algorithm I need transition probabilities ($ a_{i,j} \newcommand{\Count}{\text{Count}}$) and ...
0
votes
2answers
841 views

Kneser-Ney for unigrams?

I was wondering if it is at all possible to use Kneser-Ney to smooth word unigram probabilites? The basic idea behind back-off is to use (n-1)-gram frequencies when an n-gram has 0 count. This is ...
3
votes
1answer
2k views

Is the Laplace/Lidstone smoothing parameter (talking about Multinomial/Bernoulli Naive Bayes) related to the particular structure of the dataset?

I'm working with Multinomial and Bernoulli Naive Bayes implementation of scikit-learn (python) for text classification. I'm using the 20_newsgroups dataset. From the scikit documentation we have: <...
1
vote
1answer
300 views

Alternative Smoothing techniques for Naive Bayes?

I've always wondered why no one using smoothing techniques for Naive Bayes which implicitly test which conditional probabilities should actually be used and which ones shouldn't. The formula for ...
1
vote
1answer
330 views

Choosing smoothing parameters across multiple Naïve Bayes classifiers with different number of categories

I would like to train multiple Naïve Bayes classifiers with different number of categories, and also have a global threshold for how certain one classifier must be in order for the classification to ...
26
votes
7answers
64k views

In Naive Bayes, why bother with Laplace smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
3
votes
1answer
3k views

How to handle unseen features in a Naive Bayes classifier?

I am writing a naive bayes classifier for a text classification problem. I have a bunch of words and an associated label: ...
1
vote
1answer
218 views

Question about Good Turing Discounting

I've got a question about Good Turing discounting. I understand the how and the why, but I'm having trouble wrapping my head around it. Say we're discounting the probability of each n-gram that ...
11
votes
2answers
5k views

Laplace smoothing and Dirichlet prior

On the wikipedia article of Laplace smoothing (or additive smoothing), it is said that from a Bayesian point of view, this corresponds to the expected value of the posterior distribution, using a ...
1
vote
1answer
2k views

In what conditions does naive Bayes classifier perform poorly?

When does naive Bayes perform poorly? Can you think of any specific examples of problems in which it wouldn't work? We can ignore not having seen given data points before as that can be corrected by ...
2
votes
1answer
487 views

How to approximate 0 in transition probability matrix without loss of generality?

In trying to implement Mixture Markov Model, (see question here), I have extreme cases ( e.g. 0's in the Transition Probability Matrix). I have approached this with replacing 0 with 1e-17. However, I ...
5
votes
0answers
1k views

Laplace smoothing parameter choice for Markov chain transitions

Let $Y_{t}$ be the state of the process at time $t$, ${\bf P}$ be the transition matrix then: $$ {\bf P}_{ij} = P(Y_{t} = j | Y_{t-1} = i) $$ Since this is a Markov chain, this probability depends ...
2
votes
1answer
209 views

Markov chain getting stuck due to insufficient data samples

There is a lot of theory on Markov models and output generation out there, but I cannot locate any information on models getting stuck. I'm trying to create a model of a data set using a Markov model....
3
votes
1answer
250 views

Smoothing a 2-by-2 contingency table

I am trying to implement a system for automatic document categorization, where each document of a corpus belongs to some class. I define the following contingency table for every class C and every ...
7
votes
2answers
3k views

What's a good approach to estimate the probability of word frequencies?

I have a document corpus and I want to estimate the probability of occurrence of a certain word $w$. Simply calculating the frequencies and use such a number as an estimation is not a good choice. Is ...
6
votes
2answers
1k views

What if a numerator term is zero in Naive Bayes?

I'm trying to predict the probability that a user will visit a particular website based on several factors (day of the week, time since last visit, etc). My question is what to do if one of the ...