# Naive Bayes - Additive Smoothing (or altering/dropping columns) when you get 0 probability

I had an exam recently where we had to train a Naive Bayes model on a given set.

The dataset had a column which would give 0 probability for a YES value. I was told later that we were supposed to either do additive smoothing, or drop the columns which gets you 0 probabilities or make the zero probability, a very small number by adding a small value to a zero probability to make it non-zero.

I have a few concerns:

1) Isn't additive smoothing or changing probability, data tampering? Aren't we introducing our own bias in the data set by doing those methods? Like we are changing the data. And wouldn't this affect the accuracy of the model sometimes, when the some attributes in the dataset are for sure, deterministic with a 1/0 probability for the class attribute?

2) Suppose I was to make a classification/decision tree instead of the same dataset. There I would be allowed to always make decision based on the values of those attributes, right? Doesn't this imply a huge decisional-gap between Naive Bayes classification and Decision Trees?

3) Finally, what is the logic and reason behind these methods to deal with 1/0 probability attributes?

• I don't fully understand this, but shouldn't that be 'adding a small value to a zero probability'? Multiplying by zero wouldn't change anything unless I'm missing some part of the explanation. – mkt - Reinstate Monica Apr 16 '18 at 5:42
• @mkt you are right. sorry about that. edited the post. – hsnsd Apr 16 '18 at 5:46

In general, 0/1 probabilities occur in limited-size datasets, or when the number of possible events becomes very large, compared to the number of variables. When for example using the present-day level of a stock-index as predictive variable $Si$, discretizing this real-number variable into 1000 bins is likely to yield some bins of which the pertaining stock-index value has never occcurred. Hence, a zero probability (It is not recommended to discretize a continuously distributed variable into a large number of bins, but this example serves the explanation given here).
Using a frequentist approach, it is also possible to use a uniform prior distribution as regularization prior. Basically you can add $n$ counts to each probability outcome, before computing the probability distribution over the bins. If $n=1$, you add solely one observation to each outcome - and the unifom prior has the minimal influence on the outcome of your Naive Bayes classifier.