I had an exam recently where we had to train a Naive Bayes model on a given set.

The dataset had a column which would give 0 probability for a YES value. I was told later that we were supposed to either do additive smoothing, or drop the columns which gets you 0 probabilities or make the zero probability, a very small number by adding a small value to a zero probability to make it non-zero.

I have a few concerns:

1) Isn't additive smoothing or changing probability, data tampering? Aren't we introducing our own bias in the data set by doing those methods? Like we are changing the data. And wouldn't this affect the accuracy of the model sometimes, when the some attributes in the dataset are for sure, deterministic with a 1/0 probability for the class attribute?

2) Suppose I was to make a classification/decision tree instead of the same dataset. There I would be allowed to always make decision based on the values of those attributes, right? Doesn't this imply a huge decisional-gap between Naive Bayes classification and Decision Trees?

3) Finally, what is the logic and reason behind these methods to deal with 1/0 probability attributes?

  • $\begingroup$ I don't fully understand this, but shouldn't that be 'adding a small value to a zero probability'? Multiplying by zero wouldn't change anything unless I'm missing some part of the explanation. $\endgroup$ – mkt - Reinstate Monica Apr 16 '18 at 5:42
  • 1
    $\begingroup$ @mkt you are right. sorry about that. edited the post. $\endgroup$ – hsnsd Apr 16 '18 at 5:46

In statistics, all is uncertain, also the class membership of each pattern, as assigned by a (naive bayes) classifier. Probabilities that are purely 0 and 1, that indicates a deterministic mechanism, for which statistics was not developed. As soon as uncertainty comes into play, determinism is replaced by statistics and probability theory.

In general, 0/1 probabilities occur in limited-size datasets, or when the number of possible events becomes very large, compared to the number of variables. When for example using the present-day level of a stock-index as predictive variable $Si$, discretizing this real-number variable into 1000 bins is likely to yield some bins of which the pertaining stock-index value has never occcurred. Hence, a zero probability (It is not recommended to discretize a continuously distributed variable into a large number of bins, but this example serves the explanation given here).

Smoothing of discrete probability tables with 0-probabilities has been approached by Bayesian methods. The Dirichlet distribution mixes a set of observations with an expected, prior distribution. This prior distribution is also called a regularization prior. It recommended to review the theory of the Dirichlet distribution. It makes possible a seamless mixing of the observations in a dataset with an (expected) prior distribution over the outcomes. Within the Bayesian paradigm, this prior distribution is given by expert knowledge by someone who is into the domain for which the classifier is being built. Bayesian probabilities are also called subjective probabilities.

Using a frequentist approach, it is also possible to use a uniform prior distribution as regularization prior. Basically you can add $n$ counts to each probability outcome, before computing the probability distribution over the bins. If $n=1$, you add solely one observation to each outcome - and the unifom prior has the minimal influence on the outcome of your Naive Bayes classifier.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.