4
$\begingroup$

I'm currently implementing a Gaussian Naive Bayes classifier. With a Naive Bayes classifier the zero observation or zero probability problem can occur, see e.g. point 11 on http://machinelearningmastery.com/better-naive-bayes/. This can be solved by e.g. laplacian smoothing.

Am I right that this problem cannot happen for Gaussian Naive Bayes? Because there we use the normal distribution for calculating the probabilities (using the mean and standard deviation calculated for each feature).

$\endgroup$
2
$\begingroup$

Generally, the zero probability problem is about the fact that you lack a certain kind of cases in your data that are actually possible. In this case assigning probability of zero to such cases does not seem appropriate, so you assign some low probabilities to such cases (you can find some examples in here).

Now, how does this relate to continuous variables? Recall that in continuous case $P(X = x) = 0$, so correcting for missing point-values does not make sens. In this case, "zero probability" problem would be that (a) you know that your data is censored, truncated, has outliers etc., and you want to make corrections in your model for that fact, or (b) you, in general, want to include out-of-sample knowledge (prior) in your model (a fully Bayesian model).

So, with discrete values using Laplacian correction can be easily applied, but it does not have a simple analogue with continuous data. With continuous data it is rather a matter of understanding your data and building a valid statistical model of it. Laplacian smoothing is in fact using a uniform prior and for continuous data you also can use some prior distribution to include out-of-sample knowledge in your model.

$\endgroup$
  • $\begingroup$ Well, what happens if for given class the variance is exactly zero? Isn't this the analogy of zero observations problem in gaussian naive bayes? $\endgroup$ – hans Feb 3 at 22:17
  • $\begingroup$ @hans if variance is zero, then it is constant, so there is a problem with your data. Moreover, it is not continuous, because such case would be impossible. Use discrete distribution and Laplace correction. $\endgroup$ – Tim Feb 3 at 22:21
  • $\begingroup$ good point! I was think of such an example: you have 20 classes and there is one class that is very rare and has only two samples. One feature just happens to have the same value for both samples in this class. Generally this feature is continuous in the physical sens, but the accuracy of measurement is limited, so it possible. It can be for example height measured with 1 mm accuracy. $\endgroup$ – hans Feb 3 at 22:53
  • $\begingroup$ @hans then you don't have enough data for such class. Nonetheless, as in Laplace smoothing, you could use Bayesian approach with some prior. Estate for the class with two samples would be simply dominated by the prior. $\endgroup$ – Tim Feb 4 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.