Every source in the literature I could find about naive Bayes mentions using a gaussian's probability density function, using the mean and variance estimated from the data itself.

This strikes me as odd. If one estimates the variance and mean of the data (as opposed to has a prior assumption about it), than the distribution should be student t, not gaussian.

I recognize that the nuance may be unimportant in some cases, but still. This surprises me as it seems like using gaussian pdf is just a sort of mistake, and using the "correct" distribution has no cost involved. What am I missing here?

  • 2
    $\begingroup$ Are you referring to the prior or the posterior, and are you referring to a particular empirical Bayes method? These make a lot of difference (citing a specific example would help). In any case, assuming a normal distribution with a normal-inverse gamma prior gets you to a t-distribution posterior, see here. $\endgroup$ – David Robinson Feb 17 '16 at 20:31
  • $\begingroup$ @DavidRobinson He means the distributions of the predictors in a classification problem. Naive Bayes is really just a density estimation technique and has little to do with Bayesian statistics. $\endgroup$ – dsaxton Feb 18 '16 at 5:11
  • $\begingroup$ ^ Really? You have a prior belief about the conditional distributions, that features follow some distribution and you update them after seeing empirical evidence. Can you elaborate on Naive Bayes not being a Bayesian approach? $\endgroup$ – user46925 Feb 20 '16 at 1:32

I think the confusion here is that the $t$ distribution is used as the sampling distribution for the test statistic $\sqrt{n} (\bar{x} - \mu_0) / s$, and not as a model for the underlying distribution. In the case of naive Bayes we're talking about the distributions of the features themselves, not the distribution of a statistic. The fact that the mean and variance are estimated from the data doesn't really enter into our choice of distribution for a given feature. If anything, the Gaussian should be preferred over the $t$ if for no reason other than the fact that it has a simpler density function. Still, the Gaussian assumption is almost always very wrong.

  • $\begingroup$ Dsaxton, I almost got it, but not quite. I would expect the pdf of the distribution for each sample will be in of itself a t distribution density function, since sigma and mu were estimated. Your answer seems to imply that the t distribution is only well defined or valid for hypothesis tests. Could you clarify why you think that is the case? $\endgroup$ – Uri Merhav Feb 18 '16 at 19:26
  • $\begingroup$ Perhaps it would help to ask why you'd expect predictors in a classification problem to have $t$ distributions. Whether or not you're estimating means and variances from data has nothing to do with whether or not a predictor belongs to some family of distributions. It either does or doesn't. However, dividing by $s$ instead of $\sigma$ does affect the distribution of the $t$ statistic. $\endgroup$ – dsaxton Feb 18 '16 at 19:36
  • $\begingroup$ dsxaton: The point is that if the measurements are distirbuted in a gaussian around a certain $\mu$ and $\sigma$, since you're using the estimated gaussian parameters to estimate probability, you're using the gaussian pdf in a "wrong" way. Specifically, you will greatly underestimate the probability of outliers (since for example $\sigma$ estimate may be lower than it should be). In any case, I now realize that student t is only defined for the test statistics and hence useless. Still it seems like there's a self contradiction in the way naive bayes is used. $\endgroup$ – Uri Merhav Feb 18 '16 at 22:12
  • $\begingroup$ If you assume something is normal, it doesn't make sense to decide it's $t$ just because you don't know the parameters. That would be a contradiction. Also, you can just as easily overestimate $\sigma$ as underestimate it. In any case, the Gaussian assumption is just a simplifying convention used in naive Bayes and it would partially defeat the purpose to use a complicated density like that of the $t$. $\endgroup$ – dsaxton Feb 18 '16 at 23:52
  • $\begingroup$ of course $\sigma$ may be either overestimated or underestimated. The point is that it as it may be underestimated, the likelihood of something being "5 $\sigma$ away is actually much higher than what's implied by a normal distribution. We're choosing a pdf which will certainly significantly underestimate the chance of extreme outliers even if all the model's assumptions were completely true. Thanks to your feedback I'm not sure that I have a reasonable alternative to this, but I still find this self contradiction to be odd. $\endgroup$ – Uri Merhav Feb 19 '16 at 1:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.