13
$\begingroup$

Quoting from a Wikipedia article on parameter estimation for a naive Bayes classifier: "a typical assumption is that the continuous values associated with each class are distributed according to a Gaussian distribution."

I understand that a Gaussian distribution is convenient for analytical reasons. However, is there any other real-world reason to make this supposition? What if the population consists of two sub-populations (smart/dumb people, large/small apples)?

$\endgroup$
  • 5
    $\begingroup$ Perhaps because of the central limit theorem, Gaussian distributions do fit many, though by no means all, measurements of physical phenomena? With sub-populations, one may get mixture Gaussian distributions. $\endgroup$ – Dilip Sarwate Feb 7 '12 at 14:34
  • 1
    $\begingroup$ The same section (I'm assuming you are looking at the Naive Bayes article) points out that binning is probably a better idea if you don't know the distribution. Someone should probably edit the wikipedia article to make it more clear that one should only assume gaussian if he can argue why it is gaussian (e.g. plot the data, or it follows the additive pattern of the CLT). $\endgroup$ – rm999 Feb 7 '12 at 15:42
  • 1
    $\begingroup$ relevant: mathdl.maa.org/images/upload_library/22/Allendoerfer/… $\endgroup$ – Elvis Feb 7 '12 at 23:33
6
$\begingroup$

At least for me, the assumption of normality arises from two (very powerfull) reasons:

  1. The Central Limit Theorem.

  2. The Gaussian distribution is a maximum entropy (with respect to the continuous version of Shannon's entropy) distribution.

I think you are aware of the first point: if your sample is the sum of many procceses, then as long as some mild conditions are satisfied, the distribution is pretty much gaussian (there are generalizations of the CLT where you in fact don't have to assume that the r.v.s of the sum are identically distributed, see, e.g., the Lyapunov CLT).

The second point is one that for some people (specially physicists) make more sense: given the first and second moments of a distribution, the distribution which less information assumes (i.e. the most conservative) with respect to the continuous Shannon's entropy measure (which is somewhat arbitrary on the continuous case, but, at least for me, totally objective in the discrete case, but that's other story), is the gaussian distribution. This is a form of the so called "maximum entropy principle", which is not so widespread because the actual usage of the form of the entropy is somewhat arbitrary (see this Wikipedia article for more information about this measure).

Of course, this last statement is true also for the multi-variate case, i.e., the maximum entropy distribution (again, with respect to the continuous version of Shannon's entropy) given first ($\vec{\mu}$) and second order information (i.e., the covariance matrix $\mathbf{\Sigma}$), can be shown to be a multivariate gaussian.

PD: I must add to the maximum entropy principle that, according to this paper, if you happen to known the range of variation of your variable, you have to make adjustments to the distribution you get by the maximum entropy principle.

$\endgroup$
3
$\begingroup$

My answer agrees with the first responder. The central limit theorem tells you that if your statistic is a sum or average it will be approximately normal under certain technical conditions regardless of the distribution of the individual samples. But you are right that sometimes people carry this too far just because it seems convenuent. If your statistic is a ratio and the denominator can be zero or close to it the ratio will be too heavytailed for the normal. Gosset found that even when you sample from a normal distribution a normalized average where the sample standard deviation is used for the normalization constant the distribution is the t distribution with n-1 degrees of freedom when n is the sample size. In his field experiments at the Guiness Brewery he has sample sizes that could be in the range of 5-10. In those cases the t distribution is similar to the standard normal distribution in that it is symmetric about 0 but it has much heavier tails. Note that the t distribution does converge to the standard normal as n gets large. In many cases the distribution you have might be bimodal as it is a mixture of two populations. Some times these distributions can be fit as a mixture of normal distributions. But they certain do not look like a normal distribution. If you look at a basic statistics textbook you will find many parametric continuous and discrete distributions that often come up in inference problems. For discrete data we have the binomial, Poisson, geometric, hypergeometric and negative binomial to name a few. Continous examples include the chi square, lognormal, Cauchy, negative exponential, Weibull and Gumbel.

$\endgroup$
2
$\begingroup$

The use of the CLT for justifying the use of the Gaussian distribution is a common fallacy because the CLT is applied to the sample mean, not to individual observations. Therefore, increasing your sample size, does not mean that the sample is closer to normallity.

The Gaussian distribution is commonly used because:

  1. Maximum likelihood estimation is straightforward.
  2. Bayesian inference is simple (using conjugate priors or Jeffreys-type priors).
  3. It is implemented in most of the numerical packages.
  4. There is a lot of theory about this distribution in terms of hypothesis testing.
  5. Lack of knowledge about other options (more flexible). ...

Of course, the best option is to use a distribution that takes into account the characteristics of your context, but this can be challenging. However, is something that people should do

"Everything should be made as simple as possible, but not simpler." (Albert Einstein)

I hope this helps.

Best wishes.

$\endgroup$
  • $\begingroup$ Why the downvote? what counterargument is for this explanation? $\endgroup$ – lmsasu Feb 7 '12 at 15:32
  • 4
    $\begingroup$ The belief that "The use of the CLT for justifying the use of the Gaussian distribution is a common fallacy because the CLT is applied to the sample mean" is itself a fallacy. For example, the electrons in a conductor are moving about at random. The small charge on each electron contributes to a net noise voltage (called thermal noise) that can be measured across the terminals of the conductor. Each contribution is small, there are many electrons, and so via the CLT, the noise is modeled as a Gaussian random process. This model has been cross-validated in numerous experimental studies. $\endgroup$ – Dilip Sarwate Feb 7 '12 at 15:40
  • 1
    $\begingroup$ This first paragraph is confusing and seems off-topic. When applying the CLT we are often saying that a distribution is gaussian because each individual observation is the sum/mean of many processes. If the first paragraph were removed I think this would be good answer. $\endgroup$ – rm999 Feb 7 '12 at 15:51
  • 1
    $\begingroup$ @rm999 "If the first paragraph were removed I think this would be a good answer". Actually, the first paragraph is the crux of the answer since the rest merely points out how the Gaussian model is helpful analytically -- which the OP already understands -- and is not responsive to the question asked. $\endgroup$ – Dilip Sarwate Feb 7 '12 at 16:07
  • $\begingroup$ @Dilip: (+1) The kernel of a very good answer is present in your first comment. Please consider expanding on it in a separate post. $\endgroup$ – cardinal Feb 7 '12 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.