# Can anyone tell me why we always use the Gaussian distribution in Machine learning?

For example, we always assumed that the data or signal error is a Gaussian distribution? why?

I have asked this question on stackoverflow, the link: http://stackoverflow.com/questions/12616406/anyone-can-tell-me-why-we-always-use-the-gaussian-distribution-in-machine-learni

-

I looked at the answers on SO. I don't think they are satisfactory. People often argue for the normal distribution because of the central limit theorem. That may be okay in large samples when the problem involves averages. But machine learning problems can be more complex and sample sizes are no always large enough for normal approximations to apply. Some argue for mathematical convenience. That is no justification especially when computers can easily handle added complexity and computer-intensive resampling approaches.

But I think the question should be challenged. Who says the Guassian distribution is "always" used or even just predominantly used in machine learning. Taleb claimed that statistics is dominated by the Gaussian distribution especially when applied to finance. He was very wrong about that!

In machine learning aren't kernel density classification, tree classifiers and other nonparametric methods sometimes used? Aren't nearest neighbor methods used for clustering and classification? I think they are and I know statisticians use these methods very frequently.

-
Also, not all variables/distributions are averages. –  Douglas Zare Sep 29 '12 at 7:23
Yes @DouglasZare and I think machine learning analysts are smart enough to know that they should not try to fit a square peg into a round hole. –  Michael Chernick Sep 29 '12 at 11:58

Machine learning (and statistics as well) treats data as the mix of deterministic (causal) and random parts. The random part of data usually has normal distribution. (Really, the causal relation is reverse: the distribution of random part of variable is called normal). Central limit theorem says that the sum of large number of varibles each having a small influence on the result approximate normal distribution. 1. Why data is treated as normally distributed? In machine learning we want to express dependent variable as some function of a number of independent variables. If this function is sum (or expressed as a sum of some other funstions) and we are suggesting that the number of independent variables is really high, then the dependent variable should have normal distribution (due to central limit theorem). 2. Why errors are looked to be normally distributed? The dependent variable ($Y$) consists of deterministic and random parts. In machine learning we are trying to express deterministic part as a sum of deterministic independent variables: $$deterministic + random = func(deterministic(1))+...+func(deterministic(n))+model\_error$$ If the whole deterministic part of $Y$ is explained by $X$ then the $model\_error$ depicts only $random$ part, and thus should have normal distribution. So if error distribution is normal, then we may suggest that the model is successful. Else there are some other features that are absent in model, but have large enough influence on $Y$ (the model is incomplete) or the model is incorrect.

-
This may explain why the normal distribution is often used in problems but as Douglas Zare commented nto all machine learning problems center around random components that are normal. So there are problems where other parametric distributions should be used or nonparametric methods should be applied. The question says "always". I doubt that machine learning analysts always use the Gaussian distribution. –  Michael Chernick Sep 29 '12 at 12:11
You are completely right and your answer has clearly explained why not "always". In my answer I tried to add why we assume normal distribution and in what cases. –  O_Devinyak Sep 29 '12 at 13:03