Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
For example, we always assumed that the data or signal error is a Gaussian distribution? why?
I have asked this question on stackoverflow, the link: https://stackoverflow.com/questions/12616406/anyone-can-tell-me-why-we-always-use-the-gaussian-distribution-in-machine-learni
I looked at the answers on SO. I don't think they are satisfactory. People often argue for the normal distribution because of the central limit theorem. That may be okay in large samples when the problem involves averages. But machine learning problems can be more complex and sample sizes are not always large enough for normal approximations to apply. Some argue for mathematical convenience. That is no justification especially when computers can easily handle added complexity and computer-intensive resampling approaches.
But I think the question should be challenged. Who says the Guassian distribution is "always" used or even just predominantly used in machine learning. Taleb claimed that statistics is dominated by the Gaussian distribution especially when applied to finance. He was very wrong about that!
In machine learning aren't kernel density classification approaches, tree classifiers and other nonparametric methods sometimes used? Aren't nearest neighbor methods used for clustering and classification? I think they are and I know statisticians use these methods very frequently.
Machine learning (and statistics as well) treats data as the mix of deterministic (causal) and random parts. The random part of data usually has normal distribution. (Really, the causal relation is reverse: the distribution of random part of variable is called normal). Central limit theorem says that the sum of large number of varibles each having a small influence on the result approximate normal distribution. 1. Why data is treated as normally distributed? In machine learning we want to express dependent variable as some function of a number of independent variables. If this function is sum (or expressed as a sum of some other funstions) and we are suggesting that the number of independent variables is really high, then the dependent variable should have normal distribution (due to central limit theorem). 2. Why errors are looked to be normally distributed? The dependent variable ($Y$) consists of deterministic and random parts. In machine learning we are trying to express deterministic part as a sum of deterministic independent variables: $$deterministic + random = func(deterministic(1))+...+func(deterministic(n))+model\_error$$ If the whole deterministic part of $Y$ is explained by $X$ then the $model\_error$ depicts only $random$ part, and thus should have normal distribution. So if error distribution is normal, then we may suggest that the model is successful. Else there are some other features that are absent in model, but have large enough influence on $Y$ (the model is incomplete) or the model is incorrect.
One reason that normal distributions are often (but not always!) assumed: the nature of the distribution often leads to extremely efficient computation. For example, in generalized linear regression, the solution is technically in closed form when your distribution is Gaussian:
$\hat \beta = (X^T X)^{-1} X^T Y$
where as for other distributions, iterative algorithms must be used. Technical note: using this direct computation to find $\hat \beta$ is both inefficient and unstable.
Quite often, both the theoretical math and numerical methods required are substaintially easier if the distribution is a linear transformation of normal variables. Because of this, methods are frequently first developed under the assumption that the data is normal, as the problem is considerably more tractable. Later, the more difficult problem of addressing non-normality is addressed by statistical/machine learning researchers.
I had the same question "what the is advantage of doing a Gaussian transformation on predictors or target?" Infact, caret package has a pre-processing step that enables this transformation.
I tried reasoning this out and am summarizing my understanding -
Usually the data distribution in Nature follows a Normal distribution ( few examples like - age, income, height, weight etc., ) . So its the best approximation when we are not aware of the underlying distribution pattern.
Most often the goal in ML/ AI is to strive to make the data linearly separable even if it means projecting the data into higher dimensional space so as to find a fitting "hyperplane" (for example - SVM kernels, Neural net layers, Softmax etc.,). The reason for this being "Linear boundaries always help in reducing variance and is the most simplistic, natural and interpret-able" besides reducing mathematical / computational complexities. And, when we aim for linear separability, its always good to reduce the effect of outliers, influencing points and leverage points. Why? Because the hyperplane is very sensitive to the influencing points and leverage points (aka outliers) - To undertstand this - Lets shift to a 2D space where we have one predictor (X) and one target(y) and assume there exists a good positive correlation between X and y. Given this, if our X is normally distributed and y is also normally distributed, you are most likely to fit a straight line that has many points centered in the middle of the line rather than the end-points (aka outliers, leverage / influencing points). So the predicted regression line will most likely suffer little variance when predicting on unseen data.
Extrapolating the above understanding to a n-dimensional space and fitting a hyperplane to make things linearly separable does infact really makes sense because it helps in reducing the variance.
i'm currently studying machine learning and the same question popped to my mind. What I think the reason should be is that in every machine learning problem we assume we have abundant observational data available and whenever data tends to infinity it gets normally distributed around its mean and thats what Normal distribution(Gaussian Distribution) says. Although its not necessary that Gaussian DIstribution will always be a perfect fit to any data that tends to infinity like take a case when your data is always positive then you can easily see if you try to fit gaussian dist to it it will give some weight to negative values of x also(although negligible but still some weight is given) so in such case distribution like Zeta is more suited.
A commonly used method in ML classification is Bernoulli Naive Bayes. So I dispute the claim that ML always uses the Normal distribution.
