Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
When I learned linear regression in my statistics class, we are asked to check for a few assumptions which need to be true for linear regression to make sense. I won't delve deep into those assumptions, however, these assumptions don't appear when learning linear regression from machine learning perspective.
Is it because the data is so large that those assumptions are automatically taken care of? Or is it because of the loss function (i.e. gradient descent)?
It’s because statistics puts an emphasis on model inference, while machine learning puts an emphasis on accurate predictions.
We like normal residuals in linear regression because then the usual $\hat{\beta}=(X^TX)^{-1}X^Ty$ is a maximum likelihood estimator.
We like uncorrelated predictors because then we get tighter confidence intervals on the parameters than we would if the predictors were correlated.
In machine learning, we often don’t care about how we get the answer, just that the result has a tight fit both in and out of sample.
Leo Breiman has a famous article on the “two cultures” of modeling: https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726
Breiman, Leo. "Statistical modeling: The two cultures (with comments and a rejoinder by the author)." Statistical science 16.3 (2001): 199-231.
The typical linear regression assumptions are required mostly to make sure your inferences are right.
For instance, suppose you want to check if a certain predictor is associated with your target variable. In a linear regression setting, you would calculate the p-value associated to the coefficient of that predictor. In order to get this p-value correct, you need to satisfy all the assumptions.
In ML, on the other hand, you only want a model that can fit and generalize the patterns in your data: it's all about prediction, not inference. One would mostly care about how well the linear regression generalizes to unseen data, and this can be checked by assessing MSE on train-test splitted data or by cross validation, no need for parametric assumptions.
Of course this is not as black and white as I put it, for instance, one can use parametric assumptions to derive error estimates for predictions on new data. This can still be interesting in a ML setting. Still, you are correct in noticing that these assumptions are, in general, very important from a Stats point of view and not such a big deal in ML and that's the reason: the focus on inference vs. the focus on prediction.
A linear regression is a statistical procedure that can be interpreted from both perspectives. Instead I will tackle the question of comparing linear regression (and its assumptions) to other methods.
A linear regression takes the form $$ Y_i = X_i'\beta + \varepsilon_i$$ Texbooks usually ask you to check (i) Exogeneity $\mathbb{E}[\varepsilon_i \mid X_i] = 0$, (ii) Non-colinearity: $\mathbb{E}[X_iX_i']$ is invertible and (iii) homoskedasticity, $\mathbb{E}[\varepsilon_i \mid X_i] = \sigma^2$. Only (i) and (ii) are considered identifying assumptions, and (iii) can be replaced by much weaker assumptions. Normality of residuals sometimes appears in introductory texts, but has been shown to be unnecessary to understand the large sample behavior. Why do we need it? $$ \widehat{\beta} = \beta + {\underbrace{\left(\frac{X'X}{n}\right)}_{\to^p \mathbb{E}[X_iX_i']}}^{-1} \ \underbrace{\left(\frac{X'\varepsilon_i}{n}\right)}_{\to^p \mathbb{E}[X_i\varepsilon_i']}$$ Condition (i) makes the second term zero, (ii) makes sure that the matrix is invertible, (iii) or some version of it guarantees the validity of the weak law of large numbers. Similar ideas are used to compute standard errors. The estimated prediction is $X_i'\widehat{\beta}$ which converges to $X_i'\beta$.
A typical machine learning (ML) algorithm attempts a more complicated functional form $$ Y_i = g(X_i) + \varepsilon_i $$ The ``regression'' function is defined as $g(x) = \mathbb{E}[Y_i \mid X_i = x]$. By construction $$\mathbb{E}[\varepsilon_i \mid X_i] = \mathbb{E}[Y_i - g(X_i) \mid X_i] = 0$$ Assumption (i) is automatically satisfied if the ML method is sufficiently flexible to describe the data. Assumption (ii) is still needed, with some caveats. Non-collinearity is a special case of a regularization condition. It says that your model can't be too complex relative to the sample size or include redundant information. ML methods also have that issue, but typically adjust it via a "tuning parameter". The problem is there, just that some state-of-the-art ML method push the complexity to squeeze more information from the data. Versions of (iii) are still technically there for convergence, but are usually easy to satisfy in both linear regressions and ML models.
It is also worth noting that some problems in experimental analyses involve latent variables (partially unobserved $X_i$). This sometimes changes the interpretation of the exogeneity condition in both linear regression and ML models. Off-the-shelf ML just makes the most out of observed data, but state-of-the-art research adapts ML for causal models with latent variables as well.
*PS: In the linear regression $\mathbb{E}[X_i\varepsilon_i] = 0$ can replace (i).
Assumptions do matter for regression whether it is used for inference (as is most common in statistics) or prediction (as is most common in machine learning). However, the sets of assumptions are not the same; successful prediction requires less restrictive assumptions than sensible inference does. The post "T-consistency vs. P-consistency" illustrates one of the assumptions that is needed for predictive success. If the so-called predictive consistency fails, prediction with regression will fail.
Why is so little attention paid to assumptions in machine learning context? I am not sure. Perhaps the assumptions for successful prediction are quite often satisfied (at least approximately), so they are less important. Also, it might be a historical reason, but we might also see some more discussion of assumptions in future texts (who knows).
Even ignoring inference, the normality assumption matters for machine learning. In predictive modeling, the conditional distributions of the target variable are important. Gross non-normality indicates alternative models and/or methods are needed.
My post just focuses on the assumption of normality of the dependent (or target) variable; cases can be made for all the other regression assumptions as well.
Examples:
The data are very discrete. In the most extreme case, the data have only two possible values, in which case you should be using logistic regression for your predictive model. Similarly, with only a small number of ordinal values, you should use ordinal regression, and with only a small number of nominal values, you should use multinomial regression.
The data are censored. You might realize, in the process of investigating normality, that there is an upper bound. In some cases the upper bound is not really data, just an indication that the true data value is higher. In this case, ordinary predictive models must not be used because of gross biases. Censored data models must be used instead.
In the process of investigating normality (eg using q-q plots) it may become apparent that there are occasional extreme outlier observations (part of the process that you are studying) that will grossly affect ordinary predictive models. In such cases it would be prudent to use a predictive model that minimizes something other than squared errors, such as median regression, or (the negative of) a likelihood function that assumes heavy-tailed distributions. Similarly, you should evaluate predictive ability in such cases using something other than squared errors.
If you do use an ordinary predictive model, you would often like to bound the prediction error in some way for any particular prediction. The usual 95% bound $\hat Y \pm 1.96 \hat \sigma$ is valid for normal distributions (assuming that $\hat \sigma$ correctly estimates the conditional standard deviation), but not otherwise. With non-normal conditional distributions, the interval should be asymmetric and/or a different multiplier is needed.
All that having been said, there is no "thou shalt check normality" commandment. You don't have to do it at all. It's just that in certain cases, you can do better by using alternative methods when the conditional distributions are grossly non-normal.
The real answer is because most people peddling machine learning are deceptive con artists.
The curse of dimensionality precludes most complex regressions that have any sort of chaotic relationship, since you are trying to build a surface of best fit over an N-1 dimensional space. See Page 41 of David Kristjanson Duvenaud's PhD thesis. Tools like Facebook Prophet provide a great delusion to the user since they just ignore all mathematical verification and give users "what they want".
Classification models are typically easier because the surface has more potential fits that yield meaningful separation in the data. Most regression fits are not "meaningful". It is likely when 2 people see the same thing, they are actually identifying it with different separation procedures in their "neural nets".
You should think long and hard about your assumptions and try to poke holes in any failure you can imagine, because mathematical proofs are still few and far between in this protoscience.
EDIT: I've wrote a fairly simple proof of why the SMOTE algorithm actually makes no sense. I would suggest it be immediately abandoned as a legitimate method in Machine Learning. The work is here: https://mikaeltamillow96.medium.com/smote-ml-hocus-pocus-ddee12506b39
