45
$\begingroup$

When I learned linear regression in my statistics class, we are asked to check for a few assumptions which need to be true for linear regression to make sense. I won't delve deep into those assumptions, however, these assumptions don't appear when learning linear regression from machine learning perspective.

Is it because the data is so large that those assumptions are automatically taken care of? Or is it because of the loss function (i.e. gradient descent)?

$\endgroup$
7
  • 33
    $\begingroup$ Linear regression is linear regression, whether or not you call it "machine learning" or "statistics." $\endgroup$
    – Sycorax
    Sep 9 '20 at 1:19
  • 1
    $\begingroup$ I was about to say almost the exact same words. Sounds like bad teaching in the latter case $\endgroup$
    – Elenchus
    Sep 9 '20 at 1:21
  • 1
    $\begingroup$ I think ML is more likely to try to make good predictions, so eventually they use techniques to evaluate if their predictions are good for different portions of the data (cross validation etc). In the Statistics, sometimes we want a model to explain phenomenas, not only predicting. Residuals are a nice way to evaluate the goodness of the model. $\endgroup$
    – igorkf
    Sep 9 '20 at 1:23
  • $\begingroup$ When fitting a linear model from an inferential statistics point of view we want those assumptions in place so that standard errors and p-values are not biased since those values are derived from unobserved sampling distributions. If your goal is to have a model that is used for prediction only and it does it well, those assumptions are less important. $\endgroup$
    – Stefan
    Sep 9 '20 at 1:40
  • 1
    $\begingroup$ One aspect is that ML has had some incredible recent successes without recourse to much statistical theory, an example being vanilla reinforcement learning. A reasoning of “if it works, it works” has some legitimacy, yet this may result in statistical theory being down-weighted more in the ML domain as a whole than it should be? $\endgroup$ Sep 10 '20 at 4:00
56
$\begingroup$

It’s because statistics puts an emphasis on model inference, while machine learning puts an emphasis on accurate predictions.

We like normal residuals in linear regression because then the usual $\hat{\beta}=(X^TX)^{-1}X^Ty$ is a maximum likelihood estimator.

We like uncorrelated predictors because then we get tighter confidence intervals on the parameters than we would if the predictors were correlated.

In machine learning, we often don’t care about how we get the answer, just that the result has a tight fit both in and out of sample.

Leo Breiman has a famous article on the “two cultures” of modeling: https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726

Breiman, Leo. "Statistical modeling: The two cultures (with comments and a rejoinder by the author)." Statistical science 16.3 (2001): 199-231.

$\endgroup$
8
  • 15
    $\begingroup$ An otherwise good answer, but the two examples (normality and uncorrelatedness) are not the best. Normality is the least interesting and useful among regression assumptions, while uncorrelatedness is not even an assumption. $\endgroup$ Sep 9 '20 at 16:52
  • 1
    $\begingroup$ @RichardHardy Can you elaborate on "Normality is the least interesting and useful among regression assumptions"? Why do you say that? $\endgroup$
    – littleO
    Sep 10 '20 at 2:32
  • 8
    $\begingroup$ @littleO, different assumptions guarantee different properties of the OLS estimator. OLS is consistent, asymptotically normal and BLUE (best linear unbiased estimator) without the normality assumption. The latter only guarantees it is normally distributed in finite samples. There are multiple threads on CV that contain the same message. $\endgroup$ Sep 10 '20 at 5:13
  • 1
    $\begingroup$ It is also worth emphasising that different domains will emphasise different aspects and even disagree about what are considered "core" aspects of the problem. The mathematics has one viewpoint (axioms first) while the 'learning from data' ML/AI/Statistics starts from a different different expectation. $\endgroup$ Sep 23 '20 at 11:30
  • 1
    $\begingroup$ "In machine learning, we often don’t care about how we get the answer, just that the result has a tight fit both in and out of sample." .... 'We' is this the royal 'We' because 'I' care. And 'I' have seen so much sloppy work that is based upon your theory of "Whatever, this looks right so let's go with it.". Some of it even, shockingly, violates simple statistical principles!! $\endgroup$ Sep 29 '20 at 20:33
25
$\begingroup$

The typical linear regression assumptions are required mostly to make sure your inferences are right.

For instance, suppose you want to check if a certain predictor is associated with your target variable. In a linear regression setting, you would calculate the p-value associated to the coefficient of that predictor. In order to get this p-value correct, you need to satisfy all the assumptions.

In ML, on the other hand, you only want a model that can fit and generalize the patterns in your data: it's all about prediction, not inference. One would mostly care about how well the linear regression generalizes to unseen data, and this can be checked by assessing MSE on train-test splitted data or by cross validation, no need for parametric assumptions.

Of course this is not as black and white as I put it, for instance, one can use parametric assumptions to derive error estimates for predictions on new data. This can still be interesting in a ML setting. Still, you are correct in noticing that these assumptions are, in general, very important from a Stats point of view and not such a big deal in ML and that's the reason: the focus on inference vs. the focus on prediction.

$\endgroup$
8
$\begingroup$

A linear regression is a statistical procedure that can be interpreted from both perspectives. Instead I will tackle the question of comparing linear regression (and its assumptions) to other methods.

A linear regression takes the form $$ Y_i = X_i'\beta + \varepsilon_i$$ Texbooks usually ask you to check (i) Exogeneity $\mathbb{E}[\varepsilon_i \mid X_i] = 0$, (ii) Non-colinearity: $\mathbb{E}[X_iX_i']$ is invertible and (iii) homoskedasticity, $\mathbb{E}[\varepsilon_i \mid X_i] = \sigma^2$. Only (i) and (ii) are considered identifying assumptions, and (iii) can be replaced by much weaker assumptions. Normality of residuals sometimes appears in introductory texts, but has been shown to be unnecessary to understand the large sample behavior. Why do we need it? $$ \widehat{\beta} = \beta + {\underbrace{\left(\frac{X'X}{n}\right)}_{\to^p \mathbb{E}[X_iX_i']}}^{-1} \ \underbrace{\left(\frac{X'\varepsilon_i}{n}\right)}_{\to^p \mathbb{E}[X_i\varepsilon_i']}$$ Condition (i) makes the second term zero, (ii) makes sure that the matrix is invertible, (iii) or some version of it guarantees the validity of the weak law of large numbers. Similar ideas are used to compute standard errors. The estimated prediction is $X_i'\widehat{\beta}$ which converges to $X_i'\beta$.

A typical machine learning (ML) algorithm attempts a more complicated functional form $$ Y_i = g(X_i) + \varepsilon_i $$ The ``regression'' function is defined as $g(x) = \mathbb{E}[Y_i \mid X_i = x]$. By construction $$\mathbb{E}[\varepsilon_i \mid X_i] = \mathbb{E}[Y_i - g(X_i) \mid X_i] = 0$$ Assumption (i) is automatically satisfied if the ML method is sufficiently flexible to describe the data. Assumption (ii) is still needed, with some caveats. Non-collinearity is a special case of a regularization condition. It says that your model can't be too complex relative to the sample size or include redundant information. ML methods also have that issue, but typically adjust it via a "tuning parameter". The problem is there, just that some state-of-the-art ML method push the complexity to squeeze more information from the data. Versions of (iii) are still technically there for convergence, but are usually easy to satisfy in both linear regressions and ML models.

It is also worth noting that some problems in experimental analyses involve latent variables (partially unobserved $X_i$). This sometimes changes the interpretation of the exogeneity condition in both linear regression and ML models. Off-the-shelf ML just makes the most out of observed data, but state-of-the-art research adapts ML for causal models with latent variables as well.

*PS: In the linear regression $\mathbb{E}[X_i\varepsilon_i] = 0$ can replace (i).

$\endgroup$
3
  • $\begingroup$ "Only (i) and (ii) are actually needed,".... needs some major explanation imo. Sure, $\hat{\beta}$ is still unbiased in the presence of heteroscedasticity, but your inferences can be wrong because the covariance matrix won't be right.... $\endgroup$
    – duckmayr
    Sep 9 '20 at 23:24
  • $\begingroup$ Great point!The main point was about how robust standard errors require relatively weak assumptions. I replaced "(i) and (ii) are actually needed" with "only (i) and (ii) are considered identifying assumptions." $\endgroup$ Sep 10 '20 at 0:25
  • $\begingroup$ More precise, good edit $\endgroup$
    – duckmayr
    Sep 10 '20 at 0:27
6
$\begingroup$

Assumptions do matter for regression whether it is used for inference (as is most common in statistics) or prediction (as is most common in machine learning). However, the sets of assumptions are not the same; successful prediction requires less restrictive assumptions than sensible inference does. The post "T-consistency vs. P-consistency" illustrates one of the assumptions that is needed for predictive success. If the so-called predictive consistency fails, prediction with regression will fail.

Why is so little attention paid to assumptions in machine learning context? I am not sure. Perhaps the assumptions for successful prediction are quite often satisfied (at least approximately), so they are less important. Also, it might be a historical reason, but we might also see some more discussion of assumptions in future texts (who knows).

$\endgroup$
6
$\begingroup$

Even ignoring inference, the normality assumption matters for machine learning. In predictive modeling, the conditional distributions of the target variable are important. Gross non-normality indicates alternative models and/or methods are needed.

My post just focuses on the assumption of normality of the dependent (or target) variable; cases can be made for all the other regression assumptions as well.

Examples:

  1. The data are very discrete. In the most extreme case, the data have only two possible values, in which case you should be using logistic regression for your predictive model. Similarly, with only a small number of ordinal values, you should use ordinal regression, and with only a small number of nominal values, you should use multinomial regression.

  2. The data are censored. You might realize, in the process of investigating normality, that there is an upper bound. In some cases the upper bound is not really data, just an indication that the true data value is higher. In this case, ordinary predictive models must not be used because of gross biases. Censored data models must be used instead.

  3. In the process of investigating normality (eg using q-q plots) it may become apparent that there are occasional extreme outlier observations (part of the process that you are studying) that will grossly affect ordinary predictive models. In such cases it would be prudent to use a predictive model that minimizes something other than squared errors, such as median regression, or (the negative of) a likelihood function that assumes heavy-tailed distributions. Similarly, you should evaluate predictive ability in such cases using something other than squared errors.

  4. If you do use an ordinary predictive model, you would often like to bound the prediction error in some way for any particular prediction. The usual 95% bound $\hat Y \pm 1.96 \hat \sigma$ is valid for normal distributions (assuming that $\hat \sigma$ correctly estimates the conditional standard deviation), but not otherwise. With non-normal conditional distributions, the interval should be asymmetric and/or a different multiplier is needed.

All that having been said, there is no "thou shalt check normality" commandment. You don't have to do it at all. It's just that in certain cases, you can do better by using alternative methods when the conditional distributions are grossly non-normal.

$\endgroup$
4
  • $\begingroup$ "The data are very discrete. In the most extreme case, the data have only two possible values", here (presumably) you're referring to the target variable only, I'm not sure if that's always the case when you write "the data" though, so it would be helpful to others like myself if this was more clear in the post (to me "the data" just means the dataset used, which contains dependent and independent variables. Perhaps I'm wrong here though). $\endgroup$
    – baxx
    Nov 14 '20 at 15:09
  • $\begingroup$ Yes, that was implicit. The normality assumption in regression only refers to the dependent variable. $\endgroup$ Nov 14 '20 at 17:03
  • $\begingroup$ "that was implicit", I think saying "the data" to refer to the target variable would be unclear to many, so worth highlighting. $\endgroup$
    – baxx
    Nov 14 '20 at 18:54
  • 1
    $\begingroup$ Ok, I edited it. $\endgroup$ Nov 14 '20 at 19:33
2
$\begingroup$

The real answer is because most people peddling machine learning are deceptive con artists.

The curse of dimensionality precludes most complex regressions that have any sort of chaotic relationship, since you are trying to build a surface of best fit over an N-1 dimensional space. See Page 41 of David Kristjanson Duvenaud's PhD thesis. Tools like Facebook Prophet provide a great delusion to the user since they just ignore all mathematical verification and give users "what they want".

Classification models are typically easier because the surface has more potential fits that yield meaningful separation in the data. Most regression fits are not "meaningful". It is likely when 2 people see the same thing, they are actually identifying it with different separation procedures in their "neural nets".

You should think long and hard about your assumptions and try to poke holes in any failure you can imagine, because mathematical proofs are still few and far between in this protoscience.

EDIT: I've wrote a fairly simple proof of why the SMOTE algorithm actually makes no sense. I would suggest it be immediately abandoned as a legitimate method in Machine Learning. The work is here: https://mikaeltamillow96.medium.com/smote-ml-hocus-pocus-ddee12506b39

$\endgroup$
1
  • 2
    $\begingroup$ Wanted to add a nice quote from 'The Art of Doing Science and Engineering' by Richard Hamming: "I am suspicious, to this day, of getting too many solutions and not doing enough very careful thinking about what you have seen. Volume output seems to me a poor substitute for acquiring an intimate feeling for the situation being simulated." $\endgroup$ Sep 29 '20 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.