I often hear that machine-learning algorithms are not restricted to the assumptions behind the Ordinary Least Squares (OLS) estimator for linear models, e.g.:

  • The conditional mean should be zero.

  • There is no multi-collinearity (or perfect collinearity).

  • Spherical errors: There is homoscedasticity and no autocorrelation.

Is this true? If so, wouldn't it be much easier to abandon OLS and only use machine learning?


As pointed in the other answer, OLS may perfectly well be viewed as a machine learning algorithm, so the distinction you refer to seems misguided.

You also need to distinguish the purposes of the assumptions you refer to.

E.g., having no multicollinearity is indeed necessary in order to be able obtain a (unique) OLS estimate for a given sample. That, e.g., ridge regression does not require lack of multicollinearity might be seen as an advantage over OLS. One might however also argue that multicollinearity, most of the time, just highlights poor specification of the model that can easily be avoided (e.g., the dummy variable trap).

The spherical error assumptions afford that "standard" standard errors yield valid statistical inference, such as tests and confidence intervals. Statistical inference means that you draw conclusions for an underlying population, and that cannot work without assumptions, be it that we deal with an OLS estimator or some more recent ML technique. That said, we may of course use other, more robust variance estimators when such assumptions are violated and still obtain valid inference.

Finally, and most importantly, you need to think about the purposes of your model fitting exercise. The zero conditional mean assumption may be read as saying that you correctly specified you model, and that you may hence interpret the coefficient estimates as causal effects of the regressors on the dependent variable, and not just correlations.

ML methods tend to have a focus on prediction rather than causal modeling - in a nutshell, you don't care if the predictions come from a model that reflects a causal relationship as long as they are good (e.g., nobody believes that economic growth is high in the current quarter because it was high in the previous one, and yet, such business cycle properties often allow to fairly accurately predict economic growth).

If you want to come up with causal claims from a ML method, you will need assumptions of the above type, too.


Assumptions you listed (and some other) are the ones in the wiki page, yes. Many machine learning algorithms have some assumptions by the way; some of them are restrictive and some not. And, OLS is a simple Machine Learning algorithm. It's not out of the scope. It also has more general versions such as Generalized Linear Model, Weighted Least Squares, which are also ML models. It's not that we always prefer complex ML models, some problems may be cracked via OLS, some others (possibly many) cannot be.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.