To elaborate more on the question and give a few examples:

classical statistical methods: Frequentist linear regression that involves estimating beta, t-test, p-values, R-squared, F-test. Model selection using AIC/BIC by forward/backward stepwise feature selection.

machine learning paradigm: Empirical risk minimization, incorporating regularizer, train-test split and cross validation to do model selection.

It just seems that by adding regularizer and using cross validation for model selection is a much more general and easier approach, especially when dealing with a lot of variables.

However, in most of the cases, would that not make a lot of the statistical tests not applicable? (For example, trying to come up with a p-value for beta when we have a Lasso regularizer present).

One might argue that through the classical approach we have a model that is more interpretable as we know which features are important. Is this the tradeoff? Through the shrinkage of Lasso regularizer, wouldn't that give individuals an idea of what features are important?

Should I abandon one approach and embrace the other or does it depend on the circumstances?

  • 1
    $\begingroup$ The generic answer is that different tools solve different problems. ML methods tend to be solely focused on prediction, while the statistical methods you list can be useful to make inferences: how large is an effect, how likely is an effect due to chance under certain assumptions. Abandoning one approach for the other implies giving up on the ability to learn certain kinds of things from your data. Whether or not that matters depends on what kinds of problems you need to solve. More info: stats.stackexchange.com/search?q=two+cultures $\endgroup$
    – Sycorax
    Mar 30, 2021 at 2:22
  • $\begingroup$ Mandatory link to Breiman’s “Two Cultures” paper: projecteuclid.org/journalArticle/… $\endgroup$
    – Dave
    Mar 30, 2021 at 2:23


Browse other questions tagged or ask your own question.