Regularization in Statistics and Machine Learning

Reading the Scikit-learn docs on logistic regression (https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression), I came across this note:

Note: Regularization is applied by default, which is common in machine learning but not in statistics.

Would it be right to say that the idea of regularization (especially the ridge l2 or lasso l1 penalty) is a contribution that came out of the machine learning community? Or were such methods already known to statisticians?

For instance, ridge regression and lasso are mentioned in Elements of Statistical Learning, but I am unsure whether the authors are representative of most statisticians, or if they should rather be considered machine learning researchers. Thank you!

• Ridge regression was introduced by Hoerl & Kennard in 1970, so well before the ML/AI vs. statistics debate. – chl Jan 4 at 20:24
• The reason for that comment in sklearn is that when most of the world talks about logistic regression (statisticians, scientists etc) it is performed without regularisation, since essentially you cannot perform inferential statistics with regularised models. therefore they are "apologising" for the fact that they are applying regularisation. in particular, whilst many other ML libraries have a default of 0 regularisation, their regularisation parameter is 1/(regularisation strength), so one would have to specify infinity. given this they chose an arbitrary regularisation of 1 as default. – seanv507 Jan 4 at 21:12