I remember reading somewhere in another post about the different viewpoints between people from statistics and from machine learning or neural networks, where one user was mentioning this idea as an example of bad practice.

Even then, I cannot find anyone asking this question, so I guess there is something evident I am missing. I can only think of two hypothetical scenarios where regularization would not be preferred:

  1. The researcher is interested in unbiasedness of the estimates.
  2. Due to a large volume of real-time data, one looks to minimize computation time.

In the former case, I am not convinced there is any practical reason for a researcher to look for unbiasedness over a lower error, specially considering a single study. In the latter, I am not even convinced there is a relevant gain in computation time.

What am I missing?

  • 6
    $\begingroup$ Testing hypothesis? $\endgroup$
    – Michael M
    Jan 2 '20 at 21:07
  • 1
    $\begingroup$ I can't see how regularization could possibly minimize computation time (?) $\endgroup$
    – Tim
    Jan 2 '20 at 22:24
  • 2
    $\begingroup$ @Tim, the claim is the opposite: regularization would not be preferred... Due to a large volume of real-time data, one looks to minimize computation time. $\endgroup$ Jan 3 '20 at 5:31
  • $\begingroup$ Related: stats.stackexchange.com/questions/403459/…. Personally, I'm in the "feel free to regularize" camp. $\endgroup$ Jan 7 '20 at 21:41

In short, regularization changes the distribution of the test statistic, rendering tests of hypothesis moot. In instances where we want to use regression to make inferences about interventions, we want unbiasedness.

Not everything to do with data is a prediction problem.

  • 7
    $\begingroup$ In most sciences, models that will be used in the future for prediction are very uncommon. $\endgroup$ Jan 2 '20 at 21:10
  • 5
    $\begingroup$ I am not sure I follow how unbiasedness is more important than lower mean squared error when we are dealing with inference. On finite studies (i.e. real life) inference will be wrong in any case, why priorize one type of error over the other one? $\endgroup$
    – Kuku
    Jan 2 '20 at 21:12
  • 2
    $\begingroup$ @Kuku how do you propose we do parameter inference in a regularized regression? $\endgroup$
    – Dave
    Jan 2 '20 at 21:15
  • 4
    $\begingroup$ @kuku sure, but being Bayesian has it's own problems. The two largest obstacles are a) justifying priors which actually reflect our present knowledge about the phenomenon, and b) the computational burden of modern MCMC methods. Besides, with enough data, the priors are dominated by the likelihood, and the regularization is overcome. $\endgroup$ Jan 2 '20 at 21:20
  • 2
    $\begingroup$ @Dave in special cases, yes. There is a one to one correspondence between the prior standard deviation for regression coefficients and the regularization strength in Ridge regression. But more generally, no. $\endgroup$ Jan 2 '20 at 22:01

People often assume that regularization is superior to un-regularized models because they reduce multicollinearity, reduce model overfitting, and improve forecasting. They also like regularization because it explicitly avoid the entire body of model testing associated with the Gauss-Markov Theorem and other related underlying assumptions of regular regression (testing for heteroskedasticity, autocorrelation, Normal distribution, and stationarity of residuals, etc.).

In reality, regularization very often fails to deliver on any of the mentioned benefit above. You can do a search on the Internet for images of LASSO and Ridge Regression and you will readily see a bunch of dramatically failing regularization models.

Check the graph of Mean Squared Error (MSE) on the Y-axis vs. the Lambda penalty factor (on the X-axis). And, you will find many regularization models whereby the MSE increases the minute the penalty factor is greater than zero. That means that such a regularization model is more inaccurate than the un-regularized model when forecasting (and most probably in in-sample backtesting too). It also means such a model has not decreased model overfitting; instead it has increased model under-fitting.

Another graph to watch is the graph of coefficient paths with the penalty factor on the X-axis and the variables regression coefficient path on the Y-axis. You often will observe that the most influential variables at the onset see their coefficients shrunk much faster than far less influential variables (Ridge Regression) or in some cases such influential variables are entirely taken out of the model far faster than other variables (LASSO).

Also, problematic is that often the variables coefficients will change signs as the penalty factor increases. In other word, whatever underlying explanatory logic you had embedded in your model, the regularization process has completely dismantled it. If a model process changes the directional sign of some of your most explanatory causal variable in your model... that's a big problem.

LASSO is also promoted as a very good variable selection method (because it does not only shrink coefficients, it zeroes them out). Often, LASSO can make very erroneous variable selection. An easy way to check that is after having run LASSO, rerun your model using only the variables selected by LASSO. And, you may find that many of the variables are not statistically significant, or may have the wrong sign, or that too few or too many variables were selected in the model vs. other more robust variable selection methods.

Why does regularization run into so many issues (model under-fitting, poor forecasting, dismantling underlying logic of a model, poor variable selection)? It may be due to the underlying algorithm of regularization models. This algorithm has two components. The first one is an error reduction mechanism such as MIN(SSE) to find the best fit (just like for un-regularized regression) and the second one is a penalty factor that penalizes higher regression coefficients. These two algorithms components push in opposite direction. And, a regularization model has no way of distinguishing between a variable that is very weak, non-causal, and has a very low coefficient vs. another one that is very influential or causal, and has a very high coefficient. The way regularization works, it is just as likely to prefer the weak non-causal variable over the more influential/causal variable. That's a real problem.


My comment would be that all boils down to assumptions. While we would like a hard and fast rule to everything, the world is at the very least a little more complicated than this. Blindly applying either is bound to mislead our interpretation. While we cannot test if the data fits every possible model or assumption, nor should we, we can get into blindspots if we test one model or methodology only.


another issue is that regression is often used to control for effects of other variables. Lets say I want to know if A is related to Y controlling for B, A and B are strongly correlated and my answer is no, however if I regularize A and B coefficients then my answer will be yes, which is wrong.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.