People often assume that regularization is superior to un-regularized models because they reduce multicollinearity, reduce model overfitting, and improve forecasting. They also like regularization because it explicitly avoid the entire body of model testing associated with the Gauss-Markov Theorem and other related underlying assumptions of regular regression (testing for heteroskedasticity, autocorrelation, Normal distribution, and stationarity of residuals, etc.).
In reality, regularization very often fails to deliver on any of the mentioned benefit above. You can do a search on the Internet for images of LASSO and Ridge Regression and you will readily see a bunch of dramatically failing regularization models.
Check the graph of Mean Squared Error (MSE) on the Y-axis vs. the Lambda penalty factor (on the X-axis). And, you will find many regularization models whereby the MSE increases the minute the penalty factor is greater than zero. That means that such a regularization model is more inaccurate than the un-regularized model when forecasting (and most probably in in-sample backtesting too). It also means such a model has not decreased model overfitting; instead it has increased model under-fitting.
Another graph to watch is the graph of coefficient paths with the penalty factor on the X-axis and the variables regression coefficient path on the Y-axis. You often will observe that the most influential variables at the onset see their coefficients shrunk much faster than far less influential variables (Ridge Regression) or in some cases such influential variables are entirely taken out of the model far faster than other variables (LASSO).
Also, problematic is that often the variables coefficients will change signs as the penalty factor increases. In other word, whatever underlying explanatory logic you had embedded in your model, the regularization process has completely dismantled it. If a model process changes the directional sign of some of your most explanatory causal variable in your model... that's a big problem.
LASSO is also promoted as a very good variable selection method (because it does not only shrink coefficients, it zeroes them out). Often, LASSO can make very erroneous variable selection. An easy way to check that is after having run LASSO, rerun your model using only the variables selected by LASSO. And, you may find that many of the variables are not statistically significant, or may have the wrong sign, or that too few or too many variables were selected in the model vs. other more robust variable selection methods.
Why does regularization run into so many issues (model under-fitting, poor forecasting, dismantling underlying logic of a model, poor variable selection)? It may be due to the underlying algorithm of regularization models. This algorithm has two components. The first one is an error reduction mechanism such as MIN(SSE) to find the best fit (just like for un-regularized regression) and the second one is a penalty factor that penalizes higher regression coefficients. These two algorithms components push in opposite direction. And, a regularization model has no way of distinguishing between a variable that is very weak, non-causal, and has a very low coefficient vs. another one that is very influential or causal, and has a very high coefficient. The way regularization works, it is just as likely to prefer the weak non-causal variable over the more influential/causal variable. That's a real problem.
