In regression, why not use regularization by default? I remember reading somewhere in another post about the different viewpoints between people from statistics and from machine learning or neural networks, where one user was mentioning this idea as an example of bad practice.
Even then, I cannot find anyone asking this question, so I guess there is something evident I am missing. I can only think of two hypothetical scenarios where regularization would not be preferred:


*

*The researcher is interested in unbiasedness of the estimates.

*Due to a large volume of real-time data, one looks to minimize computation time.


In the former case, I am not convinced there is any practical reason for a researcher to look for unbiasedness over a lower error, specially considering a single study. In the latter, I am not even convinced there is a relevant gain in computation time.
What am I missing?
 A: People often assume that regularization is superior to un-regularized models because they reduce multicollinearity, reduce model overfitting, and improve forecasting.  They also like regularization because it explicitly avoid the entire body of model testing associated with the Gauss-Markov Theorem and other related underlying assumptions of regular regression (testing for heteroskedasticity, autocorrelation, Normal distribution, and stationarity of residuals, etc.).  
In reality, regularization very often fails to deliver on any of the mentioned benefit above.  You can do a search on the Internet for images of LASSO and Ridge Regression and you will readily see a bunch of dramatically failing regularization models.  
Check the graph of Mean Squared Error (MSE) on the Y-axis vs. the Lambda penalty factor (on the X-axis).  And, you will find many regularization models whereby the MSE increases the minute the penalty factor is greater than zero.  That means that such a regularization model is more inaccurate than the un-regularized model when forecasting (and most probably in in-sample backtesting too).  It also means such a model has not decreased model overfitting; instead it has increased model under-fitting.  
Another graph to watch is the graph of coefficient paths with the penalty factor on the X-axis and the variables regression coefficient path on the Y-axis.  You often will observe that the most influential variables at the onset see their coefficients shrunk much faster than far less influential variables (Ridge Regression) or in some cases such influential variables are entirely taken out of the model far faster than other variables (LASSO).  
Also, problematic is that often the variables coefficients will change signs as the penalty factor increases.  In other word, whatever underlying explanatory logic you had embedded in your model, the regularization process has completely dismantled it.  If a model process changes the directional sign of some of your most explanatory causal variable in your model... that's a big problem.  
LASSO is also promoted as a very good variable selection method (because it does not only shrink coefficients, it zeroes them out).  Often, LASSO can make very erroneous variable selection.  An easy way to check that is after having run LASSO, rerun your model using only the variables selected by LASSO.  And, you may find that many of the variables are not statistically significant, or may have the wrong sign, or that too few or too many variables were selected in the model vs. other more robust variable selection methods. 
Why does regularization run into so many issues (model under-fitting, poor forecasting, dismantling underlying logic of a model, poor variable selection)?  It may be due to the underlying algorithm of regularization models.  This algorithm has two components.  The first one is an error reduction mechanism such as MIN(SSE) to find the best fit (just like for un-regularized regression) and the second one is a penalty factor that penalizes higher regression coefficients.  These two algorithms components push in opposite direction.  And, a regularization model has no way of distinguishing between a variable that is very weak, non-causal, and has a very low coefficient vs. another one that is very influential or causal, and has a very high coefficient.  The way regularization works, it is just as likely to prefer the weak non-causal variable over the more influential/causal variable.  That's a real problem.            
A: In short, regularization changes the distribution of the test statistic, rendering tests of hypothesis moot.  In instances where we want to use regression to make inferences about interventions, we want unbiasedness.
Not everything to do with data is a prediction problem.
A: My comment would be that all boils down to assumptions.  While we would like a hard and fast rule to everything, the world is at the very least a little more complicated than this.  Blindly applying either is bound to mislead our interpretation. While we cannot test if the data fits every possible model or assumption,  nor should we,  we can get into blindspots if we test one model or methodology only. 
A: another issue is that regression is often used to control for effects of other variables. Lets say I want to know if A is related to Y controlling for B, A and B are strongly correlated and my answer is no, however if I regularize A and B coefficients then my answer will be yes, which is wrong.
