Question on the usage of regularization in applied statistics/science I was reading the paper ``A significance test for the lasso'' by Lockhart, Tibshirani et al and was considering the issue of applying regularization in the applied sciences (for example, behavioral sciences).
When do statisticians consider it acceptable to apply regularization methods like LASSO to shrink coefficients? There seems to be a clash between statistical convenience and external scientific theory here. Or does one use regularization only when it fits a priori knowledge of what we want to model?
I realize this answer might vary between statisticians and I hope the question is not too soft/vague. Any insights are appreciated.
 A: If you do any significance testing (and remove insignificant parameters from your final model), you are biasing your nonzero parameters away from zero. In fact this problem affects model selection of various kinds. 
One way to avoid this bias away from zero is to regularize by shrinkage. 
There are still other reasons why one might do so, but that's a good one.
A: Regularization is just another way of estimating parameters. In ordinary least squares, we get our parameter vector via $\hat\beta_{ols}=(X^TX)^{-1}X^Ty$. In ridge regression, we modify the estimate.
$$
\hat\beta_{ridge}=(X^TX+\lambda I)^{-1}X^Ty
$$
Either approach gives an estimate of the true quantity of interest, $\beta$.
In LASSO regression, there is not a closed-form solution, but it's still just another way to estimate $\beta$.
When regularized methods give desirable outcomes, such as greater predictive accuracy, we can defend such a choice of estimator of $\beta$. Yes, regularization gives biased parameter estimates, but it's not as if $\hat\beta_{ols}=\beta$. We know that our estimate of $\beta$ is wrong. If we get a model that performs better, however, then perhaps we are less wrong.
