I know about the benefits of regularization when building predictive models (bias vs. variance, preventing overfitting). But, I'm wondering if it is a good idea to also do regularization (lasso, ridge, elastic net) when the main purpose of the regression model is inference on the coefficients (seeing which predictors are statisically significant). I'd love to hear people's thoughts as well as links to any academic journals or non-academic articles addressing this.
The term "regularization" covers a very wide variety of methods. For the purpose of this answer, I am going to narrow in to mean "penalized optimization", i.e. adding an $L_1$ or $L_2$ penalty to your optimization problem.
If that's the case, then the answer is a definitive "Yes! Well kinda".
The reason for this is that adding an $L_1$ or $L_2$ penalty to the likelihood function leads to exactly the same mathematical function as adding either a Laplace or Gaussian a prior to a likelihood to get the posterior distribution (elevator pitch: prior distribution describes uncertainty of parameters before seeing data, posterior distribution describes uncertainty of parameters after seeing data), which leads to Bayesian statistics 101. Bayesian statistics is very popular and performed all the time with the goal of inference of estimated effects.
That was the "Yes!" part. The "Well kinda" is that optimizing your posterior distribution is done and is called "Maximum A Posterior" (MAP) estimation. But most Bayesian don't use MAP estimation, they sample from the posterior distribution using MCMC algorithms! This has several advantages, one which being that it tends to have less downward bias in the variance components.
For the sake of brevity, I have tried not to go into details about Bayesian statistics, but if this interests you, that's the place to start looking.
There is a major difference between performing estimating using ridge type penalties and lasso-type penalties. Ridge type estimators tend to shrink all regression coefficients towards zero and are biased, but have an easy to derive asymptotic distribution because they do not shrink any variable to exactly zero. The bias in the ridge estimates may be problematic in subsequent performing hypothesis testing, but I am not an expert on that. On the other hand, Lasso/elastic-net type penalties shrink many regression coefficients to zero and can therefore be viewed as model selection techniques. The problem of performing inference on models that were selected based on data is usually referred to as the selective inference problem or post-selection inference. This field has seen many developments in recent years.
The main problem with performing inference after model selection is that selection truncates the sample space. As a simple example, suppose that we observe $y\sim N(\mu,1)$ and only want to estimate $\mu$ if we have evidence that it is larger than zero. Then, we estimate $\mu$ if $|y| > c >0$ for some pre-specified threshold $c$. In such a case, we only observe $y$ if it is larger than $c$ in absolute value and therefore $y$ is no longer normal but truncated normal.
Similarly, the Lasso (or elastic net) constrains the sample space in such a way as to ensure that the selected model has been selected. This truncation is more complicated, but can be described analytically.
Based on this insight, one can perform inference based on the truncated distribution of the data to obtain valid test statistics. For confidence intervals and test statistics see the work of Lee et al.: http://projecteuclid.org/euclid.aos/1460381681
Their methods are implemented in the R package selectiveInference.
Optimal estimation (and testing) after model selection is discussed in (for the lasso): https://arxiv.org/abs/1705.09417
and their (far less comprehensive) software package is available in: https://github.com/ammeir2/selectiveMLE
I would particularly recommend LASSO if you are attempting to use regression for inference based on "which predictors are statisically significant"--but not for the reason you might expect.
In practice, predictors in a model tend to be correlated. Even if there isn't substantial multicollinearity, regression's choice of "significant" predictors among the set of correlated predictors can vary substantially from sample to sample.
So yes, go ahead and do LASSO for your regression. Then repeat the complete model building process (including cross-validation to pick the LASSO penalty) on multiple bootstrap samples (a few hundred or so) from the original data. See how variable the set of "significant" predictors selected this way can be.
Unless your predictors are highly orthogonal to each other, this process should make you think twice about interpreting p-values in a regression in terms of which individual predictors are "significantly" important.