# Using regularization when doing statistical inference

I know about the benefits of regularization when building predictive models (bias vs. variance, preventing overfitting). But, I'm wondering if it is a good idea to also do regularization (lasso, ridge, elastic net) when the main purpose of the regression model is inference on the coefficients (seeing which predictors are statisically significant). I'd love to hear people's thoughts as well as links to any academic journals or non-academic articles addressing this.

• Regularization can be seen with bayesian eyes, lasso, for instance, corresponds to some double exponential prior (with scale choosen by cross validation). So one possibility is to go full bayes. – kjetil b halvorsen Jul 14 '17 at 20:47
• determining which predictors are nonzero is what lasso is all about! If you want to determine which are statistically significantly distinct from zero, it's extremely worthwhile to consider methods like lasso – user795305 Jul 15 '17 at 23:54

The term "regularization" covers a very wide variety of methods. For the purpose of this answer, I am going to narrow in to mean "penalized optimization", i.e. adding an $L_1$ or $L_2$ penalty to your optimization problem.

If that's the case, then the answer is a definitive "Yes! Well kinda".

The reason for this is that adding an $L_1$ or $L_2$ penalty to the likelihood function leads to exactly the same mathematical function as adding either a Laplace or Gaussian a prior to a likelihood to get the posterior distribution (elevator pitch: prior distribution describes uncertainty of parameters before seeing data, posterior distribution describes uncertainty of parameters after seeing data), which leads to Bayesian statistics 101. Bayesian statistics is very popular and performed all the time with the goal of inference of estimated effects.

That was the "Yes!" part. The "Well kinda" is that optimizing your posterior distribution is done and is called "Maximum A Posterior" (MAP) estimation. But most Bayesian don't use MAP estimation, they sample from the posterior distribution using MCMC algorithms! This has several advantages, one which being that it tends to have less downward bias in the variance components.

For the sake of brevity, I have tried not to go into details about Bayesian statistics, but if this interests you, that's the place to start looking.

• (+1) But if I've used those priors only because they give good predictions - indeed I may well have tuned them for that purpose - then what am I to make of the MAP estimates or posterior distributions? (Of course if I elicited the priors to represent knowledge about the parameters before seeing the data I know exactly what to make of them.) – Scortchi - Reinstate Monica Jul 14 '17 at 21:53
• @Scortchi: that's a very good point: using cross-validation to choose penalties takes you well out of the classical Bayesian framework (as far as I know). Building a model with CV to choose regularization parameters would not fall coincide with this answer, but using regularization with fixed penalties, chosen based on expert information would. – Cliff AB Jul 14 '17 at 22:53
• A word of caution: The prior + MCMC approach will only give valid results if the posteriors for all potential coefficients are examined and reported. Otherwise, we are in a selective inference setting and most naive inference methodologies will be invalid. – user3903581 Jul 14 '17 at 23:08
• (+1) Good answer! However, I think it's may be worthwhile to clarify the sentence " But most Bayesian don't use MAP estimation, they sample from the posterior distribution using MCMC algorithms!" It seems like you're trying to say that most Bayesians use the full posterior in choosing their estimator. To see the problem, note that an estimate for the MAP can be made from the sample for the posterior distribution. – user795305 Jul 19 '17 at 6:24
• If L1 and L2 penalties are equivalent to certain Laplace and Gaussian priors, how can one determine the regularization parameter lambda required for given Laplace or Gaussian distribution parameters? Say I have multiple regression with categorical predictors and want to impose prior ~N(0, 1), how do I determine the lambda? – Melkor.cz May 19 at 16:03

There is a major difference between performing estimating using ridge type penalties and lasso-type penalties. Ridge type estimators tend to shrink all regression coefficients towards zero and are biased, but have an easy to derive asymptotic distribution because they do not shrink any variable to exactly zero. The bias in the ridge estimates may be problematic in subsequent performing hypothesis testing, but I am not an expert on that. On the other hand, Lasso/elastic-net type penalties shrink many regression coefficients to zero and can therefore be viewed as model selection techniques. The problem of performing inference on models that were selected based on data is usually referred to as the selective inference problem or post-selection inference. This field has seen many developments in recent years.

The main problem with performing inference after model selection is that selection truncates the sample space. As a simple example, suppose that we observe $y\sim N(\mu,1)$ and only want to estimate $\mu$ if we have evidence that it is larger than zero. Then, we estimate $\mu$ if $|y| > c >0$ for some pre-specified threshold $c$. In such a case, we only observe $y$ if it is larger than $c$ in absolute value and therefore $y$ is no longer normal but truncated normal.

Similarly, the Lasso (or elastic net) constrains the sample space in such a way as to ensure that the selected model has been selected. This truncation is more complicated, but can be described analytically.

Based on this insight, one can perform inference based on the truncated distribution of the data to obtain valid test statistics. For confidence intervals and test statistics see the work of Lee et al.: http://projecteuclid.org/euclid.aos/1460381681

Their methods are implemented in the R package selectiveInference.

Optimal estimation (and testing) after model selection is discussed in (for the lasso): https://arxiv.org/abs/1705.09417

and their (far less comprehensive) software package is available in: https://github.com/ammeir2/selectiveMLE

I would particularly recommend LASSO if you are attempting to use regression for inference based on "which predictors are statisically significant"--but not for the reason you might expect.

In practice, predictors in a model tend to be correlated. Even if there isn't substantial multicollinearity, regression's choice of "significant" predictors among the set of correlated predictors can vary substantially from sample to sample.

So yes, go ahead and do LASSO for your regression. Then repeat the complete model building process (including cross-validation to pick the LASSO penalty) on multiple bootstrap samples (a few hundred or so) from the original data. See how variable the set of "significant" predictors selected this way can be.

Unless your predictors are highly orthogonal to each other, this process should make you think twice about interpreting p-values in a regression in terms of which individual predictors are "significantly" important.

• +1 I agree with everything written, very pragmatic answer but why not use elastic net instead of LASSO? (given the OP also mentions it too) The ridge regularisation would control for correlations between predictors a bit more prominently. – usεr11852 Jul 14 '17 at 22:46
• It is in fact possible to compute valid p-values, estimates and confidence intervals in models which were selected via the lasso OR elastic-net, just needs to be done right. – user3903581 Jul 14 '17 at 23:11
• @user3903581 I don't dispute that one can get valid frequentist LASSO p-values, in the sense that a true null hypothesis would result in so large a coefficient less than in, say, 5% of replicate samples. The problem is in the all-too-frequent attempts to attribute causal inferences only to the predictors thus deemed "significant" without considering the issues raised by correlated predictors. – EdM Jul 15 '17 at 0:45