I know about the benefits of regularization when building predictive models (bias vs. variance, preventing overfitting). But, I'm wondering if it is a good idea to also do regularization (lasso, ridge, elastic net) when the main purpose of the regression model is inference on the coefficients (seeing which predictors are statistically significant). I'd love to hear people's thoughts as well as links to any academic journals or non-academic articles addressing this.
$\begingroup$
$\endgroup$
2
-
4$\begingroup$ Regularization can be seen with bayesian eyes, lasso, for instance, corresponds to some double exponential prior (with scale choosen by cross validation). So one possibility is to go full bayes. $\endgroup$– kjetil b halvorsen ♦Commented Jul 14, 2017 at 20:47
-
1$\begingroup$ determining which predictors are nonzero is what lasso is all about! If you want to determine which are statistically significantly distinct from zero, it's extremely worthwhile to consider methods like lasso $\endgroup$– user795305Commented Jul 15, 2017 at 23:54
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
There is a major difference between performing estimating using ridge type penalties and lasso-type penalties. Ridge type estimators tend to shrink all regression coefficients towards zero and are biased, but have an easy to derive asymptotic distribution because they do not shrink any variable to exactly zero. The bias in the ridge estimates may be problematic in subsequent performing hypothesis testing, but I am not an expert on that. On the other hand, Lasso/elastic-net type penalties shrink many regression coefficients to zero and can therefore be viewed as model selection techniques. The problem of performing inference on models that were selected based on data is usually referred to as the selective inference problem or post-selection inference. This field has seen many developments in recent years.
The main problem with performing inference after model selection is that selection truncates the sample space. As a simple example, suppose that we observe $y\sim N(\mu,1)$ and only want to estimate $\mu$ if we have evidence that it is larger than zero. Then, we estimate $\mu$ if $|y| > c >0$ for some pre-specified threshold $c$. In such a case, we only observe $y$ if it is larger than $c$ in absolute value and therefore $y$ is no longer normal but truncated normal.
Similarly, the Lasso (or elastic net) constrains the sample space in such a way as to ensure that the selected model has been selected. This truncation is more complicated, but can be described analytically.
Based on this insight, one can perform inference based on the truncated distribution of the data to obtain valid test statistics. For confidence intervals and test statistics, see the work of Lee et al. (2016): Exact post-selection inference, with application to the lasso
Their methods are implemented in the R package selectiveInference.
Optimal estimation (and testing) after model selection is discussed in (for the lasso): Tractable Post-Selection Maximum Likelihood Inference for the Lasso | Cornell University Statistics Archives
and their (far less comprehensive) software package is available in: selectiveMLE by ammeir2 | GitHub
$\begingroup$
$\endgroup$
5
The term "regularization" covers a very wide variety of methods. For the purpose of this answer, I am going to narrow in to mean "penalized optimization", i.e. adding an $L_1$ or $L_2$ penalty to your optimization problem.
If that's the case, then the answer is a definitive "Yes! Well kinda".
The reason for this is that adding an $L_1$ or $L_2$ penalty to the likelihood function leads to exactly the same mathematical function as adding either a Laplace or Gaussian a prior to a likelihood to get the posterior distribution (elevator pitch: prior distribution describes uncertainty of parameters before seeing data, posterior distribution describes uncertainty of parameters after seeing data), which leads to Bayesian statistics 101. Bayesian statistics is very popular and performed all the time with the goal of inference of estimated effects.
That was the "Yes!" part. The "Well kinda" is that optimizing your posterior distribution is done and is called "Maximum A Posterior" (MAP) estimation. But most Bayesian don't use MAP estimation, they sample from the posterior distribution using MCMC algorithms! This has several advantages, one which being that it tends to have less downward bias in the variance components.
For the sake of brevity, I have tried not to go into details about Bayesian statistics, but if this interests you, that's the place to start looking.
-
2$\begingroup$ (+1) But if I've used those priors only because they give good predictions - indeed I may well have tuned them for that purpose - then what am I to make of the MAP estimates or posterior distributions? (Of course if I elicited the priors to represent knowledge about the parameters before seeing the data I know exactly what to make of them.) $\endgroup$– Scortchi ♦Commented Jul 14, 2017 at 21:53
-
1$\begingroup$ @Scortchi: that's a very good point: using cross-validation to choose penalties takes you well out of the classical Bayesian framework (as far as I know). Building a model with CV to choose regularization parameters would not fall coincide with this answer, but using regularization with fixed penalties, chosen based on expert information would. $\endgroup$– Cliff ABCommented Jul 14, 2017 at 22:53
-
2$\begingroup$ A word of caution: The prior + MCMC approach will only give valid results if the posteriors for all potential coefficients are examined and reported. Otherwise, we are in a selective inference setting and most naive inference methodologies will be invalid. $\endgroup$ Commented Jul 14, 2017 at 23:08
-
1$\begingroup$ (+1) Good answer! However, I think it's may be worthwhile to clarify the sentence " But most Bayesian don't use MAP estimation, they sample from the posterior distribution using MCMC algorithms!" It seems like you're trying to say that most Bayesians use the full posterior in choosing their estimator. To see the problem, note that an estimate for the MAP can be made from the sample for the posterior distribution. $\endgroup$ Commented Jul 19, 2017 at 6:24
-
$\begingroup$ If L1 and L2 penalties are equivalent to certain Laplace and Gaussian priors, how can one determine the regularization parameter lambda required for given Laplace or Gaussian distribution parameters? Say I have multiple regression with categorical predictors and want to impose prior ~N(0, 1), how do I determine the lambda? $\endgroup$ Commented May 19, 2020 at 16:03
$\begingroup$
$\endgroup$
3
I would particularly recommend LASSO if you are attempting to use regression for inference based on "which predictors are statisically significant"--but not for the reason you might expect.
In practice, predictors in a model tend to be correlated. Even if there isn't substantial multicollinearity, regression's choice of "significant" predictors among the set of correlated predictors can vary substantially from sample to sample.
So yes, go ahead and do LASSO for your regression. Then repeat the complete model building process (including cross-validation to pick the LASSO penalty) on multiple bootstrap samples (a few hundred or so) from the original data. See how variable the set of "significant" predictors selected this way can be.
Unless your predictors are highly orthogonal to each other, this process should make you think twice about interpreting p-values in a regression in terms of which individual predictors are "significantly" important.
-
1$\begingroup$ +1 I agree with everything written, very pragmatic answer but why not use elastic net instead of LASSO? (given the OP also mentions it too) The ridge regularisation would control for correlations between predictors a bit more prominently. $\endgroup$ Commented Jul 14, 2017 at 22:46
-
$\begingroup$ It is in fact possible to compute valid p-values, estimates and confidence intervals in models which were selected via the lasso OR elastic-net, just needs to be done right. $\endgroup$ Commented Jul 14, 2017 at 23:11
-
1$\begingroup$ @user3903581 I don't dispute that one can get valid frequentist LASSO p-values, in the sense that a true null hypothesis would result in so large a coefficient less than in, say, 5% of replicate samples. The problem is in the all-too-frequent attempts to attribute causal inferences only to the predictors thus deemed "significant" without considering the issues raised by correlated predictors. $\endgroup$– EdMCommented Jul 15, 2017 at 0:45