6
$\begingroup$

I found a textbook online that had the following to say regarding the Lasso,

"Computing the p-values or confidence intervals for the coefficients of a model fitted with lasso, remains an open problem."

Why is it that currently inference on the coefficients is not possible? Is it that structurally the variance of the coefficient estimators have no closed form? Or is it something else?

$\endgroup$
3

2 Answers 2

3
$\begingroup$

Confidence intervals are a frequentist measure of uncertainty.

The researcher determines a population parameter of interest (say average income in a country) that they want to learn.

Then, the researcher collects a random sample from the population and feeds this data into a formula that puts out an interval. The formula is designed so that, if it is applied to a random sample, it yields an interval that covers the true population parameter with a fixed probability - say 95%. In other words, before the researcher collects the data they anticipate that in 95% of cases they will collect a sample on which they will compute a confidence interval that covers the true parameter.

Translating this idea to LASSO regression, one may declare the "true values" of the estimated non-zero coefficients to be the parameters of interest. In this case, the justification above does not go through since parameter selection is a property of the data sample, i.e., the researcher does not know what the parameters of interest are before they collect the data sample.

The framework of conditional inference provides a way of translating the idea of a confidence interval to scenarios where the parameters of interest depend on the data. This is not a strategy for computing traditional confidence intervals. This is a strategy for computing something that is similar to confidence intervals. The interpretation and justification is slightly different though and, as pointed out by in the answer by Pananos, not familiar to most practitioners.

I also want to note that in many applications of LASSO regression it is not really meaningful to compute any kind measure of significance for the estimated coefficients since the coefficients do not (and are not meant to) have any kind of interpretation as population parameters. In pure prediction excercises, the researcher is only interested in computing good predictive values and very different configurations of parameter values will give similar predicted values. To assess the predictive power of the estimated model one would typically assess the properties of the predicted values (e.g. via out-of-sample error), not "significance" of the estimated coefficients.

Lastly, if it is known that e.g. out of 100 variables 90 have a zero coefficient and the others have "sufficiently large" coefficients (we don't know which ones!) then it can be shown that the Lasso selects the 10 variables with non-zero coefficients with probability one if the sample size is sufficiently large. In this case, fitting a normal OLS ("post Lasso) using only the variables selected by the Lasso will yield valid confidence intervals.

$\endgroup$
1
  • $\begingroup$ Regarding your last paragraph, what about the inconsistency of LASSO as opposed to consistency of the adaptive LASSO w.r.t. variable selection? $\endgroup$ Aug 15, 2022 at 9:30
3
$\begingroup$

Why is it that currently inference on the coefficients is not possible? Is it that structurally the variance of the coefficient estimators have no closed form? Or is it something else?

Prior to some work in the area of selective inference, the bias in the estimates of the coefficients complicated the theory for testing coefficients. What's more, there may (or may not have, I can't recall) been some theory for fixed regularization strength, but we almost always estimate the regularization strength from the data, which further added to the complexity.

Around 2014/2015, work started to come out on selective inference which provided some theoretical grounding on how to do inference on penalized models such as LASSO. I'm not sure if it is very mainstream as of yet, I haven't seen it be used often, but it is an active area of research.

$\endgroup$
5
  • 1
    $\begingroup$ If you run a Bayesian lasso (or any other penalized regression) you automatically get the full inference machine. Posterior distributions are interpreted as simply as if there were no penalization. It's the frequentist setting in which things are very complicated. $\endgroup$ Aug 15, 2022 at 11:49
  • $\begingroup$ @FrankHarrell Makes a good point (though it should be mentioned that the posterior medians correspond to the estimate resulting from a LASSO). However, doing a Bayesian sort of Lasso would require some intuition on the variance of the priors, which correspond to the regularizing strength. $\endgroup$ Aug 16, 2022 at 13:12
  • $\begingroup$ The intuition required to specify the scaling factor for the priors is exactly the same challenge of finding the value of the lasso penalty factor $\lambda$. $\endgroup$ Aug 16, 2022 at 13:24
  • $\begingroup$ @FrankHarrell I’m not sure I follow. Cross validation is usually the method for selecting the lambda, where as either prior knowledge or prior predictive checks are the method for determining the scaling factor. How are these equivalent? $\endgroup$ Aug 16, 2022 at 13:40
  • $\begingroup$ The scale factor on the Laplace prior for the Bayesian lasso is exactly the reciprocal of lasso's $\lambda$. The question is how do you solve for $\lambda$ with lasso or how do you select/pre-specify $\lambda$ for Bayes. $\endgroup$ Aug 16, 2022 at 14:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.