I found a textbook online that had the following to say regarding the Lasso,
"Computing the p-values or confidence intervals for the coefficients of a model fitted with lasso, remains an open problem."
Why is it that currently inference on the coefficients is not possible? Is it that structurally the variance of the coefficient estimators have no closed form? Or is it something else?
Confidence intervals are a frequentist measure of uncertainty.
The researcher determines a population parameter of interest (say average income in a country) that they want to learn.
Then, the researcher collects a random sample from the population and feeds this data into a formula that puts out an interval. The formula is designed so that, if it is applied to a random sample, it yields an interval that covers the true population parameter with a fixed probability - say 95%. In other words, before the researcher collects the data they anticipate that in 95% of cases they will collect a sample on which they will compute a confidence interval that covers the true parameter.
Translating this idea to LASSO regression, one may declare the "true values" of the estimated non-zero coefficients to be the parameters of interest. In this case, the justification above does not go through since parameter selection is a property of the data sample, i.e., the researcher does not know what the parameters of interest are before they collect the data sample.
The framework of conditional inference provides a way of translating the idea of a confidence interval to scenarios where the parameters of interest depend on the data. This is not a strategy for computing traditional confidence intervals. This is a strategy for computing something that is similar to confidence intervals. The interpretation and justification is slightly different though and, as pointed out by in the answer by Pananos, not familiar to most practitioners.
I also want to note that in many applications of LASSO regression it is not really meaningful to compute any kind measure of significance for the estimated coefficients since the coefficients do not (and are not meant to) have any kind of interpretation as population parameters. In pure prediction excercises, the researcher is only interested in computing good predictive values and very different configurations of parameter values will give similar predicted values. To assess the predictive power of the estimated model one would typically assess the properties of the predicted values (e.g. via out-of-sample error), not "significance" of the estimated coefficients.
Lastly, if it is known that e.g. out of 100 variables 90 have a zero coefficient and the others have "sufficiently large" coefficients (we don't know which ones!) then it can be shown that the Lasso selects the 10 variables with non-zero coefficients with probability one if the sample size is sufficiently large. In this case, fitting a normal OLS ("post Lasso) using only the variables selected by the Lasso will yield valid confidence intervals.
Why is it that currently inference on the coefficients is not possible? Is it that structurally the variance of the coefficient estimators have no closed form? Or is it something else?
Prior to some work in the area of selective inference, the bias in the estimates of the coefficients complicated the theory for testing coefficients. What's more, there may (or may not have, I can't recall) been some theory for fixed regularization strength, but we almost always estimate the regularization strength from the data, which further added to the complexity.
Around 2014/2015, work started to come out on selective inference which provided some theoretical grounding on how to do inference on penalized models such as LASSO. I'm not sure if it is very mainstream as of yet, I haven't seen it be used often, but it is an active area of research.
