I have been using the R package glmnet to do penalized regression. As part of this the package does not produce confidence intervals or p-values with regression coefficients. This is different from non-penalized regression functions like glm which all provide confidence intervals and p-values. Why are p-values not usually reported for such coefficients?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
Little late to the party, but in case anyone stumbles across this question in the future. . . .
Best answer: have a look at section 6 of the vignette for the penalized R package ("L1 and L2 Penalized Regression Models" Jelle Goeman, Rosa Meijer, Nimisha Chaturvedi, Package version 0.9-47), https://cran.r-project.org/web/packages/penalized/vignettes/penalized.pdf.
We don't get CIs or standard errors on the coefficients when we use penalized regression because they aren't meaningful. Ordinary linear regression, or logistic regression, or whatever, provides unbiased estimates of the coefficients. A CI around that point estimate, then, can give some indication of how point estimates will be distributed around the true value of the coefficient. Penalized regression, though, uses the bias-variance tradeoff to give us coefficient estimates with lower variance, but with bias. Reporting a CI around a biased estimate will give an unrealistically optimistic indication of how close the true value of the coefficient may be to the point estimate.
("Penalized Regression, Standard Errors, and Bayesian Lassos" Minjung Kyung, Jeff Gill, Malay Ghosh, and George Casella, Bayesian Analysis (2010) pages 369 - 411, discusses non-parametric (bootstrapped) estimates of p values for penalized regression and, if I understand correctly, they are not impressed. https://doi.org/10.1214/10-BA607 (Wayback machine link))
-
$\begingroup$ +1. You say "we": are you one of the authors of this vignette? $\endgroup$– amoebaCommented Jan 10, 2017 at 20:11
-
2$\begingroup$ No, I'm not that smart! It was a figurative "we". $\endgroup$– eac2222Commented Jan 12, 2017 at 2:53
-
1$\begingroup$ Well one can also do variable selection using say adaptive LASSO or broken adaptive ridge (which approximates L0 penalized regression, implemented in the l0ara package) and those estimates are unbiased (they have the oracle property), so I don't see anything wrong with estimating CIs and p values in those cases, e.g. using nonparametric bootstrapping... And even if the penalized method does result in biased estimates (eg in the case of regular LASSO) the CIs and p values would still be correct conditional on the chosen regularization parameter... $\endgroup$ Commented Aug 16, 2019 at 14:57
-
$\begingroup$ @TomWenseleers: "Yeah, but": 1. The regularisation parameter is a meaningless/dimensionless for a particular problem ("what do we mean by a CI of [L,U] if our $\lambda$ is 0.4?"); 2. should we choose $\lambda$ dynamically (e.g. through bootstrapping the original data and then doing CV to get $\lambda_{opt}$ on that bootstrap sample and then getting CIs) we then have a distribution of $\lambda$s and we are in a worse hole than before and 3. pretty much all non-zero coefficient will be significant by construction (otherwise how would they survive the CV procedure to get $\lambda$ anyway). $\endgroup$ Commented Mar 28, 2021 at 2:49
-
$\begingroup$ @usεr11852 For 1. It would be the confidence intervals conditional on the chosen regularisation parameter and covariates are usually normalized beforehand, so that they are all scaled the same way, for 2. Re-tuning the lambdas on the bootstrapped samples is sometimes done - one would then also take into account the uncertainty on the choice of lambda and for 3. Many of the nonzero coefficients would come out as significant, but certainly not all (I did many tests using L0 penalized regression that showed this, with LASSO even more false positives would be included, cf. selectiveInference). $\endgroup$ Commented Mar 29, 2021 at 4:57