6
$\begingroup$

I have been using the R package glmnet to do penalized regression. As part of this the package does not produce confidence intervals or p-values with regression coefficients. This is different from non-penalized regression functions like glm which all provide confidence intervals and p-values. Why are p-values not usually reported for such coefficients?

$\endgroup$
8
$\begingroup$

Little late to the party, but in case anyone stumbles across this question in the future. . . .

Best answer: have a look at section 6 of the vignette for the penalized R package ("L1 and L2 Penalized Regression Models" Jelle Goeman, Rosa Meijer, Nimisha Chaturvedi, Package version 0.9-47), https://cran.r-project.org/web/packages/penalized/vignettes/penalized.pdf.

We don't get CIs or standard errors on the coefficients when we use penalized regression because they aren't meaningful. Ordinary linear regression, or logistic regression, or whatever, provides unbiased estimates of the coefficients. A CI around that point estimate, then, can give some indication of how point estimates will be distributed around the true value of the coefficient. Penalized regression, though, uses the bias-variance tradeoff to give us coefficient estimates with lower variance, but with bias. Reporting a CI around a biased estimate will give an unrealistically optimistic indication of how close the true value of the coefficient may be to the point estimate.

("Penalized Regression, Standard Errors, and Bayesian Lassos" Minjung Kyung, Jeff Gilly, Malay Ghosh, and George Casella, Bayesian Analysis (2010) pages 369 - 412, discusses non-parametric (bootstrapped) estimates of p values for penalized regression and, if I understand correctly, they are not impressed. http://www.stat.ufl.edu/archived/casella/Papers/BL-Final.pdf)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1. You say "we": are you one of the authors of this vignette? $\endgroup$ – amoeba Jan 10 '17 at 20:11
  • 1
    $\begingroup$ No, I'm not that smart! It was a figurative "we". $\endgroup$ – eac2222 Jan 12 '17 at 2:53
  • $\begingroup$ Well one can also do variable selection using say adaptive LASSO or broken adaptive ridge (which approximates L0 penalized regression, implemented in the l0ara package) and those estimates are unbiased (they have the oracle property), so I don't see anything wrong with estimating CIs and p values in those cases, e.g. using nonparametric bootstrapping... And even if the penalized method does result in biased estimates (eg in the case of regular LASSO) the CIs and p values would still be correct conditional on the chosen regularization parameter... $\endgroup$ – Tom Wenseleers Aug 16 '19 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.