3
$\begingroup$

In a regularized linear regression model (e.g., ridge regression, lasso, etc.), what is the best way to obtain standard errors for parameter estimates? If cross-validation is used, is it statistically sound to estimate the standard error for each parameter as the standard deviation across folds?

$\endgroup$

1 Answer 1

4
$\begingroup$

It is probably safer to use bootstrapping or two fold cv for standard errors as this properly accounts for the "plug-in" error. Recall the conditional variance formula: $$\newcommand{\var}{\mathrm{var}}\var(b)=E[\var(b|\lambda)]+\var[E(b|\lambda)]$$

The standard deviation over the folds gives an estimate of the second term but not the first (pretty sure it's the second; definitely sure it's one of them which gets ignored). Using a two fold cv amounts to "double cross" where you do cv within each "inner" cv fold. Similarly you would do cv on each bootstrap sample. The variance of beta is given by the variance across the final result in the "outer" resample.

UPDATE

There is a fair bit of literature on the "double bootstrap" for ridge regression. Here are a few references.

Vinod H.D. (1995). Double bootstrap for shrinkage estimators. Journal of Econometrics, 68(287-302).

Note that for L1 penalty an adjustment is required to handle the zeros.

Chatterjee, A. and Lahiri, S. N. (2011). Bootstrapping Lasso estimators. Journal of the American Statistical Association, Vol. 106, No. 494: 608–625

$\endgroup$
3
  • $\begingroup$ could you please point to a reference explaining this in more detail? $\endgroup$
    – Pardis
    Commented May 17, 2012 at 9:15
  • $\begingroup$ I've tried to do some minor cleanup; it looked a bit like this answer was posted in a bit of a rush compared to the thoughtful and careful exposition in most of your posts. Feel free to edit further or roll back. Cheers. :) $\endgroup$
    – cardinal
    Commented May 17, 2012 at 13:11
  • $\begingroup$ yeah it was a bit rushed. also written via mobile phone which is not suited to long answers. ill update it soon. $\endgroup$ Commented May 17, 2012 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.