# Standard error of parameter estimates in regularized regression

In a regularized linear regression model (e.g., ridge regression, lasso, etc.), what is the best way to obtain standard errors for parameter estimates? If cross-validation is used, is it statistically sound to estimate the standard error for each parameter as the standard deviation across folds?

-

It is probably safer to use bootstrapping or two fold cv for standard errors as this properly accounts for the "plug-in" error. Recall the conditional variance formula: $$\newcommand{\var}{\mathrm{var}}\var(b)=E[\var(b|\lambda)]+\var[E(b|\lambda)]$$

The standard deviation over the folds gives an estimate of the second term but not the first (pretty sure it's the second; definitely sure it's one of them which gets ignored). Using a two fold cv amounts to "double cross" where you do cv within each "inner" cv fold. Similarly you would do cv on each bootstrap sample. The variance of beta is given by the variance across the final result in the "outer" resample.

UPDATE

There is a fair bit of literature on the "double bootstrap" for ridge regression. Here are a few references.

Vinod H.D. (1995). Double bootstrap for shrinkage estimators. Journal of Econometrics, 68(287-302).

Note that for L1 penalty an adjustment is required to handle the zeros.

Chatterjee, A. and Lahiri, S. N. (2011). Bootstrapping Lasso estimators. Journal of the American Statistical Association, Vol. 106, No. 494: 608–625

-
 could you please point to a reference explaining this in more detail? – Pardis May 17 '12 at 9:15 I've tried to do some minor cleanup; it looked a bit like this answer was posted in a bit of a rush compared to the thoughtful and careful exposition in most of your posts. Feel free to edit further or roll back. Cheers. :) – cardinal May 17 '12 at 13:11 yeah it was a bit rushed. also written via mobile phone which is not suited to long answers. ill update it soon. – probabilityislogic May 17 '12 at 23:32