Standard error of parameter estimates in regularized regression

Question

In a regularized linear regression model (e.g., ridge regression, lasso, etc.), what is the best way to obtain standard errors for parameter estimates? If cross-validation is used, is it statistically sound to estimate the standard error for each parameter as the standard deviation across folds?

Answer 1

It is probably safer to use bootstrapping or two fold cv for standard errors as this properly accounts for the "plug-in" error. Recall the conditional variance formula: $$\newcommand{\var}{\mathrm{var}}\var(b)=E[\var(b|\lambda)]+\var[E(b|\lambda)]$$

The standard deviation over the folds gives an estimate of the second term but not the first (pretty sure it's the second; definitely sure it's one of them which gets ignored). Using a two fold cv amounts to "double cross" where you do cv within each "inner" cv fold. Similarly you would do cv on each bootstrap sample. The variance of beta is given by the variance across the final result in the "outer" resample.

UPDATE

There is a fair bit of literature on the "double bootstrap" for ridge regression. Here are a few references.

Vinod H.D. (1995). Double bootstrap for shrinkage estimators. Journal of Econometrics, 68(287-302).

Note that for L1 penalty an adjustment is required to handle the zeros.

Chatterjee, A. and Lahiri, S. N. (2011). Bootstrapping Lasso estimators. Journal of the American Statistical Association, Vol. 106, No. 494: 608–625