How can I estimate coefficient standard errors when using ridge regression?

Question

I am using ridge regression on highly multicollinear data. Using OLS I get large standard errors on the coefficients due to the multicollinearity. I know ridge regression is a way to deal with this problem, but in all the implementations of ridge regression that I've looked at, there are no standard errors reported for the coefficients. I would like some way of estimating how much the ridge regression is helping by seeing how much it is decreasing the standard errors of specific coefficients. Is there some way to estimate them in ridge regression?

Answer 1

I think boostrap would the best option to obtain robust SEs. This was done in some applied work using shrinkage methods, e.g. Analysis of North American Rheumatoid Arthritis Consortium data using a penalized logistic regression approach (BMC Proceedings 2009). There is also a nice paper from Casella on SE computation with penalized model, Penalized Regression, Standard Errors, and Bayesian Lassos (Bayesian Analysis 2010 5(2)). But they are more concerned with lasso and elasticnet penalization.

I always thought of ridge regression as a way to get better predictions than standard OLS, where the model is generally not parcimonious. For variable selection, the lasso or elasticnet criteria are more appropriate, but then it is difficult to apply a bootstrap procedure (since selected variables would change from one sample to the other, and even in the inner $k$-fold loop used to optimize the $\ell_1$/$\ell_2$ parameters); this is not the case with ridge regression, since you always consider all variables.

I have no idea about R packages that would give this information. It doesn't seem to be available in the glmnet package (see Friedman's paper in JSS, Regularization Paths for Generalized Linear Models via Coordinate Descent). However, Jelle Goeman who authored the penalized package discuss this point too. Cannot find the original PDF on the web, so I simply quote his words:

It is a very natural question to ask for standard errors of regression coefficients or other estimated quantities. In principle such standard errors can easily be calculated, e.g. using the bootstrap.

Still, this package deliberately does not provide them. The reason for this is that standard errors are not very meaningful for strongly biased estimates such as arise from penalized estimation methods. Penalized estimation is a procedure that reduces the variance of estimators by introducing substantial bias. The bias of each estimator is therefore a major component of its mean squared error, whereas its variance may contribute only a small part.

Unfortunately, in most applications of penalized regression it is impossible to obtain a sufficiently precise estimate of the bias. Any bootstrap-based cal- culations can only give an assessment of the variance of the estimates. Reliable estimates of the bias are only available if reliable unbiased estimates are available, which is typically not the case in situations in which penalized estimates are used.

Reporting a standard error of a penalized estimate therefore tells only part of the story. It can give a mistaken impression of great precision, completely ignoring the inaccuracy caused by the bias. It is certainly a mistake to make confidence statements that are only based on an assessment of the variance of the estimates, such as bootstrap-based confidence intervals do.

Answer 2

Assuming that the data generating process follows the standard assumptions behind OLS the standard errors for ridge regression is given by:

$ \sigma^2 (A^T A + \Gamma^T \Gamma)^{-1} A^T A (A^T A + \Gamma^T \Gamma)^{-1}$

The notation above follows the wiki notation for ridge regression. Specifically,

$A$ is the covraiate matrix,

$\sigma^2$ is the error variance.

$\Gamma$ is the Tikhonov matrix chosen suitably in ridge regression.