How can I estimate coefficient standard errors when using ridge regression?

Question

I am using ridge regression on highly multicollinear data. Using OLS I get large standard errors on the coefficients due to the multicollinearity. I know ridge regression is a way to deal with this problem, but in all the implementations of ridge regression that I've looked at, there are no standard errors reported for the coefficients. I would like some way of estimating how much the ridge regression is helping by seeing how much it is decreasing the standard errors of specific coefficients. Is there some way to estimate them in ridge regression?

Answer 1

I think boostrap would the best option to obtain robust SEs. This was done in some applied work using shrinkage methods, e.g. Analysis of North American Rheumatoid Arthritis Consortium data using a penalized logistic regression approach (BMC Proceedings 2009). There is also a nice paper from Casella on SE computation with penalized model, Penalized Regression, Standard Errors, and Bayesian Lassos (Bayesian Analysis 2010 5(2)). But they are more concerned with lasso and elasticnet penalization.

I always thought of ridge regression as a way to get better predictions than standard OLS, where the model is not parsimonious. For variable selection, the lasso or elasticnet criteria are more appropriate, but then it is difficult to apply a bootstrap procedure (since selected variables would change from one sample to the other, and even in the inner $k$-fold loop used to optimize the $\ell_1$/$\ell_2$ parameters); this is not the case with ridge regression, since you always consider all variables.

I have no idea about R packages that would give this information. It doesn't seem to be available in the glmnet package (see Friedman's paper in JSS, Regularization Paths for Generalized Linear Models via Coordinate Descent). However, Jelle Goeman who authored the penalized package discuss this point too. Cannot find the original PDF on the web, so I simply quote his words:

It is a very natural question to ask for standard errors of regression coefficients or other estimated quantities. In principle such standard errors can easily be calculated, e.g. using the bootstrap.

Still, this package deliberately does not provide them. The reason for this is that standard errors are not very meaningful for strongly biased estimates such as arise from penalized estimation methods. Penalized estimation is a procedure that reduces the variance of estimators by introducing substantial bias. The bias of each estimator is therefore a major component of its mean squared error, whereas its variance may contribute only a small part.

Unfortunately, in most applications of penalized regression it is impossible to obtain a sufficiently precise estimate of the bias. Any bootstrap-based cal- culations can only give an assessment of the variance of the estimates. Reliable estimates of the bias are only available if reliable unbiased estimates are available, which is typically not the case in situations in which penalized estimates are used.

Reporting a standard error of a penalized estimate therefore tells only part of the story. It can give a mistaken impression of great precision, completely ignoring the inaccuracy caused by the bias. It is certainly a mistake to make confidence statements that are only based on an assessment of the variance of the estimates, such as bootstrap-based confidence intervals do.

Answer 2

Assuming that the data generating process follows the standard assumptions behind OLS the standard errors for ridge regression is given by:

$ \sigma^2 (A^T A + \Gamma^T \Gamma)^{-1} A^T A (A^T A + \Gamma^T \Gamma)^{-1}$

The notation above follows the wiki notation for ridge regression. Specifically,

$A$ is the covraiate matrix,

$\sigma^2$ is the error variance.

$\Gamma$ is the Tikhonov matrix chosen suitably in ridge regression.

Answer 3

Ridge regression is a subset of Tikhonov regularization (Tk) that normalizes the smoothing factors. The more general regularizing term $\Gamma ^T\Gamma$ is replaced in ridge regression by $\text{$\lambda $I}$, where $\text{I}$ is the identity matrix, and $\lambda $ is a Lagrange (i.e., constraint) multiplier, also commonly called the smoothing, shrinkage, Tikhonov or damping factor. Both Tk and ridge regression are used to solve ill-posed integrals and other inverse problems. "An inverse problem in science is the process of calculating from a set of observations the causal factors that produced them: for example, calculating an image in computer tomography, source reconstructing in acoustics, or calculating the density of the Earth from measurements of its gravity field. here" SPSS contains supplementary code that gives the standard deviation of all the parameters and additional parameters can be derived using error propagation as in the appendix to this paper.

What is generally misunderstood about Tikhonov regularization is that the amount of smoothing has very little to do with fitting the curve, the smoothing factor should be used to minimize the error of the parameters of interest. You would have to explain a lot more about the specific problem you are trying to solve to use ridge regression properly in some valid inverse problem context, and many of the papers on selection of smoothing factors, and many of the published uses of Tikhonov regularization are a bit heuristic.

Moreover Tikhonov regularization is only one inverse problem treatment among many. Follow the link to the journal Inverse Problems.