# PRESS statistic for ridge regression

In ordinary least squares, regressing a target vector $y$ against a set of predictors $X$, the hat matrix is computed as

$$H = X (X^tX)^{-1} X^t$$

and the PRESS (predicted residual sum of squares) is calculated by

$$SS_P = \sum_i \left( \frac{e_i}{1-h_{ii}}\right)^2$$

where $e_i$ is the $i$th residual and the $h_{ii}$ are the diagonal elements of the hat matrix.

In ridge regression with penalty coefficient $\lambda$, the hat matrix is modified to be

$$H = X (X^t X + \lambda I)^{-1} X^t$$

Can the PRESS statistic be calculated in the same way, using the modified hat matrix?

yes, I use this method a lot for kernel ridge regression, and it is a good way of selecting the ridge parameter (see e.g. this paper [doi,preprint]).

A search for the optimal ridge parameter can be made very efficient if the computations a performed in canonical form (see e.g. this paper), where the model is re-parametersied so that the inverse of a diagonal matrix is required.

• Thanks. In your experience, if you use PRESS to select the ridge parameter, how does your actual prediction error on a test set compare with your measured PRESS on the training set? Presumably (PRESS / n) is an underestimate of the prediction error, but is it reliable in practice? – Chris Taylor Jul 18 '12 at 13:27
• PRESS is approximately unbiased, the real problem with it is the variance, which means that there is a lot of variability depending on the particular sample of data on which it is evaluated. This means that if you optimise PRESS in model selection, you can over-fit the model selection criterion and end up with a poor model. However for the type of model I am interested in (kernel learning methods) it is pretty effective and the variance problem doesn't seem to be much worse than other criterion that might be expected to work better. – Dikran Marsupial Jul 18 '12 at 13:32
• If in doubt, you can always use bagging in addition to ridge regression as a sort of "belt-and-braces" approach to avoiding over-fitting. – Dikran Marsupial Jul 18 '12 at 13:33
• Thanks for your help! I was under the impression that bagging didn't give any improvement in linear models, e.g. as claimed in the Wikipedia article? Can you clarify? – Chris Taylor Jul 18 '12 at 14:27
• no problem. I suspect the Wikipedia article is incorrect, subset selection in linear regression is one of the examples Brieman uses in the original paper on Bagging. It is possible that least-squares linear regression without subset selection is assymptotically unaffected by bagging, but even then I doubt it applies to linear models more generally (such as logistic regression). – Dikran Marsupial Jul 18 '12 at 16:03

The following approach can be taken to apply L2 regularisation and get the PRESS statistic. The method uses a data augmentation approach.

Assume you have N samples of Y, and K explanatory variables X1,X2...Xk....XK

1. Add additional variable X0 that has 1 over the N samples
2. Augment with K additional samples where:
• Y value is 0 for each of the K samples
• X0 value is 0 for each of the K samples
• Xk value is SQRT(Lambda * N) * [STDEV(Xk) over N samples] if on diagonal, and 0 otherwise
3. There are now N+K samples and K+1 variables. A normal linear regression can be solved with these inputs.
4. As this a regression done in one step the PRESS statistic can be calculated as normal.
5. The Lambda regularisation input has to be decided. Reviewing the PRESS statistic for different inputs of Lambada can help determine a suitable value.