8
$\begingroup$

In ordinary least squares, regressing a target vector $y$ against a set of predictors $X$, the hat matrix is computed as

$$H = X (X^tX)^{-1} X^t$$

and the PRESS (predicted residual sum of squares) is calculated by

$$SS_P = \sum_i \left( \frac{e_i}{1-h_{ii}}\right)^2$$

where $e_i$ is the $i$th residual and the $h_{ii}$ are the diagonal elements of the hat matrix.

In ridge regression with penalty coefficient $\lambda$, the hat matrix is modified to be

$$H = X (X^t X + \lambda I)^{-1} X^t$$

Can the PRESS statistic be calculated in the same way, using the modified hat matrix?

$\endgroup$
7
$\begingroup$

yes, I use this method a lot for kernel ridge regression, and it is a good way of selecting the ridge parameter (see e.g. this paper [doi,preprint]).

A search for the optimal ridge parameter can be made very efficient if the computations a performed in canonical form (see e.g. this paper), where the model is re-parametersied so that the inverse of a diagonal matrix is required.

$\endgroup$
  • $\begingroup$ Thanks. In your experience, if you use PRESS to select the ridge parameter, how does your actual prediction error on a test set compare with your measured PRESS on the training set? Presumably (PRESS / n) is an underestimate of the prediction error, but is it reliable in practice? $\endgroup$ – Chris Taylor Jul 18 '12 at 13:27
  • 1
    $\begingroup$ PRESS is approximately unbiased, the real problem with it is the variance, which means that there is a lot of variability depending on the particular sample of data on which it is evaluated. This means that if you optimise PRESS in model selection, you can over-fit the model selection criterion and end up with a poor model. However for the type of model I am interested in (kernel learning methods) it is pretty effective and the variance problem doesn't seem to be much worse than other criterion that might be expected to work better. $\endgroup$ – Dikran Marsupial Jul 18 '12 at 13:32
  • $\begingroup$ If in doubt, you can always use bagging in addition to ridge regression as a sort of "belt-and-braces" approach to avoiding over-fitting. $\endgroup$ – Dikran Marsupial Jul 18 '12 at 13:33
  • $\begingroup$ Thanks for your help! I was under the impression that bagging didn't give any improvement in linear models, e.g. as claimed in the Wikipedia article? Can you clarify? $\endgroup$ – Chris Taylor Jul 18 '12 at 14:27
  • $\begingroup$ no problem. I suspect the Wikipedia article is incorrect, subset selection in linear regression is one of the examples Brieman uses in the original paper on Bagging. It is possible that least-squares linear regression without subset selection is assymptotically unaffected by bagging, but even then I doubt it applies to linear models more generally (such as logistic regression). $\endgroup$ – Dikran Marsupial Jul 18 '12 at 16:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.