9
$\begingroup$

In ordinary least squares, regressing a target vector $y$ against a set of predictors $X$, the hat matrix is computed as

$$H = X (X^tX)^{-1} X^t$$

and the PRESS (predicted residual sum of squares) is calculated by

$$SS_P = \sum_i \left( \frac{e_i}{1-h_{ii}}\right)^2$$

where $e_i$ is the $i$th residual and the $h_{ii}$ are the diagonal elements of the hat matrix.

In ridge regression with penalty coefficient $\lambda$, the hat matrix is modified to be

$$H = X (X^t X + \lambda I)^{-1} X^t$$

Can the PRESS statistic be calculated in the same way, using the modified hat matrix?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
7
$\begingroup$

yes, I use this method a lot for kernel ridge regression, and it is a good way of selecting the ridge parameter (see e.g. this paper [doi,preprint]).

A search for the optimal ridge parameter can be made very efficient if the computations a performed in canonical form (see e.g. this paper), where the model is re-parametersied so that the inverse of a diagonal matrix is required.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Thanks. In your experience, if you use PRESS to select the ridge parameter, how does your actual prediction error on a test set compare with your measured PRESS on the training set? Presumably (PRESS / n) is an underestimate of the prediction error, but is it reliable in practice? $\endgroup$
    – Chris Taylor
    Commented Jul 18, 2012 at 13:27
  • 1
    $\begingroup$ PRESS is approximately unbiased, the real problem with it is the variance, which means that there is a lot of variability depending on the particular sample of data on which it is evaluated. This means that if you optimise PRESS in model selection, you can over-fit the model selection criterion and end up with a poor model. However for the type of model I am interested in (kernel learning methods) it is pretty effective and the variance problem doesn't seem to be much worse than other criterion that might be expected to work better. $\endgroup$
    – Dikran Marsupial
    Commented Jul 18, 2012 at 13:32
  • $\begingroup$ If in doubt, you can always use bagging in addition to ridge regression as a sort of "belt-and-braces" approach to avoiding over-fitting. $\endgroup$
    – Dikran Marsupial
    Commented Jul 18, 2012 at 13:33
  • $\begingroup$ Thanks for your help! I was under the impression that bagging didn't give any improvement in linear models, e.g. as claimed in the Wikipedia article? Can you clarify? $\endgroup$
    – Chris Taylor
    Commented Jul 18, 2012 at 14:27
  • $\begingroup$ no problem. I suspect the Wikipedia article is incorrect, subset selection in linear regression is one of the examples Brieman uses in the original paper on Bagging. It is possible that least-squares linear regression without subset selection is assymptotically unaffected by bagging, but even then I doubt it applies to linear models more generally (such as logistic regression). $\endgroup$
    – Dikran Marsupial
    Commented Jul 18, 2012 at 16:03
0
$\begingroup$

The following approach can be taken to apply L2 regularisation and get the PRESS statistic. The method uses a data augmentation approach.

Assume you have N samples of Y, and K explanatory variables X1,X2...Xk....XK

  1. Add additional variable X0 that has 1 over the N samples
  2. Augment with K additional samples where:
    • Y value is 0 for each of the K samples
    • X0 value is 0 for each of the K samples
    • Xk value is SQRT(Lambda * N) * [STDEV(Xk) over N samples] if on diagonal, and 0 otherwise
  3. There are now N+K samples and K+1 variables. A normal linear regression can be solved with these inputs.
  4. As this a regression done in one step the PRESS statistic can be calculated as normal.
  5. The Lambda regularisation input has to be decided. Reviewing the PRESS statistic for different inputs of Lambada can help determine a suitable value.
Improve this answer
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.