How to use press statistic for model selection

I am confused about how to use the PRESS statistic to compare models.I understand that the PRESS statistic is calculated by summing the square of the residuals as :

$$\text{PRESS} = \sum_{i=1}^n (y_i - \hat{y}_{i, -i})^2$$

where the residual is the difference between the observed and predicted value for the $$i$$-th data point, with the prediction coming from a model trained on data with the $$i$$-th data point removed. My confusion lies in the fact that a new regression equation (hence a new model) is estimated each time a data point is dropped (so $$n$$ different models are trained in the process of calculating PRESS) - so the final PRESS statistic is not tied to a single model. In that case, how can you use the PRESS statistics to compare two different models? How do you calculate a PRESS statistic for a given regression model? I think I am making a basic mistake somewhere here but not sure where my reasoning is off. Thanks for any help.

You calculate PRESS on a model trained on $$n$$ values to get an idea of its out-of-sample performance, by leaving out one sample at a time. So while you indeed end up with $$n$$ models to determine the statistic, you eventually use the original model trained on all $$n$$ values.

Since you are only leaving out a single observation at a time (like in LOOCV), the addition of the 'last' sample has minimal influence on the final model. Because of this you can safely use PRESS to compare models, even though the actual models you are comparing were not used to calculate it.

If you have a larger sample size, you could consider a form of nested cross-validation, comparing models with the inner cross-validation and evaluating the performance of the 'winning' model on the outer cross-validation loop.

• So the $n$ models that are calculated are basically the same since leaving out a single observation does not change the result much? So I can use Press to compare two models with different variables? I'm a little confused how the Press is calculated for a single model. Commented May 28, 2019 at 13:31
• Also what if the omitted point was an outlier and caused differences in the estimated model? Commented May 28, 2019 at 13:54
• Upvoted for the excellence of the answer. Commented May 28, 2019 at 17:31
• @user2450223 Just like least squares, PRESS is not very robust to outliers, because the differences are squared. However, remember that you are adding $n$ models, so the larger $n$ is, the smaller this outlier's contribution is to the total PRESS statistic. Commented May 28, 2019 at 22:38
• @user2450223: if leaving out a singe data point causes substantial changes in the model, then the model is unstable. Checking that your model fitting procedure is stable is worth while and recommended. Unfortunately, LOO cross validation schemes do not allow this (iterated aka repeated cross validation does, and that requires more than 1 case to be left out) as they have collinearity between (surrogate) model and test case so that "case-to-case" variance cannot be separated from "model-to-model" variance. Commented May 29, 2019 at 14:13

"My confusion lies in the fact that a new regression equation (hence a new model) is estimated each time a data point is dropped"

This usually isn't the case, at least for standard [ridge-] regression models; you don't actually create a new model each time, but instead can work out what the output of that model would be by looking at the "hat matrix"

$$\matrix{H} = \matrix{X}[\matrix{X}^T\matrix{X}]^{-1}\matrix{X}^T$$

You can think of the hat matrix as providing information about the stability of the model to each datapoint ("leverage"), so you can view this as being a representation of the quality of the model. If the model is highly sensitive to the particular sample of data to which it has been fitted (points have high leverage), it's PRESS statistic will be much higher than the resubstitution MSE (the MSE on the training data).

However, cross-validation (including PRESS) is best thought of as a means of evaluating the performance of a method of constructing a model, rather than directly of the model itself. So PRESS is an estimate of the generalisation performance of a model trained using that procedure on the entire dataset.