2
$\begingroup$

I am confused about how to use the PRESS statistic to compare models.I understand that the PRESS statistic is calculated by summing the square of the residuals as : enter image description here

where the residual is the difference between the observed and predicted value for the $i$-th data point, with the prediction coming from a model trained on data with the $i$-th data point removed. My confusion lies in the fact that a new regression equation (hence a new model) is estimated each time a data point is dropped (so $n$ different models are trained in the process of calculating PRESS) - so the final PRESS statistic is not tied to a single model. In that case, how can you use the PRESS statistics to compare two different models? How do you calculate a PRESS statistic for a given regression model? I think I am making a basic mistake somewhere here but not sure where my reasoning is off. Thanks for any help.

$\endgroup$
3
$\begingroup$

You calculate PRESS on a model trained on $n$ values to get an idea of its out-of-sample performance, by leaving out one sample at a time. So while you indeed end up with $n$ models to determine the statistic, you eventually use the original model trained on all $n$ values.

Since you are only leaving out a single observation at a time (like in LOOCV), the addition of the 'last' sample has minimal influence on the final model. Because of this you can safely use PRESS to compare models, even though the actual models you are comparing were not used to calculate it.

If you have a larger sample size, you could consider a form of nested cross-validation, comparing models with the inner cross-validation and evaluating the performance of the 'winning' model on the outer cross-validation loop.

$\endgroup$
  • $\begingroup$ So the $n$ models that are calculated are basically the same since leaving out a single observation does not change the result much? So I can use Press to compare two models with different variables? I'm a little confused how the Press is calculated for a single model. $\endgroup$ – user2450223 May 28 at 13:31
  • $\begingroup$ Also what if the omitted point was an outlier and caused differences in the estimated model? $\endgroup$ – user2450223 May 28 at 13:54
  • $\begingroup$ Upvoted for the excellence of the answer. $\endgroup$ – James Phillips May 28 at 17:31
  • $\begingroup$ @user2450223 Just like least squares, PRESS is not very robust to outliers, because the differences are squared. However, remember that you are adding $n$ models, so the larger $n$ is, the smaller this outlier's contribution is to the total PRESS statistic. $\endgroup$ – Frans Rodenburg May 28 at 22:38
  • $\begingroup$ @user2450223: if leaving out a singe data point causes substantial changes in the model, then the model is unstable. Checking that your model fitting procedure is stable is worth while and recommended. Unfortunately, LOO cross validation schemes do not allow this (iterated aka repeated cross validation does, and that requires more than 1 case to be left out) as they have collinearity between (surrogate) model and test case so that "case-to-case" variance cannot be separated from "model-to-model" variance. $\endgroup$ – cbeleites supports Monica May 29 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.