Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
When calculating the predicted $R^2$ value for a linear model using the equation
$R^2 = 1 - \frac{PRESS}{TSS}$
should the currently left out sample also be excluded when working out the mean value of the response variable ($\bar{y}$) for the calculation of TSS? That is to say, should $TSS = PRESS_{NULL}$?
You can do whatever you want, and if you define what you're calculating, you are on pretty solid ground. Anyone can reproduce your work or apply it in their own work and compare their results.
However, the usual $R^2$ has a few nice interpretations that, ideally, I would like to maintain in a modified metric like your PRESS-based $R^2$.
Proportion of variance explained
A comparison of model performance to the performance of a baseline model that always predicts the mean
For #1, there are issues once you move away from ordinary least squares. Linear models estimated by other methods (e.g., minimizing absolute loss, regularization) and out-of-sample testing with the OLS-fit model, and nonlinear models lack this interpretation.
However, #2 can be applied to the PRESS-based $R^2$ statistic. You get the PRESS statistic by fitting a model to all but one observation and then testing it on that omitted observation. Then you do it again and again until you have omitted every observation exactly once.
Do the same but fit an intercept-only model that always predicts the mean of the included points. Then do it again for another omitted point, then again and again until you have omitted every observation exactly once.
Now you have the PRESS statistic for your model and the PRESS statistic of a baseline model that always guesses the mean of the training points. This seems to be exactly what you propose, so your metric would be a model comparison, just like the usual $R^2$.
What people do or what software implements by default might not align with this, but I do believe this to be the right way of doing it in almost every case.
This idea of comparing your model to a baseline model that always guesses the same value comes up elsewhere, such as McFadden's $R^2$.
