9
$\begingroup$

When calculating the predicted $R^2$ value for a linear model using the equation

$R^2 = 1 - \frac{PRESS}{TSS}$

should the currently left out sample also be excluded when working out the mean value of the response variable ($\bar{y}$) for the calculation of TSS? That is to say, should $TSS = PRESS_{NULL}$?

$\endgroup$
1
  • $\begingroup$ I addressed something similar to this last night! I contend that the answer is yes, as that fits with the spirit of $R^2$ comparing model performance to the performance of a naïve model that always predicts the mean of $y$. $\endgroup$
    – Dave
    Commented Dec 30, 2021 at 16:28

1 Answer 1

1
$\begingroup$

You can do whatever you want, and if you define what you're calculating, you are on pretty solid ground. Anyone can reproduce your work or apply it in their own work and compare their results.

However, the usual $R^2$ has a few nice interpretations that, ideally, I would like to maintain in a modified metric like your PRESS-based $R^2$.

  1. Proportion of variance explained

  2. A comparison of model performance to the performance of a baseline model that always predicts the mean

For #1, there are issues once you move away from ordinary least squares. Linear models estimated by other methods (e.g., minimizing absolute loss, regularization) and out-of-sample testing with the OLS-fit model, and nonlinear models lack this interpretation.

However, #2 can be applied to the PRESS-based $R^2$ statistic. You get the PRESS statistic by fitting a model to all but one observation and then testing it on that omitted observation. Then you do it again and again until you have omitted every observation exactly once.

Do the same but fit an intercept-only model that always predicts the mean of the included points. Then do it again for another omitted point, then again and again until you have omitted every observation exactly once.

Now you have the PRESS statistic for your model and the PRESS statistic of a baseline model that always guesses the mean of the training points. This seems to be exactly what you propose, so your metric would be a model comparison, just like the usual $R^2$.

What people do or what software implements by default might not align with this, but I do believe this to be the right way of doing it in almost every case.

This idea of comparing your model to a baseline model that always guesses the same value comes up elsewhere, such as McFadden's $R^2$.

$\endgroup$
1
  • 1
    $\begingroup$ That's a nice list of different kinds of $R^2$. $\endgroup$
    – Galen
    Commented Nov 30, 2022 at 1:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.