3
$\begingroup$

The formula for adjusted $R^2$ is: $$ 1 - \frac{(n-1)}{(n-p-1)}(1-R^2) $$

where $r^2$ is the coefficient of determination, $n$ is the number of points, and $p$ is the number of parameters the model has.

If I want the adjusted $R^2$ from a holdout set, is $n$ the number of points in the data the model was trained on, or the number of points in the holdout set?

$\endgroup$
3
$\begingroup$

There is no need to make the adjustment on a hold-out sample, or if you did, $p$ would equal 1. For evaluation in a hold-out sample, all coefficients estimated in the training sample must be frozen. But you really should not be using that formula for $R^2$ as it implies you are forcing $R^2$ to be positive. The correct formula allows $R^{2} < 0$ because predictions can be worse than chance. So compute $R^{2} = 1 - \frac{SSE}{SST}$ where $SSE$ is sum of squared errors and $SST$ is sum of squares total ($(n-1) \times$ the variance of $Y$).

$\endgroup$
  • $\begingroup$ Frank, thanks for your response. I don't understand why $p$ would be 1. The same model is being applied to a holdout set, so the same weights and their values are being used. Also, this isn't the formula for R^2, but rather, for adjusted R^2. $\endgroup$ – Andrew Sep 23 '13 at 14:23
  • 1
    $\begingroup$ Yes I know it's the formula for $R^{2}_{adj}$. $p=1$ since the coefficients are frozen, i.e., all predictors are reduced down to a single linear predictor for application to the independent sample. But you need to use the primitive formula for $R^2$ above anyway. $\endgroup$ – Frank Harrell Sep 23 '13 at 15:29
  • $\begingroup$ This is indeed a delicate matter. Some authors propose using the train variance in the denominator, but I agree with @FrankHarrell, that $R^2$ should indeed be allowed to be smaller than 0. Which is to say: predictions have MORE variability than raw data. $\endgroup$ – JohnRos Dec 30 '18 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.