# Adjusted R squared on a holdout set

The formula for adjusted $R^2$ is: $$1 - \frac{(n-1)}{(n-p-1)}(1-R^2)$$

where $r^2$ is the coefficient of determination, $n$ is the number of points, and $p$ is the number of parameters the model has.

If I want the adjusted $R^2$ from a holdout set, is $n$ the number of points in the data the model was trained on, or the number of points in the holdout set?

There is no need to make the adjustment on a hold-out sample, or if you did, $p$ would equal 1. For evaluation in a hold-out sample, all coefficients estimated in the training sample must be frozen. But you really should not be using that formula for $R^2$ as it implies you are forcing $R^2$ to be positive. The correct formula allows $R^{2} < 0$ because predictions can be worse than chance. So compute $R^{2} = 1 - \frac{SSE}{SST}$ where $SSE$ is sum of squared errors and $SST$ is sum of squares total ($(n-1) \times$ the variance of $Y$).
• Frank, thanks for your response. I don't understand why $p$ would be 1. The same model is being applied to a holdout set, so the same weights and their values are being used. Also, this isn't the formula for R^2, but rather, for adjusted R^2. – Andrew Sep 23 '13 at 14:23
• Yes I know it's the formula for $R^{2}_{adj}$. $p=1$ since the coefficients are frozen, i.e., all predictors are reduced down to a single linear predictor for application to the independent sample. But you need to use the primitive formula for $R^2$ above anyway. – Frank Harrell Sep 23 '13 at 15:29
• This is indeed a delicate matter. Some authors propose using the train variance in the denominator, but I agree with @FrankHarrell, that $R^2$ should indeed be allowed to be smaller than 0. Which is to say: predictions have MORE variability than raw data. – JohnRos Dec 30 '18 at 9:10