# External model validation using new data for prediction: How large of a drop in $R^2$ is significant?

I need to validate a model using `external model validation' and I have a question relating to deciding when a drop in $R^2$ when compared to $R_{prediction}^2$ is significant.

DISCLOSURE: This is not a homework question. I am seeking clarification on something that is in the textbook that I am using for an applied regression class (textbook: A second course in statistics: regression analysis).

I have model whose goal is to predict $y$ from a set of covariates. The model has already been fit and I would like to validate the model using new data. My textbook states the following:

FROM MY TEXTBOOK One simple technique is to calculate the percentage of variability in the new data explained by the the model, denoted $R_{prediction}^2$, and compare it to the coefficient of determination $R^2$ for the least squares fit of the final model. Let $y_1, y_2, ..., y_n$ represent the $n$ observations used to build and fit the final regression model and $y_{n+1}, y_{n+2}, ..., y_{n+m}$ represent the $m$ observations in the new data set. Then $$R_{prediction}^2= 1 - \left(\frac{\sum\limits_{i=n+1}^{n+m}\left(y_i - \hat{y_i}\right)^2}{\sum\limits_{i=n+1}^{n+m} \left(y_i - \bar{y} \right)^2}\right)$$ where $\hat{y_i}$ is the predicted value for the $i$th observation using the $\beta$ estimates from the fitted model and $\bar{y}$ is the sample mean of the original data. If $R_{prediction}^2$ compares favorably to $R^2$ from the least squares fit, we will have increased confidence in the usefulness of the model. However if a significant drop in $R^2$ is observed, we should be cautious about using the model for prediction in practice. (Page 316)

MY QUESTION Unfortunately, this is all the textbook says about using this technique and I have no idea how I would go about determining if a significant drop occurred. For my model I have an $R^2$ of 0.8283 and using the technique described in the text, I have a calculated $R_{prediction}^2$ of 0.7444. Could anyone point me into the direction of other sources that would be able to describe the method of determining whether or not this is a significant drop---I performed several google searches but the results have not turned up anything useful.

Thank you for taking the time to read my question.

This is a good question and one that hasn't really been answered in the literature. But I would rather emphasize (1) what is the original goal, (2) what is an acceptable final (unbiased) $R^2$, and (3) is our approach good in relationship to other possible modeling approaches, i.e., could someone using the same dataset do significantly better than us? Related issues include (1) is $Y$ properly transformed and (2) did we assume linearity of the $X$ effects when this assumption was warranted? In my experience the most frequent assumption violation that matters is the assumption of linearity.
The drop in $R^2$ does tell us about overfitting, and there are some occasions where we rethink a problem when the overfitting is subjectively large.
Note that you are assuming that $m$ is huge (at least several hundred and likely needed to be in the several thousand range). Unbiased estimates of $R^2$ from data splitting is the most imprecise method of estimating $R^2$, which is why bootstrapping and repeated cross-validation are popular.