3
$\begingroup$

I need to validate a model using `external model validation' and I have a question relating to deciding when a drop in $R^2$ when compared to $R_{prediction}^2$ is significant.

DISCLOSURE: This is not a homework question. I am seeking clarification on something that is in the textbook that I am using for an applied regression class (textbook: A second course in statistics: regression analysis).

I have model whose goal is to predict $y$ from a set of covariates. The model has already been fit and I would like to validate the model using new data. My textbook states the following:

FROM MY TEXTBOOK One simple technique is to calculate the percentage of variability in the new data explained by the the model, denoted $R_{prediction}^2$, and compare it to the coefficient of determination $R^2$ for the least squares fit of the final model. Let $y_1, y_2, ..., y_n$ represent the $n$ observations used to build and fit the final regression model and $y_{n+1}, y_{n+2}, ..., y_{n+m}$ represent the $m$ observations in the new data set. Then $$R_{prediction}^2= 1 - \left(\frac{\sum\limits_{i=n+1}^{n+m}\left(y_i - \hat{y_i}\right)^2}{\sum\limits_{i=n+1}^{n+m} \left(y_i - \bar{y} \right)^2}\right)$$ where $\hat{y_i}$ is the predicted value for the $i$th observation using the $\beta$ estimates from the fitted model and $\bar{y}$ is the sample mean of the original data. If $R_{prediction}^2$ compares favorably to $R^2$ from the least squares fit, we will have increased confidence in the usefulness of the model. However if a significant drop in $R^2$ is observed, we should be cautious about using the model for prediction in practice. (Page 316)

MY QUESTION Unfortunately, this is all the textbook says about using this technique and I have no idea how I would go about determining if a significant drop occurred. For my model I have an $R^2$ of 0.8283 and using the technique described in the text, I have a calculated $R_{prediction}^2$ of 0.7444. Could anyone point me into the direction of other sources that would be able to describe the method of determining whether or not this is a significant drop---I performed several google searches but the results have not turned up anything useful.

Thank you for taking the time to read my question.

$\endgroup$
2
$\begingroup$

This is a good question and one that hasn't really been answered in the literature. But I would rather emphasize (1) what is the original goal, (2) what is an acceptable final (unbiased) $R^2$, and (3) is our approach good in relationship to other possible modeling approaches, i.e., could someone using the same dataset do significantly better than us? Related issues include (1) is $Y$ properly transformed and (2) did we assume linearity of the $X$ effects when this assumption was warranted? In my experience the most frequent assumption violation that matters is the assumption of linearity.

The drop in $R^2$ does tell us about overfitting, and there are some occasions where we rethink a problem when the overfitting is subjectively large.

Note that you are assuming that $m$ is huge (at least several hundred and likely needed to be in the several thousand range). Unbiased estimates of $R^2$ from data splitting is the most imprecise method of estimating $R^2$, which is why bootstrapping and repeated cross-validation are popular.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.