# Predicted R squared - when is it good enough?

In order to access whether I am overfitting a multilinear model, I have calculated the predicted $$R^2$$, based on the info found here.

My question is, when is a predicted $$R^2$$ "good enough", compared to the adjusted $$R^2$$?

I guess, it will (almost) always be lower than adjusted $$R^2$$. In my specific case, the adjusted $$R^2$$ of the model is $$0.8788$$, the predicted $$R^2 = 0.8129$$. Thereby, around $$6\%$$ of the explained variance is just noise - as I understand it. Would one regard this as acceptable or not?

Background and detail: My data is a sample of $$n = 15$$ humans, that shall be used for prediction of missing data regarding a larger population. Based on 'rules of thumb', $$n=15$$ would allow for only $$1$$, max $$2$$, predictors. Limiting to max $$2$$ predictors, I can only reach adjusted $$R^2 = 0.55$$. Running a stepwise regression analysis with $$4$$ scientifically relevant predictors + interactions, brings me to the above model, with $$7$$ significant coefficients (min. $$0.05$$ level; counting interactions separately), adjusted $$R^2=0.8788$$ and predicted $$R^2=0.8129$$. Most of the final terms are interactions (e.g. weight:age). The two non-interaction terms are significant ($$p<0.05$$), but have high standard error ($$2.6$$ and $$3.4$$), and the intercept has a high standard error of $$43$$, but *** significance. All interaction terms have low ($$<0.02$$) standard error and $$p<0.05$$. Overall model statistics are: $$F = 13.69, df = (8,6), p = 0.002488$$.

The aim is to be able to predict missing $$y$$ values for the population. I am thus not aiming at saying anything on the predictors themselves (then one should be more conservative, I suppose).

I am using R functions such as: lm(), summary() and PRESS.

Addition for clarrification of question: I am aware that regular $$R^2$$ isn't a trustworthy measure as it increases with increased amount of variables. Adjusted $$R^2$$ should adjust for this, but may also increase, when noise is well modelled. I am also aware that while the p-value of coefficients can be used to remove predictors in a stepwise regression, it does not tell me whether the model overall is good or not - and that doing stepwise regression is "dangerous". However, only including one predictor - which is what the sample size allows for according to rules of thumb - would likely lead to biases due to underfitting, and leave a lot of variance unexplained, so it's also not necesarily a good way to go about it. I am therefore looking for a measure to tell me whether my model is overfitted as I know there is a large risk of this. According to among other [this reference] 2, comparing predicted $$R^2$$, calculated by e.g. PRESS, to adjusted $$R^2$$ is exactly such a measure. If predicted $$R^2$$ is much lower than adjusted $$R^2$$, it means that the model cannot be generalized and therefore is no good. In my case, the predicted $$R^2 = 0.8129$$, while the adjusted $$R^2$$ of the model is $$0.8788$$. Thereby "only" 6% of the model should be noise - which in my case would be acceptable when the alternative is using only 1 predictor with an regular $$R^2$$ of max 0.5 (that is, only very little is expained).

Can one "trust" the predicted $$R^2$$ in this way, or should I actually be looking at different parameters, or rather model by Bayesian regression (outside my current skill set), or PCA?

• Welcome to Cross Validated! A duplicate vote by me to a regression-tagged question is binding, and I don't want to unilaterally close this, but does this answer your question, even if you find the answer disappointing?
– Dave
Feb 20 at 16:03
• Dave is spot on. This may be helpful. Or this, because it does not really matter whether you are looking at $R^2$, MAPE or any other metric. Feb 20 at 16:43
• @Dave: Hi, Thank you for the welcome! The link looks really good. But please keep it open a little longer and allow me to dive in a little deeper before closing :) Feb 20 at 18:56
• @Dave. Thanks again for the link. What I get from it is that the answer to "is it good enough" depends on context. This is obvious, but good to be reminded. In my context an "overfitting of 6%" is acceptable - because the alternative (only explaining around 50% of the variance) is much worse. However, the answer given to my question below sounds like that I am not "allowed" to use stepwise regression at all, especially on such a small sample size, thus it is coming to a larger question, of whether it is valid to use the PREDICTED R^2 at all in assessment of overfitting...? Feb 21 at 10:08
• That is, based on my context, the predicted R^2 to adjusted R^2 appears acceptable, but is it acceptable to use this measure to evaluate whether the model is overfitting or not at all? (my conclusion from R^2 is that it's not dramatically overfitting, but I am using stepwise regression and have many - but all scientifically supported - terms). Feb 21 at 10:11