1
$\begingroup$

In order to access whether I am overfitting a multilinear model, I have calculated the predicted $R^2$, based on the info found here.

My question is, when is a predicted $R^2$ "good enough", compared to the adjusted $R^2$?

I guess, it will (almost) always be lower than adjusted $R^2$. In my specific case, the adjusted $R^2$ of the model is $0.8788$, the predicted $R^2 = 0.8129$. Thereby, around $6\%$ of the explained variance is just noise - as I understand it. Would one regard this as acceptable or not?

Background and detail: My data is a sample of $n = 15$ humans, that shall be used for prediction of missing data regarding a larger population. Based on 'rules of thumb', $n=15$ would allow for only $1$, max $2$, predictors. Limiting to max $2$ predictors, I can only reach adjusted $R^2 = 0.55$. Running a stepwise regression analysis with $4$ scientifically relevant predictors + interactions, brings me to the above model, with $7$ significant coefficients (min. $0.05$ level; counting interactions separately), adjusted $R^2=0.8788$ and predicted $R^2=0.8129$. Most of the final terms are interactions (e.g. weight:age). The two non-interaction terms are significant ($p<0.05$), but have high standard error ($2.6$ and $3.4$), and the intercept has a high standard error of $43$, but *** significance. All interaction terms have low ($<0.02$) standard error and $p<0.05$. Overall model statistics are: $F = 13.69, df = (8,6), p = 0.002488$.

The aim is to be able to predict missing $y$ values for the population. I am thus not aiming at saying anything on the predictors themselves (then one should be more conservative, I suppose).

I am using R functions such as: lm(), summary() and PRESS.

Addition for clarrification of question: I am aware that regular $R^2$ isn't a trustworthy measure as it increases with increased amount of variables. Adjusted $R^2$ should adjust for this, but may also increase, when noise is well modelled. I am also aware that while the p-value of coefficients can be used to remove predictors in a stepwise regression, it does not tell me whether the model overall is good or not - and that doing stepwise regression is "dangerous". However, only including one predictor - which is what the sample size allows for according to rules of thumb - would likely lead to biases due to underfitting, and leave a lot of variance unexplained, so it's also not necesarily a good way to go about it. I am therefore looking for a measure to tell me whether my model is overfitted as I know there is a large risk of this. According to among other [this reference] 2, comparing predicted $R^2$, calculated by e.g. PRESS, to adjusted $R^2$ is exactly such a measure. If predicted $R^2$ is much lower than adjusted $R^2$, it means that the model cannot be generalized and therefore is no good. In my case, the predicted $R^2 = 0.8129$, while the adjusted $R^2$ of the model is $0.8788$. Thereby "only" 6% of the model should be noise - which in my case would be acceptable when the alternative is using only 1 predictor with an regular $R^2$ of max 0.5 (that is, only very little is expained).

Can one "trust" the predicted $R^2$ in this way, or should I actually be looking at different parameters, or rather model by Bayesian regression (outside my current skill set), or PCA?

$\endgroup$
5
  • 1
    $\begingroup$ Welcome to Cross Validated! A duplicate vote by me to a regression-tagged question is binding, and I don't want to unilaterally close this, but does this answer your question, even if you find the answer disappointing? $\endgroup$
    – Dave
    Feb 20 at 16:03
  • $\begingroup$ Dave is spot on. This may be helpful. Or this, because it does not really matter whether you are looking at $R^2$, MAPE or any other metric. $\endgroup$ Feb 20 at 16:43
  • $\begingroup$ @Dave: Hi, Thank you for the welcome! The link looks really good. But please keep it open a little longer and allow me to dive in a little deeper before closing :) $\endgroup$
    – Bettina
    Feb 20 at 18:56
  • $\begingroup$ @Dave. Thanks again for the link. What I get from it is that the answer to "is it good enough" depends on context. This is obvious, but good to be reminded. In my context an "overfitting of 6%" is acceptable - because the alternative (only explaining around 50% of the variance) is much worse. However, the answer given to my question below sounds like that I am not "allowed" to use stepwise regression at all, especially on such a small sample size, thus it is coming to a larger question, of whether it is valid to use the PREDICTED R^2 at all in assessment of overfitting...? $\endgroup$
    – Bettina
    Feb 21 at 10:08
  • $\begingroup$ That is, based on my context, the predicted R^2 to adjusted R^2 appears acceptable, but is it acceptable to use this measure to evaluate whether the model is overfitting or not at all? (my conclusion from R^2 is that it's not dramatically overfitting, but I am using stepwise regression and have many - but all scientifically supported - terms). $\endgroup$
    – Bettina
    Feb 21 at 10:11

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.