2
$\begingroup$

Thanks in advance for the advice.

I am trying to build a generalized linear model that has many predictors. The $R^{2}$ value of the model is quite low (.21), but when I use the model to predict against my validation set I am getting very good results.

I was under the impression that a low $R^{2}$ value generally means that the predictive power of a model is low. What could be going on here (I am looking for reasons why a model may make good predictions but have a low coefficient of determination)?

My training and validation sets have a similar distribution and I believe my validation and training sets to representative of the whole space.

Improve this question
$\endgroup$
4
  • $\begingroup$ I'm assuming that you're after prediction, rather than inference? Low R2 in models aimed at the latter are normal, and fine. And "R2" isn't really defined for non-gaussian GLMs. There are pseudo-R2's that are sometimes used, but read up on them. $\endgroup$
    – generic_user
    Commented Aug 17, 2014 at 12:04
  • $\begingroup$ What GLM family are you using? $\endgroup$
    – Momo
    Commented Aug 17, 2014 at 15:08
  • $\begingroup$ I'm using gamma $\endgroup$
    – HXSP1947
    Commented Aug 18, 2014 at 22:23
  • 4
    $\begingroup$ How do you measure "very good results"? $\endgroup$
    – whuber ♦
    Commented Jan 12, 2022 at 17:33

2 Answers 2

Reset to default
1
$\begingroup$

I'm aware that the question was asked a long time ago but in case anyone stumbles across it in the future - An Introduction to Statistical Learning (Chapter 3.2) has a good explanation to that question:

The $R^2$ statistic has an interpretational advantage over the $RSE$, since unlike the $RSE$, it always lies between 0 and 1. However, it can still be challenging to determine what is a good $R^2$ value, and in general, this will depend on the application. For instance, in certain problems in physics, we may know that the data truly comes from a linear model with a small residual error. In this case, we would expect to see an $R^2$ value that is extremely close to 1, and a substantially smaller $R^2$ value might indicate a serious problem with the experiment in which the data were generated. On the other hand, in typical applications in biology, psychology, marketing, and other domains, the linear model is at best an extremely rough approximation to the data, and residual errors due to other unmeasured factors are often very large. In this setting, we would expect only a very small proportion of the variance in the response to be explained by the predictor, and an $R^2$ value well below 0.1 might be more realistic!

Improve this answer
$\endgroup$
4
  • $\begingroup$ This seems totally consistent with the claim that low $R^2$ means a model that does not make accurate predictions. $\endgroup$
    – Dave
    Commented Jan 12, 2022 at 17:22
  • $\begingroup$ @Dave Yes, but it depends on what the OP means by "$R^2.$" If that's based only on the training set, it's possible to obtain a terrible $R^2$ but, if the fit happens to extrapolate well, the $R^2$ on a test set covering a wider range of explanatory values could have $R^2$ arbitrarily close to $1.$ In other words, $R^2$ at best tells us something about prediction relative to the amount of variation in the data on which it is based, but not about prediction generally. (Here, the OP was careful to indicate the training and test sets are comparable.) $\endgroup$
    – whuber ♦
    Commented Jan 12, 2022 at 17:31
  • $\begingroup$ @whuber the split of data between test and train is generally 80% and 20% respectively - so the training set is 4 times more likely to represent the true variation in explanatory variables of the dataset, than the testing set. So, could this be by chance that the testing set represented better the true variation rather than the training set? $\endgroup$
    – Jay Ekosanmi
    Commented Jan 12, 2022 at 18:38
  • $\begingroup$ @Jay It all depends on what the testing set is. Sometimes it's much, much larger than the training set. In the situation you posit, one might suspect some outliers occurred in the testing set that were not represented in the training set. (Whether that's the "true" variation is a matter for further investigation.) In extreme cases, a single outlying value can create an $R^2$ close to $1$ for a useless model. This is one reason I asked (in another comment) for an explanation of how "very good results" have been assessed in this circumstance. $\endgroup$
    – whuber ♦
    Commented Jan 12, 2022 at 18:54
1
$\begingroup$

Keep in mind what the equation for $R^2$ is.

$$ R^2 = 1 - \dfrac{ \sum_{i=1}^n \big( y_i - \hat y_i \big)^2 }{ \sum_{i=1}^n\big( y_i - \bar y \big)^2 } $$

If you are measuring model performance in terms of the numerator (or something equivalent, like $MSE$ or $RMSE$), and you find that to be acceptable despite a low $R^2$, it means that the denominator is relatively large (in other words, much variation in the unconditional $y$). If the numerator performance is poor despite a high $R^2$, it means that the denominator is relatively small (little variation in the unconditional $y$).

It is reasonable to think that you have a low $MSE$ despite a low $R^2$ simply because you don't have that much variability in the unconditional $y$ to explain.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.