3
$\begingroup$

I read an article that explained why high values of R-squared values do not always indicate a good model .

But I have trouble understanding their justification .

Here is what I read :

" No! A high R-squared does not necessarily indicate that the model has a good fit. That might be a surprise, but look at the fitted line plot and residual plot below. The fitted line plot displays the relationship between semiconductor electron mobility and the natural log of the density for real experimental data.

enter image description here

enter image description here

The fitted line plot shows that these data follow a nice tight function and the R-squared is 98.5%, which sounds great. However, look closer to see how the regression line systematically over and under-predicts the data (bias) at different points along the curve. You can also see patterns in the Residuals versus Fits plot, rather than the randomness that you want to see. This indicates a bad fit, and serves as a reminder as to why you should always check the residual plots. "

What I don't understand about this explanation is , "the regression line systematically over and under-predicts the data (bias) at different points along the curve" . To me , the curve seems pretty good and has fit well to the data . It also has a high R-squared value , then why do they say this is a bad fit ?

Can someone please elaborate on this ? I am having trouble understanding their explanation and also why high R-squared values need not always indicate a good model .

$\endgroup$
2
  • 1
    $\begingroup$ Your blue line suggests that, for high density, mobility tends to decrease as density increases. Your data dots do not suggest that. So the fitted model may be seriously misleading. More generally, the pattern of the residuals suggest systematic problems with the model since you can reasonably predict that when interpolating it will usually overestimate mobility in some obvious parts and in other parts underestimate mobility; this issue may suggest that it is not a good model. $\endgroup$
    – Henry
    Aug 28, 2020 at 17:25
  • $\begingroup$ Have you seen stats.stackexchange.com/questions/13314? $\endgroup$
    – whuber
    Aug 28, 2020 at 18:28

1 Answer 1

2
$\begingroup$

A high R-squared does not necessarily indicate that the model has a good fit.

It depends on what is meant by having a good fit.

Let’s look at a common way to calculate $R^2$ that is equivalent to many other calculations in simple settings.

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

The numerator of that fraction is the sum of squared model residuals, so a measure of by how much the predicted values miss the true observations.

The denominator is the sum of squared residuals for a naïve model that always predicts $\bar y$, which is a sensible guess of the conditional expectation of all you know is $y$. Notice that the denominator only depends on the data, not on the model. You get the same denominator whether you use a simple linear regression, multiple linear regression, neural network, random forest, or support vector regression (as just four examples).

Since only the numerator can change as you fit different models to the same data, the above $R^2$ calculation is equivalent to the sum of squared model residuals in that a higher $R^2$ will be achieved if and only if a lower sum of squared residuals is achieved.

Since the sum of squared residuals is a measure of by how much the predictions miss the observations, $R^2$ absolutely has an interpretation as a measure of model fit, and this is true for both linear and nonlinear models.

Where that quote has legitimacy and makes a very good point is in a situation like that presented in the question where a model is able to get very close to the true values, yet there is an obvious missing piece to the model. $R^2$ will not catch such a situation. Whether or not this matters is context-dependent. If you can make a trillion dollars using your model that has $R^2=0.985$, you’ll probably be content to use it, despite its flaws.

One issue worth mentioning is that your model seems to dip down toward the right, yet the data do not support that. If you try to predict for input (x-axis) values much greater than two, you may find your model performance to be terrible. $R^2$ on its own will not catch this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.