Problem in accepting R-squared (coefficient of determination) as a perfect measure of goodness of fit

Question

R-squared (coefficient of determination) is usually used to assess the goodness of fit of a regression model to the data. Here, I provide two simple datasets that I think their best-fit lines are equally good, but they got two different r-squared values.

x1 = [1,2,3]
y1 = [1,2,3.5]
x2 = [1,2,3]
y2 = [2,4,6.5]

The best fit line to x1,y1 got r2=0.9868, and the best fit line to x2,y2 got r2=0.9959. While the r-squared values are different for these two best-fit lines, the residuals for different points are exactly the same for them: [-0.083,0.167,-0.083]. I think these two lines are equally good in fitting their respective data, while they get different r-squared values. What is wrong with my intuition about coefficient of determination.

Answer 1

What makes you think these models are equally good? If you plot the data, you will see that the second model has a regression line that is closer to the data than the first. Y2 is not simply Y1 doubled, as that would be 2, 4, 7. Y2 is closer to linear than Y1. If you convert Y1 and Y2 to z-scores you will see that they are not equal.

Answer 2

The residuals only enter the numerator of the $R^2$ formula $$ R^2=1-\frac{SSR}{TSS} $$ so that $R^2$ basically measures the residual variation relative to to the variation of the dependent variable. While the first is the same in your two datasets, the second are not.

Answer 3

$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$

For each of these, let's use some software to calculate the numerator and denominator.

# Define the data for situation 1
#
x1 <- c(1,2,3)
y1 <- c(1,2,3.5)

# OLS linear regression fit
#
L1 <- lm(y1 ~ x1)

# Make predictions
#
yhat1 <- predict(L1)

# Calculate the numerator and denominator
#
n1 <- sum((y1 - yhat1)^2)
d1 <- sum((y1 - mean(y1))^2)

# Define the data for situation 1
#

x2 <- c(1,2,3)
y2 <- c(2,4,6.5)

# OLS linear regression fit
#
L21 <- lm(y2 ~ x2)

# Make predictions
#
yhat2 <- predict(L2)

# Calculate the numerator and denominator
#
n2 <- sum((y2 - yhat2)^2)
d2 <- sum((y2 - mean(y2))^2)

# Show the numerator and denominator values
#
n1 # 0.04166667
n2 # 0.04166667, so equal sums of squared residuals in the numerators
d1 # 3.166667
d2 # 10.16667, so unequal denominators

Therefore, you are correct to identify each model as having the same sum of squared residuals. However, because the denominators are not the same, the $R^2$ values are not the same.

I think of $R^2$ as a comparison of mistakes made by two models, with the numerator quantifying the mistakes made by the model of interest and the denominator quantifying the mistakes made by a baseline "must-beat" model. In this situation, both regression models have the same quantification of their mistakes as $0.04166667$. However, because the y values for the second model have greater variance, the "must-beat" performance is worse, making its performance more impressive, hence the higher $R^2$ score.

Perhaps think of it this way. Both models leave behind $0.04166667$ units of garbage. However, model $1$ only had $3.166667$ units of garbage to remove, while model $2$ had $10.16667$ units of garbage to remove. $R^2$ measures which did a better job of cleaning, which I would say is the model that had more to clean yet wound up with the same result.

(All of this assumes that the sum of squared residuals is the measure of performance of interest. It might not be, and there are related notions for other measures of performance, such as this for assessing "pinball" loss.)