How is it possible to get a high $R^²$ & still have 'poor predictions'? [duplicate]

Question

The insulin-sensitivity check index (QUICKI) has an excellent linear correlation with the glucose clamp index of insulin sensitivity (SI_Clamp) that is better than that of many other surrogate indexes. However, correlation between a surrogate and reference standard may improve as variability between subjects in a cohort increases (i.e., with an increased range of values). Correlation may be excellent even when prediction of reference values by the surrogate is poor.

Source:
Chen H, Sullivan G, Quon MJ. Assessing the predictive accuracy of QUICKI as a surrogate index for insulin sensitivity using a calibration model. Diabetes. 2005 Jul;54(7):1914-25.

Based on the above I am supposed to answer how it is possible to have a high correlation while at the same time having 'poor' prediction capacity, probably due to the increased range of values. The way I interpret this is a high $R^²$ or $r$ and 'relatively' wide prediction intervals. While I have previously seen this occur in a paper as well, I don't quite get it how it is possible when I look at the formulas:

$$R^² = \frac{SSR}{SST}\qquad {\rm or} \qquad1 - \frac{SSE}{SSTO}$$

And the variation that determines the width of the interval:

$$\newcommand{\mean}{{\rm mean}}S^²({\rm prediction}) = MSE \frac{1 + 1/n + (X_h - X_{\mean})^² }{\sum(X_i - X_{\mean})^²}$$

With $MSE$ is mean squared error, $X_h$ is the value of $X$ for which you want to set up a prediction interval, $X_{\mean}$ is mean of all $X$ values, $X_i$ is the value of $X$ in observation $i$. To my understanding, if we were to have the same sample size ($N$) but for a wider range of $X$s, then your $R^²$ may increase as $SSR$ increases.

But at the same time, due to the wider range of $X$s the $S^²({\rm prediction})$ will also decrease as the denominator in the formula for $S^²({\rm prediction})$ gets bigger. Also, if $SSR$ increases while $SSTO$ remains the same, this also implies $SSE$ decreases.

Thus, am I right to say that this ($R^²$ high, predictions poor) can only occur when the $MSE$ remains unchanged and is relatively high, while $SSR$ and $SSTO$ increase (so that your $R^²$ increases)?

Or should I say: that the proportions of all elements in the $R^²$ formula (i.e.: $SSR, SSTO, SSE$) remain the same, but increase in absolute numbers. So that the MSE increases for the $S^²({\rm prediction})$, having less precise predictions, while $R^²$ remains high?

Answer 1

As you state, $R^2 = 1-SSE/SST$, where SSE is the sum of squared residuals of the model and SST is the sum of squared residuals of a simple model that just predicts the average response variable for each observation.

Consider two regression models. The first has data in a relatively small range:

dat <- data.frame(x=1:10, y=c(2, 1, 4, 3, 6, 5, 8, 7, 10, 9))
plot(dat)

We get an $R^2$ of 0.88, with SSE = 9.7 and SST = 82.5.

Now add the point with x=100, y=100 and re-run the experiment:

dat2 <- data.frame(x=c(1:10, 100), y=c(2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 100))
plot(dat2)

The second model has $R^2$ of 0.999, with SSE = 10.0 and SST = 8201. The model fit hasn't really improved (both models have a RMSE of roughly 1). However, the $R^2$ has improved dramatically because the simple baseline driving SST is much worse in the second case. Instead of predicting the sensible value of 5.5 for all observations as it did in the first case, it is now predicting 14.1.

We now see that "wide data" (aka a large range of response variables) can cause SST to become much worse than the "narrow data" case, causing the $R^2$ to improve without a closer model fit.

Answer 2

If variability between subjects increases then variability within subjects decreases, i.e. the intraindividual measurement error is greater. For instance, with insulin resistance we often use HOMA-IR which is a great marker because it only requires one measure of fasting glucose/insulin over time... but if you measured the same individual tomorrow, you would have a different prediction, like with blood pressure, protein urea, etc. Clinically, we rely on multiple measurements of imprecise markers to make decisions. If you predict a highly variable marker within an individual well, it still won't tell you much about that individual in general.