# How is it possible to get a high $R^²$ & still have 'poor predictions'?

The insulin-sensitivity check index (QUICKI) has an excellent linear correlation with the glucose clamp index of insulin sensitivity (SI_Clamp) that is better than that of many other surrogate indexes. However, correlation between a surrogate and reference standard may improve as variability between subjects in a cohort increases (i.e., with an increased range of values). Correlation may be excellent even when prediction of reference values by the surrogate is poor.

Source:
Chen H, Sullivan G, Quon MJ. Assessing the predictive accuracy of QUICKI as a surrogate index for insulin sensitivity using a calibration model. Diabetes. 2005 Jul;54(7):1914-25.

Based on the above I am supposed to answer how it is possible to have a high correlation while at the same time having 'poor' prediction capacity, probably due to the increased range of values. The way I interpret this is a high $R^²$ or $r$ and 'relatively' wide prediction intervals. While I have previously seen this occur in a paper as well, I don't quite get it how it is possible when I look at the formulas:

$$R^² = \frac{SSR}{SST}\qquad {\rm or} \qquad1 - \frac{SSE}{SSTO}$$

And the variation that determines the width of the interval:

$$\newcommand{\mean}{{\rm mean}}S^²({\rm prediction}) = MSE \frac{1 + 1/n + (X_h - X_{\mean})^² }{\sum(X_i - X_{\mean})^²}$$

With $MSE$ is mean squared error, $X_h$ is the value of $X$ for which you want to set up a prediction interval, $X_{\mean}$ is mean of all $X$ values, $X_i$ is the value of $X$ in observation $i$. To my understanding, if we were to have the same sample size ($N$) but for a wider range of $X$s, then your $R^²$ may increase as $SSR$ increases.

But at the same time, due to the wider range of $X$s the $S^²({\rm prediction})$ will also decrease as the denominator in the formula for $S^²({\rm prediction})$ gets bigger. Also, if $SSR$ increases while $SSTO$ remains the same, this also implies $SSE$ decreases.

Thus, am I right to say that this ($R^²$ high, predictions poor) can only occur when the $MSE$ remains unchanged and is relatively high, while $SSR$ and $SSTO$ increase (so that your $R^²$ increases)?

Or should I say: that the proportions of all elements in the $R^²$ formula (i.e.: $SSR, SSTO, SSE$) remain the same, but increase in absolute numbers. So that the MSE increases for the $S^²({\rm prediction})$, having less precise predictions, while $R^²$ remains high?

• For questions about quotes it's important to link or properly cite the origin of the quote, so that answerers can grasp the context before answering. I think the source is here: ncbi.nlm.nih.gov/pubmed/15983190 – Pere Dec 28 '16 at 16:39
• Hint: consider a dataset exhibiting excellent correlation. Introduce one more subject whose $X$ value is outlying. Consider what happens to the least-squares fit as well as $R^2$ as you vary the $Y$ value for this subject. – whuber Dec 28 '16 at 16:39
• Did you check this site for possible duplicate questions? – Michael Chernick Dec 28 '16 at 16:46
• @Ferdi Will try to do so next time. I tried to show I already put some thought in it, before immediatly posting the question here :) Btw, I notice a small mistake in the edit of the formula of S². It should be: MSE [ 1 + 1/n + (Xh - Xmean)² / sum (Xi - Xmean) ] Now 1 + 1/n is part of the nominator – Amonet Dec 28 '16 at 18:20
• I wasn't aware I could edit it myself, it's been added. – Amonet Dec 28 '16 at 19:07

As you state, $R^2 = 1-SSE/SST$, where SSE is the sum of squared residuals of the model and SST is the sum of squared residuals of a simple model that just predicts the average response variable for each observation.

Consider two regression models. The first has data in a relatively small range:

dat <- data.frame(x=1:10, y=c(2, 1, 4, 3, 6, 5, 8, 7, 10, 9))
plot(dat)

We get an $R^2$ of 0.88, with SSE = 9.7 and SST = 82.5.

Now add the point with x=100, y=100 and re-run the experiment:

dat2 <- data.frame(x=c(1:10, 100), y=c(2, 1, 4, 3, 6, 5, 8, 7, 10, 9, 100))
plot(dat2)

The second model has $R^2$ of 0.999, with SSE = 10.0 and SST = 8201. The model fit hasn't really improved (both models have a RMSE of roughly 1). However, the $R^2$ has improved dramatically because the simple baseline driving SST is much worse in the second case. Instead of predicting the sensible value of 5.5 for all observations as it did in the first case, it is now predicting 14.1.

We now see that "wide data" (aka a large range of response variables) can cause SST to become much worse than the "narrow data" case, causing the $R^2$ to improve without a closer model fit.

• Just to note, there many reasons why predictions may be poor despite high $R^2$ value. For example, I would be very skeptical about predictions from a model with $R^2 = 1$. I do believe @josliber's answer here provided correct reasoning for why high $R^2$ values may not mean a better surrogate in this particular case presented by the OP. – Cliff AB Dec 28 '16 at 19:33

If variability between subjects increases then variability within subjects decreases, i.e. the intraindividual measurement error is greater. For instance, with insulin resistance we often use HOMA-IR which is a great marker because it only requires one measure of fasting glucose/insulin over time... but if you measured the same individual tomorrow, you would have a different prediction, like with blood pressure, protein urea, etc. Clinically, we rely on multiple measurements of imprecise markers to make decisions. If you predict a highly variable marker within an individual well, it still won't tell you much about that individual in general.