2
$\begingroup$

My question is pretty straightforward: It seems to me that using a correlation coefficient instead of simply directly calculating the coefficient of determination (using one set of observations as your predictors) will erroneously inflate the reliability estimation, by introducing a "hidden" linear regression into the model.

So, why is this done?

Edit:

To help illustrate my point: Consider an examination of batting averages over the course of a baseball season. In order to test the reliability of the "batting average" statistic, we might randomly partition our data in half and then correlate each player's batting average between the two halves of the data.

However, a correlation here seems inappropriate, as it "hides" a linear regression in our analysis.

To see this, imagine that by some miracle, each player's batting average in the "first" half of the data varies from their batting average in the "second" half proportionally, by a fixed constant (this is exceedingly unlikely if we randomly partition our data, but it serves to illustrate the point). For such an arrangement, r (and thus r^2) is unity, because there's a perfectly linear relationship between the values (even though they may differ substantially!). From this, we conclude that our test is perfectly reliable. This seems pretty clearly erroneous.

On the other hand, if we simply take the sum of squared differences between each player's batting average in each half of the data, and then divide that by the variance of one half of the data (in essence, calculating a coefficient of determination for a model which naively predicts the batting average in the "second" half of the data to be the same as in the "first," rather than doing a linear regression), we clearly do not get a value of unity.

Edit #2: Another way to think of this, is that it seems that split-half reliability testing is basically 2-fold cross-validation using a linear regression as the model. My question is, what justification is there for the linear regression?

$\endgroup$
  • $\begingroup$ Isn't the coefficient of determination just R^2? And if so, it's the square of the correlation. So it's the same thing. $\endgroup$ – Jeremy Miles Nov 7 '16 at 21:19
  • $\begingroup$ The square of the correlation coefficient is the coefficient of determination of a least-squares linear regression of your predictor (in this case, one half of the data) against your dependent variable (the other half). I can't see any obvious justification for doing the linear regression, rather than simply computing the squared residuals and dividing by the variance. $\endgroup$ – user3716267 Nov 7 '16 at 21:26
  • $\begingroup$ Perhaps a toy dataset and example would help me to see what you are saying. $\endgroup$ – Jeremy Miles Nov 7 '16 at 22:05
  • $\begingroup$ This is an (admittedly) very tortured example, but: Consider a data set partitioned in two, where for each subject, the value in one half of the data set is precisely a third that of the value in the other half. The resulting correlation yields an r^2 of unity, indicating perfect reliability of the test - this clearly is incorrect. Directly calculating the coefficient of determination, sans linear regression, would not yield a value of unity. $\endgroup$ – user3716267 Nov 7 '16 at 22:24
  • $\begingroup$ In order for readers to understand what you are referring to, it would help to see a clear, simple, small example of one of those applications. Otherwise the answers might differ depending on which specific techniques or "related methods" people have in mind. $\endgroup$ – whuber Nov 7 '16 at 22:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.