This question relates to situations where we apply some test to the same people at Time 1 and Time 2, and then calculate the correlation between scores at Time 1 and Time 2.

I am reading the book Principles & Applications of Assessment in Counseling, which states here on p.52 that:

Therefore, if the reliability coefficient using test-retest were .80, using classical test theory we would interpret it by saying that 80% of the variance is true to observed variance and 20% is error to observed variance. In reliability, we don't square the correlation coefficient or use the coefficient of determination. Instead, we simply use the reliability coefficient itself to evaluate the degree of measurement error.

I've seen it suggested in many places that it's a common mistake to square the measurement error in this scenario. However, I do not understand why it is a mistake or precisely what is meant by 'reliable variance'.

Whiston, S. C. (2009). Principles & applications of assessment in counseling. Belmont, CA : Brooks/Cole, Cengage Learning.

up vote 3 down vote accepted

It seems that the textbook is comparing the determination coefficient, $R^2$, from regression analysis to the "reliability coefficient", $r$. These two are in fact comparable as they both are defined as true variance divided by total variance. Especially in regression this is often stated as the amount of explained variance.

More specifically, for a simple regression, the determination coefficient is defined as $$R^2 = \frac{Var(E(Y|X))}{Var(Y)},$$ where $Y$ is the criterion (or dependent variable) and $X$ is the predictor (or independent variable).

Similarly, the reliability of a test is defined in terms of $$r = \frac{Var(E(Y | U))}{Var(Y)},$$ where $Y$ is the test score and $U$ is the person taking the test. Looking at these formulas you can easily see that both the determination coefficient and the "reliability coefficient" are essentially the same. (Note that $E(Y|U)$ is the defintion of the true score in classical test theory, which is why you speak of true variance.)

In a simple regression analysis you can also get the determination coefficient by squaring the correlation coefficient $r$ (Note that this is only true for simple regression, nor for multiple regression). Unfortunately the coefficient for reliability is also denoted by $r$, which apparently causes some trouble, because in a simple regression analysis, you have to square the correlation coefficient to get the amount of explained variance, whereas the reliability of a test already is (at least in principle) the amount of true variance.

Reference

Steyer, R., & Eid, M. (2001). Messen und Testen (2. Aufl.). Berlin: Springer. (This is a german book, maybe someone else can point to a different source...)

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.