How is the Spearman-Brown prophecy formula affected by questions of differing difficulties?

How are the results of the Spearman-Brown prophecy formula affected by having test questions of differing difficulties or raters who are easy or hard graders. One respected text says the S-B is affected, but does not give details. (See quote below.)

Guion, R. M (2011). Assessment, Measurement, And Prediction For Personnel Decisions, 2nd edition. Pg 477

"Reliability can be increased by pooling raters, using the Spearman-Brown equation. ... If the reliability of a single rating is .50, then the reliability of two, four, or six parallel ratings will be approximately .67, .80, and .86, respectively" (Houston, Raymond, & Svec, 1991, p. 409). I like this quotation because the word approximately recognizes that statistical estimates are "on the average" statements of what might be expected if all goes as assumed. Beyond that, the operative word is parallel. Averaging ratings (or using Spearman-Brown) if one rater is, for example, systematically lenient, simply does not fit the assumption. If essays are each rated by two raters, one more lenient than the other, the problem is like that of using two multiple choice tests of unequal difficulty (nonparallel forms). Scores based on different (unequated) test forms are not comparable. So it is with mixing lenient and difficult raters; the reliability of the pooled ratings is incorrectly estimated by the Spearman-Brown equation of classical test theory. Matters are worse if each judge defines a construct a bit differently."

• I think the problem with seeking a credible source is that the answer comes from test theory, and it's kind of obvious if you understand the underlying theory, and in particular the limitations of our ability to assess reliability. That's why Guion doesn't bother to explain it. But good luck in your search anyway - perhaps someone, somewhere knows of a better explanation. – Jeremy Miles May 19 '14 at 3:48

Although I feel a little sheepish contradicting both a "respected text" as well as another CV user, it seems to me that the Spearman-Brown formula is not affected by having items of differing difficulty. To be sure, the Spearman-Brown formula is usually derived under the assumption that we have parallel items, which implies (among other things) that the items have equal difficulty. But it turns out this assumption isn't necessary; it can be relaxed to allow unequal difficulties, and the Spearman-Brown formula will still hold. I demonstrate this below.

Recall that in classical test theory, a measurement $X$ is assumed to be the sum of a "true score" component $T$ and an error component $E$, that is, $$X = T + E,$$ with $T$ and $E$ uncorrelated. The assumption of parallel items is that all items have the same true scores, differing only in their error components, although these are assumed to have equal variance. In symbols, for any pair of items $X$ and $X'$, $$T=T' \\\textrm{var}(E)=\textrm{var}(E').$$ Let's see what happens when we relax the first assumption, such that the items might differ in their difficulties, and then derive the reliability of a total test score under these new assumptions. Specifically, assume that the true scores might differ by an additive constant, but the errors still have the same variance. In symbols, $$T=T' + c' \\\textrm{var}(E)=\textrm{var}(E').$$ Any differences in difficulty are captured by the additive constant. For example, if $c'>0$, then scores on $X$ tend to be higher than scores on $X'$, so that $X$ is "easier" than $X'$. We might call these essentially parallel items, in analogy to the assumption of "essential tau-equivalence" which relaxes the tau-equivalent model in a similar way.

Now to derive the reliability of a test form of such items. Consider a test consisting of $k$ essentially parallel items, the sum of which give the test score. Reliability is, by definition, the ratio of true score variance to observed score variance. For the reliability of the individual items, it follows from the definition of essential parallelism that they have the same reliability, which we denote with $\rho = \sigma^2_T/(\sigma^2_T+\sigma^2_E)$, with $\sigma^2_T$ being the true score variance and $\sigma^2_E$ the error variance. For the reliability of the total test score, we first examine the variance of the total test score, which is \begin{aligned} \textrm{var}(\sum_{i=1}^kT_i + E_i) &= \textrm{var}(\sum_{i=1}^kT + c_i + E_i) \\ &= k^2\sigma^2_T + k\sigma^2_E, \end{aligned} where $T$ (no subscript) is any arbitrary true score that all the items' true scores can be shifted to via their constant terms, $\sigma^2_T$ is the true score variance, and $\sigma^2_E$ is the error variance. Notice that the constant terms drop out! This is key. So then the reliability of the total test score is \begin{aligned} \frac{k^2\sigma^2_T}{k^2\sigma^2_T + k\sigma^2_E} &= \frac{k\sigma^2_T}{k\sigma^2_T + \sigma^2_X - \sigma^2_T} \\&= \frac{k\rho}{1+(k-1)\rho}, \end{aligned} which is just the classical Spearman-Brown formula, unaltered. What this shows is that even when varying the "difficulty" of the items, defined as their mean scores, the Spearman-Brown formula still holds.

@JeremyMiles raises some interesting and important points about what can happen when we increase test length "in the real world," but at least according to the idealized assumptions of classical test theory, variations in item difficulty don't matter for the reliability of a test form (in stark contrast to the assumptions of modern Item Response Theory!). This same basic line of reasoning is also why we usually speak of essential tau-equivalence rather than tau-equivalence, because most all of the important results hold for the more lenient case where item difficulties (i.e., means) can differ.

• Yes, good point. What I wrote doesn't necessarily hold. – Jeremy Miles May 20 '14 at 13:53

It's not easy to say.

First, the Spearman-Brown assumes that test items (or raters) are randomly sampled from a population of test items (or raters). This is never really true, particularly of tests, because making up more items is hard, and it's likely that you'll use the better items to start with - then you'll find that the test needs to be longer, so you'll 'scrape the barrel' for items.

Second, items vary in their reliability, and the reliability isn't necessarily related to the difficulty (if it help, think of the slope and intercept of the item characteristic curve in item response theory). However, calculation of reliability (say, Cronbach's alpha, which is a form of intra-class correlation) assume that reliabilities are all equal (they assume an essential tau-equivalent measurement model - that is, that the unstandardized reliabilities of each item are all equal). That's almost certainly wrong. Adding items might go up, might go down. It depends on the items.

Here's another way to think of it. I randomly select a sample from a population, and calculate the mean and standard error of the mean. That mean will be an unbiased estimator of the population mean. Then I increase the size of my sample - the expected value of the mean is the same, but it's unlikely that it will actually be the same - it will almost certainly go up or down. Just as I expect the standard error to get smaller, but the amount it shrinks will not be consistent (and it's not impossible for the standard error to get larger.)

• Does the S-B formula give the minimum, maximum, or some intermediate value for the expected reliability? Also, since reliabilties are calculated in terms of correlations, why do easy/hard items or raters have any effect? – Joel W. May 9 '14 at 22:17
• The S-B formula gives the expected reliability. It could be higher, or lower than that. One problem is that there is more than one way to calculate reliability, and the assumptions that they make are rarely satisfied. The whole thing is kind of rooted in classical test theory - item response theory is a more modern way to think about measurement, and it makes more sense a lot of the time, for example, the reliability of a test is not the same for each person in IRT. – Jeremy Miles May 9 '14 at 22:44
• If a question is very hard, or very easy, it might affect the correlation. E.g. "7 * 11" might be a reliable question for 3rd grade, but for math undergraduates, it's not. – Jeremy Miles May 9 '14 at 22:45
• <the test needs to be longer, so you'll 'scrape the barrel' for items. Clearly you have had real world experience putting together tests. – Joel W. May 23 '14 at 19:41