I was wondering what makes for a good test information value in IRT, and came across this:

Reliability: Item and test information functions graphically reflect how reliably the individual items and the test as a whole estimate the construct over the entire scale range. Values can be converted into an estimate of reliability (i.e., reliability = 1 − [1 / information]) so that the common rule of thumb of 0.70 to 0.90 for interpreting reliability values corresponds to information of 3.3 to 10.

I get the rule of thumb for traditional (Chronbach’s) alpha values (< 0.7 means the constituent items are not sufficiently correlated; > 0.9 suggests there might be redundancy in the test). But I was curious how to make sense of the “reliability = 1 − [1 / information]” part. (They reference this, but I don't have access to it).

I'm particularly puzzled by the idea of a "ceiling" for test information at 10. For example, say that I have a three-item scale, where each item has a discrimination value of 4, and difficulty values at -0.5, 0, and 0.5, respectively. We would then have this (following DeMars 2010, pp. 14 and 81 for the calculations of P and inf):

# item 1
a1 <- 4 # discrimination value
b1 <- -0.5 # difficulty value
item1 <- data.frame("theta" = seq(-2,2,0.01))
item1$P <- (exp(1.7*a1*(item1$theta-b1)))/(1+exp(1.7*a1*(item1$theta-b1)))
item1$inf <- (1.7^2)*(a1^2)*(1-item1$P)*item1$P

# item 2
a2 <- 4
b2 <- 0
item2 <- data.frame("theta" = seq(-2,2,0.01))
item2$P <- (exp(1.7*a2*(item2$theta-b2)))/(1+exp(1.7*a2*(item2$theta-b2)))
item2$inf <- (1.7^2)*(a2^2)*(1-item2$P)*item2$P

# item 3    
a3 <- 4
b3 <- 0.5
item3 <- data.frame("theta" = seq(-2,2,0.01))
item3$P <- (exp(1.7*a3*(item3$theta-b3)))/(1+exp(1.7*a3*(item3$theta-b3)))
item3$inf <- (1.7^2)*(a3^2)*(1-item3$P)*item3$P

And to plot the ICC and IIC:

par(mfrow = c(1, 2))

# ICC for each item
plot(item1$P~item1$theta, type="l", col="red",
     xlab = expression(theta),
     ylab = expression(paste("P(", theta, ")")))
lines(item2$P~item2$theta, type="l", col="blue")
lines(item3$P~item3$theta, type="l", col="green")
legend(-2, 1, legend=c("1", "2", "3"),
       col=c("red", "blue", "green"), lty=1, cex=0.8)

# IIC for test
test <- data.frame("theta" = seq(-2,2,0.01))
test$inf <- item1$inf + item2$inf + item3$inf
plot(test$inf~test$theta, type="l",
     xlab = expression(theta),
     ylab = expression(paste("Information(", theta, ")")))
abline(h=10, col="red", lty=2)


The peaks in the test IIC (right panel) are above 10. But if we're looking to measure ability around the mean, this seems a perfectly good test (assuming it meets requirements regarding unidimensionality, local independence, etc.). And the concern about redundancy that we would have with an alpha > 0.9 doesn't seem to apply, given the spread of the ICCs around mean ability (left panel).

So, my questions are these:

  1. As for a "ceiling" for test information, 10 seems pretty arbitrary to me, even as a rule of thumb. I take it you want to maximise the information (as given by the test IIC) at the ability level that is relevant to you (e.g., the mean), while using items with difficulty values that are spread across the relevant range, to avoid achieving high information through redundancy, i.e., by having lots of items cluster at one ability value. Or am I missing something?

  2. That still leaves the question: how high of an information value is high enough for a test? It would be great to have a rule of thumb here, e.g., such that anything less than 3.3 is insufficient, as per the quote above. But I struggle to make intuitive sense of that particular value. Does anyone happen to have any insight here?

  • 1
    $\begingroup$ Both item and test information in IRT are conditional upon the underlying trait level $\theta$, so a single statistic analogous to Chronbach's $\alpha$ for a test isn't usually applicable. There's a nice discussion about this in a previous CV question here. Could you clarify whether your two questions concern values of the item or test information? I.e. in asking for rules of thumb, do you mean at the maxima of the item/test information functions? Also, could you post your code for the plots? The right one looks odd. $\endgroup$ – awhug Aug 8 '20 at 7:53
  • $\begingroup$ Thank you, @awhug, including for the link to that discussion - will take a look! To clarify: my questions concern test information, so I do indeed have in mind the maxima in the test information function. I've added code for the plots. I was under the impression that test information is just the sum of item informations at a given $\theta$ (hence, I've used test$inf <- item1$inf + item2$inf + item3$inf), but let me know if I'm wrong about that. $\endgroup$ – kh_one Aug 8 '20 at 8:42
  • 1
    $\begingroup$ You were correct, I just omitted the $D = 1.7$ normal ogive scaling factor trying to recreate your plot. My apologies! Couple of things - First, that $D$ scaling will affect any information rules of thumb. Second, I searched Nunnally (1978) but can't find that 'reliability' formula. Seems strange given it can yield negative values depending on $\theta$! Finally, $a=4$ is pretty high - Baker & Kim (2017, p. 104) report items with $a>1.7$ "result in a high general level of the amount of test information", and $a>2.8$ is unusual. $\endgroup$ – awhug Aug 8 '20 at 12:53
  • $\begingroup$ Thanks so much, @awhug - both for searching the Nunnally book, and also for the Baker & Kim reference. I’ll check the latter out. It sounds, then, like it’s perhaps better to think in terms of ideally wanting discrimination values > 1.7 for items that are spread out over the $\theta$ range of interest, rather than in terms of rules of thumb for the resulting test information. Thanks again - I appreciate your time on this! $\endgroup$ – kh_one Aug 9 '20 at 19:26

Regarding your first question, I believe the “ceiling effects” mentioned by the authors is a reference to restriction of range. Restriction of range (RR) occurs when there is less variance in the sample than in the population due to a selection procedure of some sort - which may lead to attenuated, or underestimated, estimates of reliability. This can happen when items are narrowly focused(e.g., depression inventory focusing only on social interaction). I believe this is what you where alluding to in you question when you mentioned “information redundancy”. I believe you are spot on here, and hopefully my answer adds some context!

Now to your second question. Its probably best to take a look at the inverse of information, the standard error - $SE(\theta)$ = $1/\sqrt{I(\theta)}$, instead.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.