I was wondering what makes for a good test information value in IRT, and came across this:
Reliability: Item and test information functions graphically reflect how reliably the individual items and the test as a whole estimate the construct over the entire scale range. Values can be converted into an estimate of reliability (i.e., reliability = 1 − [1 / information]) so that the common rule of thumb of 0.70 to 0.90 for interpreting reliability values corresponds to information of 3.3 to 10.
I get the rule of thumb for traditional (Chronbach’s) alpha values (< 0.7 means the constituent items are not sufficiently correlated; > 0.9 suggests there might be redundancy in the test). But I was curious how to make sense of the “reliability = 1 − [1 / information]” part. (They reference this, but I don't have access to it).
I'm particularly puzzled by the idea of a "ceiling" for test information at 10. For example, say that I have a three-item scale, where each item has a discrimination value of 4, and difficulty values at -0.5, 0, and 0.5, respectively. We would then have this (following DeMars 2010, pp. 14 and 81 for the calculations of P and inf):
# item 1
a1 <- 4 # discrimination value
b1 <- -0.5 # difficulty value
item1 <- data.frame("theta" = seq(-2,2,0.01))
item1$P <- (exp(1.7*a1*(item1$theta-b1)))/(1+exp(1.7*a1*(item1$theta-b1)))
item1$inf <- (1.7^2)*(a1^2)*(1-item1$P)*item1$P
# item 2
a2 <- 4
b2 <- 0
item2 <- data.frame("theta" = seq(-2,2,0.01))
item2$P <- (exp(1.7*a2*(item2$theta-b2)))/(1+exp(1.7*a2*(item2$theta-b2)))
item2$inf <- (1.7^2)*(a2^2)*(1-item2$P)*item2$P
# item 3
a3 <- 4
b3 <- 0.5
item3 <- data.frame("theta" = seq(-2,2,0.01))
item3$P <- (exp(1.7*a3*(item3$theta-b3)))/(1+exp(1.7*a3*(item3$theta-b3)))
item3$inf <- (1.7^2)*(a3^2)*(1-item3$P)*item3$P
And to plot the ICC and IIC:
par(mfrow = c(1, 2))
# ICC for each item
plot(item1$P~item1$theta, type="l", col="red",
xlab = expression(theta),
ylab = expression(paste("P(", theta, ")")))
lines(item2$P~item2$theta, type="l", col="blue")
lines(item3$P~item3$theta, type="l", col="green")
legend(-2, 1, legend=c("1", "2", "3"),
col=c("red", "blue", "green"), lty=1, cex=0.8)
# IIC for test
test <- data.frame("theta" = seq(-2,2,0.01))
test$inf <- item1$inf + item2$inf + item3$inf
plot(test$inf~test$theta, type="l",
xlab = expression(theta),
ylab = expression(paste("Information(", theta, ")")))
abline(h=10, col="red", lty=2)
The peaks in the test IIC (right panel) are above 10. But if we're looking to measure ability around the mean, this seems a perfectly good test (assuming it meets requirements regarding unidimensionality, local independence, etc.). And the concern about redundancy that we would have with an alpha > 0.9 doesn't seem to apply, given the spread of the ICCs around mean ability (left panel).
So, my questions are these:
As for a "ceiling" for test information, 10 seems pretty arbitrary to me, even as a rule of thumb. I take it you want to maximise the information (as given by the test IIC) at the ability level that is relevant to you (e.g., the mean), while using items with difficulty values that are spread across the relevant range, to avoid achieving high information through redundancy, i.e., by having lots of items cluster at one ability value. Or am I missing something?
That still leaves the question: how high of an information value is high enough for a test? It would be great to have a rule of thumb here, e.g., such that anything less than 3.3 is insufficient, as per the quote above. But I struggle to make intuitive sense of that particular value. Does anyone happen to have any insight here?
test$inf <- item1$inf + item2$inf + item3$inf), but let me know if I'm wrong about that. $\endgroup$ – kh_one Aug 8 '20 at 8:42