Reliability in IRT Style

Question

Reporting 'reliability' in IRT is more to 'informations' obtained and a bit different from a classical way using such as Cronbach alpha and KR20.

Therefore, how to report the test reliability in IRT style? Please help.

Answer 1

In Classical Test Theory observed test scores $X$ could be defined as:

$$X = T + E$$

where $T$ are the true scores and $E$ is an error of measurement. This means that their variance is:

$$\sigma^2_X = \sigma^2_T + \sigma^2_E$$

In this case, reliability could be defined as:

$$ \rho_{xx'} = \frac{\sigma^2_T}{\sigma^2_X} = \frac{\sigma^2_X - \sigma^2_E}{\sigma^2_X} = 1 - \frac{ \sigma^2_E }{ \sigma^2_X } $$

In classical test theory $\sigma^2_E$ error variance and $\sigma^2_X$ observed score variance.

This approach could be transfered into the IRT framework. One approach would be:

$$ \rho_{xx'} = 1 - \frac{ \sigma^2_E }{ \sigma^2_T } $$

where $\sigma^2_T$ is a variance of EAP scores. It is possible since EAP are the estimate of the true ability $\theta$. The problem is that variance of EAP is an under-estimate of the variance of true scores $\sigma^2_X$ (Wu, 2005). The problem is also that with this approach the values of $\rho_{xx'}$ could be negative and we don't want reliability estimate to be negative. In practice, the negative values could suggest that there is something wrong with the model.

Other approach would be to define $\sigma^2_E$ as mean of EAP errors and $\sigma^2_X$ as variance of the true scores, i.e. $\sigma^2_X = \sigma^2_T + \sigma^2_E$ and use simply the CTT formula:

$$ \rho_{xx'} = \frac{\sigma^2_T}{\sigma^2_X} = \frac{\sigma^2_X - \sigma^2_E}{\sigma^2_X} = 1 - \frac{ \sigma^2_E }{ \sigma^2_X } $$

Other possibility for $\sigma^2_E$, as Raju et al. notes, could also be to define it as:

$$\sigma^2_E = E\left[\left( \frac{1}{I_s} \right)^2\right]$$

where $I_s$ is the total test information function for examinee $s$.

The nice fact about defining reliability like this is that (a) it is consistent with CTT, and (b) it is easy to compute.

Other approaches would be using $\sigma^2_E$ or $\sigma^2_T$ solely as they both say something about reliability.

This kind of approach in not commonly used and it is more popular to use information content as in @robin.datadrivers answer since reliability is rather a CTT concept. So some would use for this purpose both the CTT measures (Cronbach $\alpha$) and IRT measures (information content). However, as I mentioned, it is possible to use CTT-like reliability based on IRT.

Below I post an R code for computing reliability given mirt or ltm output:

rel.mirt <- function(x) {
    eap <- mirt::fscores(x, full.scores=T, scores.only=T, full.scores.SE=T)
    e <- mean(eap[, 2]^2)
    s <- var(eap[, 1])
    1-(e/(s+e))
}

rel.ltm <- function(x) {
    eap <- ltm::factor.scores(x, method="EAP")$score.dat
    e <- mean(eap$se.z1^2)
    s <- var(eap$z1)
    1-(e/(s+e))
}

References

Raju, N.S., Price, L.R., Oshima, T.C., & Nering, M.L. (2006). Standardized Conditional SEM: A Case for Conditional Reliability. Applied Psychological Measurement, 30(X), 1-12.

Wang, T., Kolen, M.J., & Harris, D.J. (1997). Conditional Standard Errors, Reliability, and Decision Consistency Performance Levels Using Polytomous IRT. Reliability Issues with Performance Assessments: A Collection of Papers. ACT Research Report Series 97-3, 13-40.

Adams, R.J. (2005). Reliability as a measurement design effect. Studies in Educational Evaluation, 31(2–3), 162–172.

Wu, M. (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31(2–3), 114-128.

Answer 2

To start off, let's look at what we mean by reliability. Reliability is often thought of as how consistent a measure will be across different measurement scenarios, with everything being equal except the occasion (same assessment, same conditions, same people, different day, e.g.). Reliability can also be thought of as the ability to distinguish between two respondents.

One of the key differences between Classical Test Theory (CTT) and Item Response Theory (IRT) is the way it treats the variance of the latent ability ($\theta$). CTT treats the standard error of measurement (SEM) as fixed across the sample:

$SEM = \sigma \sqrt{(1-reliability)}$

where $\sigma$ is the standard deviation of the observed scores. In this context, reliability doesn't change, nor does $\sigma$, so the SEM is the same.(see here for some further explanation).

In IRT, there is a separate standard error of each value for $\theta$. This makes sense because you are estimating person parameters for each ability level, and because each of these are estimates, they have sampling error. This is captured by the standard errors of $\theta$.

These standard errors are very useful in understanding the reliability of your scale, as estimated by an item response model. One useful application is to consider the information content of the scale at different levels of $\theta$. Information here is defined as the inverse of the variance. You can create a nice chart of the Test Information Curve like the following (I'll describe the gray bars in a moment):

The R code I used to produce this (using real data I collected):

library(ltm)
plot(fit1, type = "IIC", items = 0, lwd = 2, xlab = "Factor scores",
     main=NA,cex.main = 1.5, cex.lab = 1.3, cex.axis = 1.1)

This shows you how the information content in the scale changes at different places. Information is higher where you have more items with estimated difficulty parameters. Obviously, this is going to always be true for the tails, where we have less information in the various observed items to reliability differentiate between respondents.

You can look at the information content in specific ranges (again with the ltm package). Let's say for my test I want to see the percent of information between $\theta$ between -2 and 2:

> information(fit1,range=c(-2,2))

Call:
grm(data = dt)

Total Information = 107.08
Information in (-2, 2) = 77.79 (72.64%)
Based on all the items

Here, 72% of all information is between those values. It can get interesting if you use values of $\theta$ that have meaning (these were selected totally arbitrarily).

One interesting application is, let's say you've created a cut score using a common standard setting method, like Angoff. That is often done by creating an observed score cut score - we know that observed scores do not typically perfectly align with $\theta$ values, particularly for an item response model with more than 1 parameter. One thing you can do is take the range of $\theta$ values for all respondents with an observed cut score, and look at the information content for that score. You can plot that onto the test information curve and see how well your cut score aligns with the peaks in the curve. That's what the gray bars are on my plot - they correspond to the $\theta$ values for two cut scores we created, which were created based on observed scores, not $\theta$ values (of course, if you use a method based on IRT scores, such as the Bookmark method, this would turn out different).

You can also produce a single index of reliability in IRT: person-separation and item-separation reliability. You get one for items and persons, because you get person ability and item difficulty measures from the model. This is a good quick description of the differences. Search the WINSTEPS help file for the formula (I think it's in there). The classic Rasch reference by Bond and Fox I believe has a more detailed description of it. I'm not sure how common these are outside of Rasch modeling.