5
$\begingroup$

Reporting 'reliability' in IRT is more to 'informations' obtained and a bit different from a classical way using such as Cronbach alpha and KR20.

Therefore, how to report the test reliability in IRT style? Please help.

$\endgroup$
  • $\begingroup$ In IRT, reliability varies across the range of theta. There isn't one value. $\endgroup$ – Jeremy Miles Jan 9 '15 at 0:52
5
$\begingroup$

In Classical Test Theory observed test scores $X$ could be defined as:

$$X = T + E$$

where $T$ are the true scores and $E$ is an error of measurement. This means that their variance is:

$$\sigma^2_X = \sigma^2_T + \sigma^2_E$$

In this case, reliability could be defined as:

$$ \rho_{xx'} = \frac{\sigma^2_T}{\sigma^2_X} = \frac{\sigma^2_X - \sigma^2_E}{\sigma^2_X} = 1 - \frac{ \sigma^2_E }{ \sigma^2_X } $$

In classical test theory $\sigma^2_E$ error variance and $\sigma^2_X$ observed score variance.

This approach could be transfered into the IRT framework. One approach would be:

$$ \rho_{xx'} = 1 - \frac{ \sigma^2_E }{ \sigma^2_T } $$

where $\sigma^2_T$ is a variance of EAP scores. It is possible since EAP are the estimate of the true ability $\theta$. The problem is that variance of EAP is an under-estimate of the variance of true scores $\sigma^2_X$ (Wu, 2005). The problem is also that with this approach the values of $\rho_{xx'}$ could be negative and we don't want reliability estimate to be negative. In practice, the negative values could suggest that there is something wrong with the model.

Other approach would be to define $\sigma^2_E$ as mean of EAP errors and $\sigma^2_X$ as variance of the true scores, i.e. $\sigma^2_X = \sigma^2_T + \sigma^2_E$ and use simply the CTT formula:

$$ \rho_{xx'} = \frac{\sigma^2_T}{\sigma^2_X} = \frac{\sigma^2_X - \sigma^2_E}{\sigma^2_X} = 1 - \frac{ \sigma^2_E }{ \sigma^2_X } $$

Other possibility for $\sigma^2_E$, as Raju et al. notes, could also be to define it as:

$$\sigma^2_E = E\left[\left( \frac{1}{I_s} \right)^2\right]$$

where $I_s$ is the total test information function for examinee $s$.

The nice fact about defining reliability like this is that (a) it is consistent with CTT, and (b) it is easy to compute.

Other approaches would be using $\sigma^2_E$ or $\sigma^2_T$ solely as they both say something about reliability.

This kind of approach in not commonly used and it is more popular to use information content as in @robin.datadrivers answer since reliability is rather a CTT concept. So some would use for this purpose both the CTT measures (Cronbach $\alpha$) and IRT measures (information content). However, as I mentioned, it is possible to use CTT-like reliability based on IRT.

Below I post an R code for computing reliability given mirt or ltm output:

rel.mirt <- function(x) {
    eap <- mirt::fscores(x, full.scores=T, scores.only=T, full.scores.SE=T)
    e <- mean(eap[, 2]^2)
    s <- var(eap[, 1])
    1-(e/(s+e))
}

rel.ltm <- function(x) {
    eap <- ltm::factor.scores(x, method="EAP")$score.dat
    e <- mean(eap$se.z1^2)
    s <- var(eap$z1)
    1-(e/(s+e))
}

References

Raju, N.S., Price, L.R., Oshima, T.C., & Nering, M.L. (2006). Standardized Conditional SEM: A Case for Conditional Reliability. Applied Psychological Measurement, 30(X), 1-12.

Wang, T., Kolen, M.J., & Harris, D.J. (1997). Conditional Standard Errors, Reliability, and Decision Consistency Performance Levels Using Polytomous IRT. Reliability Issues with Performance Assessments: A Collection of Papers. ACT Research Report Series 97-3, 13-40.

Adams, R.J. (2005). Reliability as a measurement design effect. Studies in Educational Evaluation, 31(2–3), 162–172.

Wu, M. (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31(2–3), 114-128.

$\endgroup$
  • 1
    $\begingroup$ Well for one thing, the reliability formulas themselves are incorrect. The ratio should be $1 - \sigma_E / \sigma_X$, where $X$ is the total variability observed (not just of the scores). So for mirt, you need e <- mean(eap[,2]^2); s <- var(eap[,1]); rxx = 1 - (e / (s + e)) (or equivalently, s / (s + e) for the usual $T / (T + E) = T / X$ ratio. $\endgroup$ – philchalmers Jan 10 '15 at 16:40
  • 1
    $\begingroup$ @philchalmers Yes, if you follow Raju et al., but Wang et al. define $\sigma^2_S$ as simply "variance of this marginal score distribution"... On other hand, using (e / (s + e)) could be better since negative values would be not possible. $\endgroup$ – Tim Jan 10 '15 at 16:59
  • 1
    $\begingroup$ Upvoted for clarity. I think the reasons this topic isn't studied is because IRT offers a reliability approach conditional on the latent trait values, which is a much better conceptualization of how reliability of tests really works. If one is in an IRT framework then working with the item/test information functions is easy, and marginal reliability seems like a step backward. Such is the problem with marginalizing a complex/conditional property to a single aggregate number. $\endgroup$ – philchalmers Jan 11 '15 at 16:49
  • 1
    $\begingroup$ The dropping item criteria with this stat is actually one area where it could be a horrible idea. Imagine dropping an item which measures a rare location in the test (like in a tail), where no other items can measure that location well. You effectively remove all measurement precision from that location, and although the quasi-$\alpha$ would remain about the same the validity/reliability for individuals around that area drops to 0. Better to take the ratios of information functions to see what happens when dropping items. $\endgroup$ – philchalmers Jan 12 '15 at 0:45
  • 1
    $\begingroup$ Yes, of course, because of that I mentioned it as an additional criterion. $\endgroup$ – Tim Jan 12 '15 at 7:19
4
$\begingroup$

To start off, let's look at what we mean by reliability. Reliability is often thought of as how consistent a measure will be across different measurement scenarios, with everything being equal except the occasion (same assessment, same conditions, same people, different day, e.g.). Reliability can also be thought of as the ability to distinguish between two respondents.

One of the key differences between Classical Test Theory (CTT) and Item Response Theory (IRT) is the way it treats the variance of the latent ability ($\theta$). CTT treats the standard error of measurement (SEM) as fixed across the sample:

$SEM = \sigma \sqrt{(1-reliability)}$

where $\sigma$ is the standard deviation of the observed scores. In this context, reliability doesn't change, nor does $\sigma$, so the SEM is the same.(see here for some further explanation).

In IRT, there is a separate standard error of each value for $\theta$. This makes sense because you are estimating person parameters for each ability level, and because each of these are estimates, they have sampling error. This is captured by the standard errors of $\theta$.

These standard errors are very useful in understanding the reliability of your scale, as estimated by an item response model. One useful application is to consider the information content of the scale at different levels of $\theta$. Information here is defined as the inverse of the variance. You can create a nice chart of the Test Information Curve like the following (I'll describe the gray bars in a moment):

enter image description here The R code I used to produce this (using real data I collected):

library(ltm)
plot(fit1, type = "IIC", items = 0, lwd = 2, xlab = "Factor scores",
     main=NA,cex.main = 1.5, cex.lab = 1.3, cex.axis = 1.1)

This shows you how the information content in the scale changes at different places. Information is higher where you have more items with estimated difficulty parameters. Obviously, this is going to always be true for the tails, where we have less information in the various observed items to reliability differentiate between respondents.

You can look at the information content in specific ranges (again with the ltm package). Let's say for my test I want to see the percent of information between $\theta$ between -2 and 2:

> information(fit1,range=c(-2,2))

Call:
grm(data = dt)

Total Information = 107.08
Information in (-2, 2) = 77.79 (72.64%)
Based on all the items

Here, 72% of all information is between those values. It can get interesting if you use values of $\theta$ that have meaning (these were selected totally arbitrarily).

One interesting application is, let's say you've created a cut score using a common standard setting method, like Angoff. That is often done by creating an observed score cut score - we know that observed scores do not typically perfectly align with $\theta$ values, particularly for an item response model with more than 1 parameter. One thing you can do is take the range of $\theta$ values for all respondents with an observed cut score, and look at the information content for that score. You can plot that onto the test information curve and see how well your cut score aligns with the peaks in the curve. That's what the gray bars are on my plot - they correspond to the $\theta$ values for two cut scores we created, which were created based on observed scores, not $\theta$ values (of course, if you use a method based on IRT scores, such as the Bookmark method, this would turn out different).

You can also produce a single index of reliability in IRT: person-separation and item-separation reliability. You get one for items and persons, because you get person ability and item difficulty measures from the model. This is a good quick description of the differences. Search the WINSTEPS help file for the formula (I think it's in there). The classic Rasch reference by Bond and Fox I believe has a more detailed description of it. I'm not sure how common these are outside of Rasch modeling.

$\endgroup$
  • $\begingroup$ Did either of us answer your question? If so please mark as answered. If not maybe we can give more details. $\endgroup$ – robin.datadrivers Jan 10 '15 at 1:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.