6
$\begingroup$

How can internal consistency reliability of a test and of individual test items be quantified in Item Response Theory models? I know I can resort to Classical Test Theory, Cronbach's alpha, and other measures, but is there a way to characterize reliability within IRT?

$\endgroup$
4
$\begingroup$

You can compute test information curves from your IRT parameter estimates. These curves give you the precision of the test at each $\theta$ of the latent trait. The information $I$ can be transformed into the standard error of estimate $SEE$, which is a direct estimate of the reliability of the test at that $\theta$: $SEE = 1 / \sqrt{I}$.

The metric of the test information can also be converted to a traditional reliability metric expressed by a correlation coefficient (Thissen, 2000): $Rel = 1 - (1/I)$. Here are the conversions from a set of TICs to correlational reliability estimates:

# following Thissen, 2000:
TIC <- seq(1, 12, by=1)
round((rel <- data.frame(TIC, SEE=sqrt(1/TIC), REL=1-1/TIC)), 2)

TIC  SEE  REL
  1 1.00 0.00
  2 0.71 0.50
  3 0.58 0.67
  4 0.50 0.75
  5 0.45 0.80
  6 0.41 0.83
  7 0.38 0.86
  8 0.35 0.88
  9 0.33 0.89
 10 0.32 0.90
 11 0.30 0.91
 12 0.29 0.92

For example, a TIC > 5 corresponds to a reliability > .80.


Thissen, D. (2000). Reliability and measurement precision. In H. Wainer (Ed.), Computerized adaptive testing: A primer (2nd ed., pp. 159–184). Lawrence Erlbaum Associates Publishers.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.