Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

How can internal consistency reliability of a test and of individual test items be quantified in Item Response Theory models? I know I can resort to Classical Test Theory, Cronbach's alpha, and other measures, but is there a way to characterize reliability within IRT?

share|improve this question
add comment

1 Answer 1

up vote 2 down vote accepted

You can compute test information curves from your IRT parameter estimates. These curves give you the precision of the test at each $\theta$ of the latent trait. The information $I$ can be transformed into the standard error of estimate $SEE$, which is a direct estimate of the reliability of the test at that $\theta$: $SEE = 1 / \sqrt{I}$.

The metric of the test information can also be converted to a traditional reliability metric expressed by a correlation coefficient (Thissen, 2000): $Rel = 1 - (1/I)$. Here are the conversions from a set of TICs to correlational reliability estimates:

# following Thissen, 2000:
TIC <- seq(1, 12, by=1)
round((rel <- data.frame(TIC, SEE=sqrt(1/TIC), REL=1-1/TIC)), 2)

TIC  SEE  REL
  1 1.00 0.00
  2 0.71 0.50
  3 0.58 0.67
  4 0.50 0.75
  5 0.45 0.80
  6 0.41 0.83
  7 0.38 0.86
  8 0.35 0.88
  9 0.33 0.89
 10 0.32 0.90
 11 0.30 0.91
 12 0.29 0.92

For example, a TIC > 5 corresponds to a reliability > .80.


Thissen, D. (2000). Reliability and measurement precision. In H. Wainer (Ed.), Computerized adaptive testing: A primer (2nd ed., pp. 159–184). Lawrence Erlbaum Associates Publishers.

share|improve this answer
    
Thanks for your answer; I've asked a follow-up stats.stackexchange.com/questions/43405/item-information-in-irt –  Jack Tanner Nov 12 '12 at 12:58
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.