I have a data set in which two raters have each rated N samples using a 5-point ordinal rating scale. My primary interest is in whether these two raters make significantly different use of those ratings (i.e. I believe I'm interested in whether the item is subject to differential item functioning, DIF).

There does not seem to be a straightforward way to ask if a single item is subject to DIF; rather, the existing methods I can find (e.g. the R package lordif) take a collection of multiple items (at least five items in the case of lordif) and responses and determine whether any of those items is subject to DIF.

I've fit graded response models to each of the two raters' answers individually to produce item characteristic curve plots and visually the thresholds for use of the five ratings look distinct from one another but I'd like a way to test these for significance.

Can anyone provide guidance on testing single items for DIF? Is this even a meaningful question in the absence of a second 'anchor' item for which no DIF is assumed?

  • $\begingroup$ Yes, you need to use anchor items to have a common scale for latent trait. But isn't your question more about inter-rater variability? How many subjects and items do you have? $\endgroup$ – chl Nov 25 '12 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.