6
$\begingroup$

My background is in machine learning and statistics, but I am relatively new to psychometrics and testing. Nearly all of the literature I've found on item reliability refers to Chronbach's alpha or methods that assume all items are administered to all subjects (and what little I've seen for adaptive or IRT models is either confusing or doesn't seem to apply to my situation).

The situation: I have an IRT-based adaptive test, consisting of tens of thousands of items that are automatically generated and placed on the IRT scale (the item pool is also in flux). I am looking for a way to measure the internal consistency of these items, given thousands of test administrations. In particular, I'm interested in finding a way to identify "problem items" or "outliers" so I can remove them from the item pool, since a few are bound to pop up with so many automatically-generated items.

I am using a 2PL IRT model, but you can sort of think of it as a 1PL model since the slope parameters are not item-specific (there is not enough data to reliably fit these across all items), but rather format-specific (there are a few different item question formats, and there is much more data for this).

Are there reasonable methods for identifying such "problem items" given a set of test-administration logs? The best I can think of is ranking the items by their negative log-likelihood according the IRT model (as a function of the model's final test scores)... but I was hoping to find for something more established/accepted in the field (if there is such a thing)...

EDIT: In light of @philchalmers's comments below, it seems that measuring "item fit/misfit" is perhaps more what I am looking for. Any advice on how to determine item misfit in this situation is also welcome.

$\endgroup$
  • $\begingroup$ It sounds like you are looking for item fit statistics for detecting potential misfit. Is this what you are after? The design you described seems a bit odd, but certainly possible. How many participants do you have to work with? $\endgroup$ – philchalmers Apr 14 '14 at 23:51
  • 1
    $\begingroup$ Just a note, reliability is intimately connected to the slope parameters in IRT (the larger the slope, the more the item correlates with your trait...and with lots of these the reliability overall tends to increase overall). Therefore if you are constraining them by design you aren't really looking for low reliability items. This is why I'm pretty sure you want item fit indices rather than reliability type measures. $\endgroup$ – philchalmers Apr 15 '14 at 1:40
  • $\begingroup$ I have about 600 tests administered, with about 20 items per test. 6000 of the items have been used, 1.8 times each on average. I see what you are saying about the relationship between slopes and reliability (also: stats.stackexchange.com/questions/43378/…). I'll update the question above. $\endgroup$ – burr Apr 15 '14 at 18:27
3
$\begingroup$

Item fit is a pretty active area of research in IRT literature. There are specific models (i.e., Rasch models) that have their own specific item fit statistics, such as the infit and outfit (which may relate to your case, within each packet of equally discriminating items), and more general ones that don't rely on the strict Rasch model format. I tend to prefer the latter, since they work equally well for Rasch items as well.

Look into the S-X2 statistic since I think it is one of the better tests overall, and is fairly cheap to compute for each item (unlike Information matrix based tests). It's also possible to use variants of the M2 statistic for each item, though this gets out of hand for larger tests, or to use some older two-stage methods which requires computing person estimates and creating expected value bins manually to form an approximate $\chi^2$ test (I wouldn't recommend this approach, despite its popularity in the past). Hope that helps.

References

Kang, T. & Chen, Troy, T. (2007). An investigation of the performance of the generalized S-X2 item-fit index for polytomous IRT models. ACT

Maydeu-Olivares, A. & Joe, H. Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 2006, 71, 713-732

Ranger, J. & Kuhn, J.-T. Assessing Fit of Item Response Models Using the Information Matrix Test. Journal of Educational Measurement, 2012, 49, 247-268

Reise, S. P. (1990). A comparison of item- and person-fit methods of assessing model-data fit in IRT. Applied Psychological Measurement, 14, 127-137.

Wright B. D. & Masters, G. N. Rating scale analysis. MESA Press, 1982

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.