Item reliability or fit in IRT-based adaptive tests?

Question

My background is in machine learning and statistics, but I am relatively new to psychometrics and testing. Nearly all of the literature I've found on item reliability refers to Chronbach's alpha or methods that assume all items are administered to all subjects (and what little I've seen for adaptive or IRT models is either confusing or doesn't seem to apply to my situation).

The situation: I have an IRT-based adaptive test, consisting of tens of thousands of items that are automatically generated and placed on the IRT scale (the item pool is also in flux). I am looking for a way to measure the internal consistency of these items, given thousands of test administrations. In particular, I'm interested in finding a way to identify "problem items" or "outliers" so I can remove them from the item pool, since a few are bound to pop up with so many automatically-generated items.

I am using a 2PL IRT model, but you can sort of think of it as a 1PL model since the slope parameters are not item-specific (there is not enough data to reliably fit these across all items), but rather format-specific (there are a few different item question formats, and there is much more data for this).

Are there reasonable methods for identifying such "problem items" given a set of test-administration logs? The best I can think of is ranking the items by their negative log-likelihood according the IRT model (as a function of the model's final test scores)... but I was hoping to find for something more established/accepted in the field (if there is such a thing)...

EDIT: In light of @philchalmers's comments below, it seems that measuring "item fit/misfit" is perhaps more what I am looking for. Any advice on how to determine item misfit in this situation is also welcome.

Answer 1

Item fit is a pretty active area of research in IRT literature. There are specific models (i.e., Rasch models) that have their own specific item fit statistics, such as the infit and outfit (which may relate to your case, within each packet of equally discriminating items), and more general ones that don't rely on the strict Rasch model format. I tend to prefer the latter, since they work equally well for Rasch items as well.

Look into the S-X2 statistic since I think it is one of the better tests overall, and is fairly cheap to compute for each item (unlike Information matrix based tests). It's also possible to use variants of the M2 statistic for each item, though this gets out of hand for larger tests, or to use some older two-stage methods which requires computing person estimates and creating expected value bins manually to form an approximate $\chi^2$ test (I wouldn't recommend this approach, despite its popularity in the past). Hope that helps.

References

Kang, T. & Chen, Troy, T. (2007). An investigation of the performance of the generalized S-X2 item-fit index for polytomous IRT models. ACT

Maydeu-Olivares, A. & Joe, H. Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 2006, 71, 713-732

Ranger, J. & Kuhn, J.-T. Assessing Fit of Item Response Models Using the Information Matrix Test. Journal of Educational Measurement, 2012, 49, 247-268

Reise, S. P. (1990). A comparison of item- and person-fit methods of assessing model-data fit in IRT. Applied Psychological Measurement, 14, 127-137.

Wright B. D. & Masters, G. N. Rating scale analysis. MESA Press, 1982