My background is in machine learning and statistics, but I am relatively new to psychometrics and testing. Nearly all of the literature I've found on item reliability refers to Chronbach's alpha or methods that assume all items are administered to all subjects (and what little I've seen for adaptive or IRT models is either confusing or doesn't seem to apply to my situation).
The situation: I have an IRT-based adaptive test, consisting of tens of thousands of items that are automatically generated and placed on the IRT scale (the item pool is also in flux). I am looking for a way to measure the internal consistency of these items, given thousands of test administrations. In particular, I'm interested in finding a way to identify "problem items" or "outliers" so I can remove them from the item pool, since a few are bound to pop up with so many automatically-generated items.
I am using a 2PL IRT model, but you can sort of think of it as a 1PL model since the slope parameters are not item-specific (there is not enough data to reliably fit these across all items), but rather format-specific (there are a few different item question formats, and there is much more data for this).
Are there reasonable methods for identifying such "problem items" given a set of test-administration logs? The best I can think of is ranking the items by their negative log-likelihood according the IRT model (as a function of the model's final test scores)... but I was hoping to find for something more established/accepted in the field (if there is such a thing)...
EDIT: In light of @philchalmers's comments below, it seems that measuring "item fit/misfit" is perhaps more what I am looking for. Any advice on how to determine item misfit in this situation is also welcome.