Inferring testlet structure in item response theory

Question

Is it possible to infer the testlet structure in a set of items using item response theory?

Specifically, I've created a lot of variations on the story recall task, each variation being scored on 25 details as passed/failed. It is clear from administering that people remember "chunks" of the stories, indicating a clear testlet structure. Thus the "plain IRT" assumption of independent items would not represent the data well. Rather than guessing, I'd like to infer the (probabilistic) testlet-membership of each item from the data.

I'm currently doing Bayesian IRT using a Rasch model and JAGS for sampling. But any thoughts on how this could be done would be appreciated.

Answer 1

Testlet theory shares a very tight relationship to a bifactor model, so yes this can be done in IRT. Testlets basically are formed when there is one construct of interest, however there are 'packets' of inter-item dependencies due to similar content, or whatever else. The usual testlet model, at the item level, gives the items with inter-dependencies equal weights to form each respective composite (whereas in the less restricted bifactor structure, each item slope may be unequal. However, constraining them to be equal in estimation gives the same effect).

If you are looking to explore where these interdependencies are you are basically looking at the problem as an item-level factor analysis. Several approaches are therefore possible, such as limited information factor analysis with polychoric matrix, or multidimensional IRT. Either way, you are looking to recover the dependency pattern empirically rather than modeling them theoretically from an a priori standpoint. If you are instead looking to 'confirm' your hypothesized testlet patter, then both frameworks also provide a confirmatory modelling approach as well. Hope that helps.