I'm studying an online course with about 3000 students who each took several quizzes and I'm trying to apply Item Response Theory (using the ltm package in R) to model the questions, determine which items are most or least important to retain and what new items may be required, and to provide an alternate way to score and compare student performance.
However, whenever I attempt to model all 16-20 questions on a single quiz together, I discover that the model is a poor fit based on the two- and three-way margins, with many Chi-squared residuals exceeding 10 and some exceeding 20, even with a three-parameter model. I tried two strategies to deal with this.
In one I tried partitioning the questions into smaller groups based on similarity of the tested concepts. However I found in order to get acceptable fit (no more than 2-3 margins with Chi-squared over 3.5), I had to limit partition size to 3-5 items at most. After doing this, I have little insight into exactly what latent variable is being measured in each partition and thus how to interpret low discrimination, or how to assemble the scores for each partition into a test score. I also am not sure how to compare items in different partitions regarding importance.
The other strategy I tried was to use "ltm" to fit to a two-dimensional latent variable space. This produced reasonable Chi-squared residuals (assuming the usual 3.5 cutoff remains reasonable in the 2D case) but my attempts to qualitatively understand what is being measured by each variable, either by looking at item parameters, 3D ICC graphs, or maximum posterior scores, failed. Lacking this understanding, I'm not sure how to combine the two latent variable scores into a single test score.
It may be that the tests I'm working with are just too short and cover too broad of an area to adequately apply IRT to (covering e.g. 2-3 topics over a multiple choice quiz of 16 items). I hope this is not the case. It's also possible I'm being a bit too concerned with fit. I'm new to Item Response Theory and would appreciate any insight on this.