# Dealing with poor fit in an Item Response Theory model

I'm studying an online course with about 3000 students who each took several quizzes and I'm trying to apply Item Response Theory (using the ltm package in R) to model the questions, determine which items are most or least important to retain and what new items may be required, and to provide an alternate way to score and compare student performance.

However, whenever I attempt to model all 16-20 questions on a single quiz together, I discover that the model is a poor fit based on the two- and three-way margins, with many Chi-squared residuals exceeding 10 and some exceeding 20, even with a three-parameter model. I tried two strategies to deal with this.

In one I tried partitioning the questions into smaller groups based on similarity of the tested concepts. However I found in order to get acceptable fit (no more than 2-3 margins with Chi-squared over 3.5), I had to limit partition size to 3-5 items at most. After doing this, I have little insight into exactly what latent variable is being measured in each partition and thus how to interpret low discrimination, or how to assemble the scores for each partition into a test score. I also am not sure how to compare items in different partitions regarding importance.

The other strategy I tried was to use "ltm" to fit to a two-dimensional latent variable space. This produced reasonable Chi-squared residuals (assuming the usual 3.5 cutoff remains reasonable in the 2D case) but my attempts to qualitatively understand what is being measured by each variable, either by looking at item parameters, 3D ICC graphs, or maximum posterior scores, failed. Lacking this understanding, I'm not sure how to combine the two latent variable scores into a single test score.

It may be that the tests I'm working with are just too short and cover too broad of an area to adequately apply IRT to (covering e.g. 2-3 topics over a multiple choice quiz of 16 items). I hope this is not the case. It's also possible I'm being a bit too concerned with fit. I'm new to Item Response Theory and would appreciate any insight on this.

-
IRT won't let you uncover 'good' items to devise a scale easily. First of all, you have to identify the dimensionality of your questionnaire, and apply basic item-level analysis (item difficulty, discriminatory power, item-scale correlation) and test-level analysis (reliability or internal consistency). Second, be careful with $\chi^2$ tests as they will be impacted by large (or low) $N$. Third, adding extra parameters to your model (discrimination, and/or guessing) should be motivated by theory, and not data driven. Could you perhaps tell us more about item content/wording and the examinees? –  chl Aug 13 '12 at 21:22
I can tell you what I know. Test reliability: Cronbach's alpha is 0.8382, which should be good. Point biserial correlation of item scores with total score (with item excluded) range from 0.34 to 0.59, most in the 0.4 to 0.5 range. I'm not sure how to quantify dimensionality, other than that the quizzes seem to cover more than one topic area (for example the one I'm looking at has 8 questions on the Ruby language, 4 on software architecture, and 4 on... various other stuff). A 3-parameter model seemed reasonable since it's a multiple-choice test, but anova shows no improvement over 2-param. –  Derrick Coetzee Aug 13 '12 at 21:47
Also: I thought that item difficulty/discrimination of single items depended on me first getting a good fit, since these are expressed in terms of the latent variable? –  Derrick Coetzee Aug 13 '12 at 21:48
Regarding the examinees: they are self-selected free online course participants drawn from a very wide range of demographics (many nations, diverse ages, work experience, etc.) The large majority are working software professionals with good education. I'm not sure how the nature of the examinees affects the analysis strategy. –  Derrick Coetzee Aug 13 '12 at 21:53
So I can tell you that (1) Cronbach's alpha will just increase when increasing the number of items, and is a measure of internal consistency for unidimensional scale, (2) choosing between 1-, 2- or 3-PL has nothing to do with item format (MCQ or yes/no), and (3) item parameters have nothing to do with goodness-of-fit. First, try to isolate some dimensions of interest with factor analysis or pragmatic considerations (e.g., Ruby vs. language architecture seem like different constructs to me). –  chl Aug 13 '12 at 21:59
show 1 more comment