I'm refining a psychometric questionnaire and working to estimate (2PL) IRT parameters for the items in order to score the items. It's a small scale consisting of 5 questions. The MAP and VSS-1 criterion both indicate unidimensionality, as do the high Omega_Heirarchical score and Explained Common Variance of the general factor (both ~.75).
The only problem is that the IRT parameters returned have a disproportionately extremely high score for one of the items (I ran this using the psych package in R but also verified it against the ltm package):
Examination of the factor analysis of the polychoric matrices makes it pretty clear why; the loading of Q4 on the trait is 0.99.
Now this is all strictly speaking correct, but the result is that any IRT scoring is hypersensitive to the response on Q4, resulting in extremely variable scores that correspond very poorly when compared to parallel outcomes under classic scoring procedure. Here's a sample illustrating the issue (IRT scores on the X axis and Classic Scores on the Y):
What would you advise doing in such a situation? I suppose I could (artificially) rotate the one-factor solution slightly, but any solution out of this would surely be pretty arbitrary? Alternatively, I could extract a general factor from the bifactor solution, and convert the general factor loadings to item parameters, but this relies on initially extracting 3 oblique factors (which is probably inappropriate given that there are only 5 items). I ran the numbers for the latter solution and the parameters are much more palatable:
What would be appropriate?