2
$\begingroup$

I'm refining a psychometric questionnaire and working to estimate (2PL) IRT parameters for the items in order to score the items. It's a small scale consisting of 5 questions. The MAP and VSS-1 criterion both indicate unidimensionality, as do the high Omega_Heirarchical score and Explained Common Variance of the general factor (both ~.75).

The only problem is that the IRT parameters returned have a disproportionately extremely high score for one of the items (I ran this using the psych package in R but also verified it against the ltm package):

enter image description here

Discrimination parameters:

Q1 0.83

Q2 0.70

Q3 0.86

Q4 6.16

Q5 0.55

Examination of the factor analysis of the polychoric matrices makes it pretty clear why; the loading of Q4 on the trait is 0.99.

Now this is all strictly speaking correct, but the result is that any IRT scoring is hypersensitive to the response on Q4, resulting in extremely variable scores that correspond very poorly when compared to parallel outcomes under classic scoring procedure. Here's a sample illustrating the issue (IRT scores on the X axis and Classic Scores on the Y):

enter image description here

What would you advise doing in such a situation? I suppose I could (artificially) rotate the one-factor solution slightly, but any solution out of this would surely be pretty arbitrary? Alternatively, I could extract a general factor from the bifactor solution, and convert the general factor loadings to item parameters, but this relies on initially extracting 3 oblique factors (which is probably inappropriate given that there are only 5 items). I ran the numbers for the latter solution and the parameters are much more palatable:

$discrimination

Q1 0.78

Q2 0.74

Q3 0.73

Q4 1.78

Q5 0.56

What would be appropriate?

Improve this question
$\endgroup$
3
  • $\begingroup$ What does it look like if you remove this item? $\endgroup$
    – Jeremy Miles
    Nov 24, 2014 at 17:54
  • $\begingroup$ The item parameters for the remaining items return to a fairly manageable level (discrimination parameters for Questions 1,2,3,5 are 0.95,0.69,0.67,0.63 respectively). It seems this is measuring the same trait. Of course, it means I would be discarding my most informative item! $\endgroup$
    – Sue Doh Nimh
    Nov 25, 2014 at 16:13
  • $\begingroup$ It's only most informative at a particular location. Overall, these kinds of items don't add much else to the overall test information. They have a similar interpretation to a 'Heywood case' in standard factor analysis since the slopes tend to $\Inf$ and act like a latent class classifying item (higher than some point or not). Whether that's okay or not really depends on the application. $\endgroup$
    – philchalmers
    Dec 3, 2014 at 7:34

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.