0
$\begingroup$

I am in a situation in which I want to investigate whether test items of a questionnaire appear to have DIF.

The items are polytomous and ranges from 1 to 5, with 5 being the most positive answer. I'm a rookie in the area of psychometrics, but I've tried to read the literature about the various methods of how to detect DIF. Based on what I've read, I decided to use the approach outlined below:

1) Using all items of the questionnaire, I estimate the Graded Response Model and obtained estimates of the latent trait parameter, $\hat{\theta}$, for all individuals.

2) For each item, run three ordered logistic regressions. The first model includes only the trait estimate as covariate, the second model includes also a group-dummy, and finally the third model also features a interaction between the trait estimate and the group-dummy.

3) An item is considered to show uniform-DIF if the LR-test between model 1 and 2 is significant; nonuniform-DIF if the LR-test between model 2 and 3 is significant; and overall-DIF if model 3 significantly outperforms model 1 in terms of log-likelihood.

Basically, what I would like to know, is whether this approach is valid? Does it make any sense?

Improve this question
$\endgroup$
6
  • $\begingroup$ could you define what is Differential Item Functioning (or add a link to a definition)? $\endgroup$
    – Antoine
    Commented Oct 2, 2015 at 10:03
  • $\begingroup$ My "definition" goes as follows: An item is said to exhibit DIF if individuals from different groups respond differently to the item, given that their trait level is identical. $\endgroup$
    – Nomis
    Commented Oct 2, 2015 at 13:03
  • $\begingroup$ Do anyone have an opinion about the approach outlined above? $\endgroup$
    – Nomis
    Commented Oct 9, 2015 at 13:20
  • $\begingroup$ Seems rather cumbersome if you ask me, better methods exist for these kinds of data that don't require two-stage approximations and don't initially pool the groups. Also, this approach doesn't account for the (un)-reliability of the $\theta$ estimates, so it will be very dependent on the initial test length. $\endgroup$
    – philchalmers
    Commented Oct 12, 2015 at 11:03
  • $\begingroup$ Thanks for your answer. I recently came across the R-package "lordif". The package is supposed to be able to decect various type of DIF, using different criterias. Does anyone have sort of experience with this package? If this is the case, how do you decide which one of three criterias Chisqr, R2, or change in coefficients, that you use in application? $\endgroup$
    – Nomis
    Commented Oct 15, 2015 at 8:57

1 Answer 1

Reset to default
1
$\begingroup$

Your approach is sound. You did not mention this, but you will need to assess unidimensionality of the response using a factor analysis. Basically, if the test is multidimensional, the item characteristic slopes may end up being of opposite sign which violates the assumption that the probability of greater sum score response is monotonic in ability.

It's slightly more complicated than your explanation. lordif does employ graded response models, which is a hybrid of latent variable modeling and proportional odds models. Just like latent variable modeling uses multivariate normal likelihood and expectation-maximization to model (in a missing data framework) the relationship between multiple exogenous variables and a unifying endogenous variable, lordif does the same with polytomous exogenous variables and a single continuous endogenous variable having a logistic probability model.

Proportional odds models arise from maximum likelihood of logistic RVs thresholded to assume ordinal categories. Just as with normal ICC curves, the logistic slope of this model measures "difficulty" of items. Also just as before, statistically significant differences in thresholds are taken to be evidence of uniform DIF but differences in logistic slopes are evidence of non-uniform DIF.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.