5
$\begingroup$

Generally when I do IRT modelling (using R software, 2PL model, ltm package) I remove records of students who have scored either 0 or 100%. The logic being that we do not have enough information about the ability of these students.

But this way I cannot allocate any score to students who scored 0 or 100%

I have tried running the model with such records also. The model converges. But I am not sure if it is correct to do it. I am getting some values for zero and 100% scorers but I am not sure how it is being calculated in the ltm package.

$\endgroup$
2
  • 1
    $\begingroup$ The R code is open source and you can look at it. $\endgroup$
    – Nick Cox
    Jul 30, 2013 at 13:49
  • $\begingroup$ What were the size of your parameter estimates when students who scored 0 or 100% were included in the model? $\endgroup$
    – Jhaltiga68
    Mar 21, 2017 at 21:43

2 Answers 2

6
$\begingroup$

In general this is a bad idea. ltm uses marginal maximum likelihood (MML) estimation, and therefore zero response vectors are still used in estimation and give information about the 'difficulty' of an item, unlike joint maximum likelihood (JML) is which case these need to be removed (like in the program LOGIT).

The same thing can be said about removing rows with missing data. While it shouldn't introduce bias if the responses containing missing data are missing at random (MAR), it will decrease the precision of recovering population parameters. However, removing extreme responses isn't removing by a MAR scheme (clearly), so bias is introduced and parameters are estimated with less precision.

Here's a quick simulation (maybe 3 minutes or so...and could easily be run in parallel by passing parallel = TRUE to runSimulation()) where you estimate the model with and without extreme response patterns, and you can see the RMSE and bias statistic are larger without these patterns due to the loss of information and introduced bias.

library(SimDesign)

Design <- createDesign(nitems=10, N=1000)

Generate <- function(condition, fixed_objects = NULL) {
    Attach(condition)
    a <- matrix(rlnorm(nitems, .2, .2))
    d <- matrix(rnorm(nitems))
    resp <- simdata(a,d, N, itemtype='2PL')
    list(resp=resp, a=a, d=d)
}

Analyse <- function(condition, dat, fixed_objects = NULL) {
    Attach(dat)
    modwith <- mirt(resp, 1, verbose = FALSE)
    sums <- rowSums(resp)
    modwithout <- mirt(resp[sums != 0 & sums != condition$nitems, ], 1, 
                       verbose = FALSE)
    cfs1 <- coef(modwith, simplify=TRUE)
    cfs2 <- coef(modwithout, simplify=TRUE)
    ret <- c(biaswith = bias(cbind(a,d) - cfs1$items[,1:2]),
             RMSDwith = RMSE(cbind(a,d) - cfs1$items[,1:2]),
             biaswithout = bias(cbind(a,d) - cfs2$items[,1:2]),
             RMSDwithout = RMSE(cbind(a,d) - cfs2$items[,1:2]))
    ret
}

Summarise <- function(condition, results, fixed_objects = NULL) {
    colMeans(results)
}

results <- runSimulation(design=Design, replications=200, generate=Generate,
                         analyse=Analyse, summarise=Summarise,
                         packages='mirt', progress=TRUE)

With the results

> print(results, drop.extras = TRUE) 

    nitems    N biaswith.a1 biaswith.d RMSDwith.a1 RMSDwith.d biaswithout.a1 biaswithout.d RMSDwithout.a1 RMSDwithout.d
1       10 1000     -0.0063   -0.00527      0.0183    0.00925          0.242      -0.00868         0.0809         0.015

This uses mirt's MML estimation engine for the two-parameter logistic (2PL) model, but the idea is the same for ltm and could easily be replicated. As you can see, bias is indeed introduced for all parameters, and the overall precision of the parameter estimates is worse when removing the max/min rows.

$\endgroup$
2
  • $\begingroup$ Would it not generally be the case that a test which has a disproportionate number of respondents scoring 0 or 100 yield difficulty parameters on the latent trait that are extreme (i.e., well beyond the usual rubric of -/+ 3)? $\endgroup$
    – Jhaltiga68
    Mar 22, 2017 at 19:30
  • $\begingroup$ @Jhaltiga68 not necessarily. The presence of perfect scores or 0's doesn't imply that any given item is too difficult/easy. $\endgroup$ Mar 22, 2017 at 19:40
0
$\begingroup$

It may be worthwhile doing two runs. The first with all the students to help guide you as to what type of score to give the maximum and minimum students (you have to give them some type of score, even though you are correct that you don't have enough information to give them an accurate score). The final run should be with those student removed so that maximum and minimum scorers don't impact on the measures of the other students or the item parameter estimates (probably a bigger issue).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.