Dealing with least and maximum score in IRT

Question

Generally when I do IRT modelling (using R software, 2PL model, ltm package) I remove records of students who have scored either 0 or 100%. The logic being that we do not have enough information about the ability of these students.

But this way I cannot allocate any score to students who scored 0 or 100%

I have tried running the model with such records also. The model converges. But I am not sure if it is correct to do it. I am getting some values for zero and 100% scorers but I am not sure how it is being calculated in the ltm package.

Answer 1

In general this is a bad idea. ltm uses marginal maximum likelihood (MML) estimation, and therefore zero response vectors are still used in estimation and give information about the 'difficulty' of an item, unlike joint maximum likelihood (JML) is which case these need to be removed (like in the program LOGIT).

The same thing can be said about removing rows with missing data. While it shouldn't introduce bias if the responses containing missing data are missing at random (MAR), it will decrease the precision of recovering population parameters. However, removing extreme responses isn't removing by a MAR scheme (clearly), so bias is introduced and parameters are estimated with less precision.

Here's a quick simulation (maybe 3 minutes or so...and could easily be run in parallel by passing parallel = TRUE to runSimulation()) where you estimate the model with and without extreme response patterns, and you can see the RMSE and bias statistic are larger without these patterns due to the loss of information and introduced bias.

library(SimDesign)

Design <- createDesign(nitems=10, N=1000)

Generate <- function(condition, fixed_objects = NULL) {
    Attach(condition)
    a <- matrix(rlnorm(nitems, .2, .2))
    d <- matrix(rnorm(nitems))
    resp <- simdata(a,d, N, itemtype='2PL')
    list(resp=resp, a=a, d=d)
}

Analyse <- function(condition, dat, fixed_objects = NULL) {
    Attach(dat)
    modwith <- mirt(resp, 1, verbose = FALSE)
    sums <- rowSums(resp)
    modwithout <- mirt(resp[sums != 0 & sums != condition$nitems, ], 1, 
                       verbose = FALSE)
    cfs1 <- coef(modwith, simplify=TRUE)
    cfs2 <- coef(modwithout, simplify=TRUE)
    ret <- c(biaswith = bias(cbind(a,d) - cfs1$items[,1:2]),
             RMSDwith = RMSE(cbind(a,d) - cfs1$items[,1:2]),
             biaswithout = bias(cbind(a,d) - cfs2$items[,1:2]),
             RMSDwithout = RMSE(cbind(a,d) - cfs2$items[,1:2]))
    ret
}

Summarise <- function(condition, results, fixed_objects = NULL) {
    colMeans(results)
}

results <- runSimulation(design=Design, replications=200, generate=Generate,
                         analyse=Analyse, summarise=Summarise,
                         packages='mirt', progress=TRUE)

With the results

> print(results, drop.extras = TRUE) 

    nitems    N biaswith.a1 biaswith.d RMSDwith.a1 RMSDwith.d biaswithout.a1 biaswithout.d RMSDwithout.a1 RMSDwithout.d
1       10 1000     -0.0063   -0.00527      0.0183    0.00925          0.242      -0.00868         0.0809         0.015

This uses mirt's MML estimation engine for the two-parameter logistic (2PL) model, but the idea is the same for ltm and could easily be replicated. As you can see, bias is indeed introduced for all parameters, and the overall precision of the parameter estimates is worse when removing the max/min rows.

Answer 2

It may be worthwhile doing two runs. The first with all the students to help guide you as to what type of score to give the maximum and minimum students (you have to give them some type of score, even though you are correct that you don't have enough information to give them an accurate score). The final run should be with those student removed so that maximum and minimum scorers don't impact on the measures of the other students or the item parameter estimates (probably a bigger issue).