Simulating responses to a test for item-response theory

Question

I am developing an online assessment system and I need to calibrate the bank of questions but I do not have enough people to implement a pilot test. That is why I decided to simulate the responses of the question bank. I will use the Rasch model for the development of a computer adaptive test (CAT); I have only questions from the bank and the number of options for each. How I can do a simulation using only this data?

I had thought to make it generate a random number $X$, if $X < 1 / N$, where $N$ is the number of options in a particular question, then the person has gotten the answer correct, otherwise it is incorrect.

Is this a correct approach for calibrating the item bank to a Rasch model via simulation?

PS: What I want to do is to calculate the level of difficulty of each question. It is my understanding that to calibrate a question bank, that bank must be applied to a sample of subjects, but I do not have enough people to do this, so I decided to try a simulation approach. I tested the function sim.rasch in the psych package, and I see that I need to take into account the number of options for each question, as this may influence their difficulty.

Answer 1

I've included a simdata() function in the mirt package in R for calculating simulated IRT data given a variety of know conditions for several different classes of uni- and multidimensional IRT models. So if you need something a little more flexible that may be a good place to look, and should save you from having to reinvent the wheel.

Answer 2

Here's a first version of this simulation in R, as an example of what NOT to do. Code untested.

# we're making a table of three columns: person, question, and correct or not
resps <- data.frame(person=integer(), question=integer(), correct=integer())

# do the simulation
for (qu in 1:nQ) { # loop over questions, nQ is number of questions
  # how many possible answers does this question have? flip a coin with max_opts sides
  opts <- sample.int(max_opts, 1)

  for (pe in 1:nP) { # loop over test-takers, nP is number of test-takers
    # did this person answer correctly? flip a coin with 1/opts probability of success
    resp <- rbinom(1, 1, 1/opts)
    resps <- rbind(resps, data.frame(person=pe, question=qu, correct=resp))
  }
}

Here are some reasons that this is a bad idea:

• it doesn't allow for missing responses
• it assumes all people are of the same ability
• it assumes all questions are of the same difficulty (given the number of answer options)

You could come up with a few more...

The point is that you want to have a feeling for who you're testing and what you're testing them on, and you want the latent properties of those people and questions to be reflected in your simulated data.

My advice would be to try to get some real humans to take your test for real. It might be pretty cheap and easy on Amazon Mechanical Turk.

Answer 3

The easiest way to do what you want is probably to use the psych package for R. This includes functions such as sim.rasch and sim.irt which will simulate appropriate data of whatever size for you.

Answer 4

I know it is a bit too late, but for historical reasons I'd like to answer this question.

Simulating a CAT using an R package

Nowadays, the package catr is able to simulate CATs, which, it seems, is exactly what you want to do. It has a lot of options, like number of starting itens, selection method for the next item, configuration of stopping rule etc.

Here is an old bit of code I have in my computer. All credits go to the catr manual.

# call the catR package, if not installed then install it
if (!require('catR')) install.packages('catR')

require('catR')
# create a bank with 3PL items
Bank <- genDichoMatrix(items = 500, cbControl = NULL, model = "Rasch",
                       seed = 1)

# list of four parameters that characterize a CAT: start, test, stop, final
# these lists will feed the randomCAT function to generate a response pattern

# one first item selected, ability level starts at 0, criterion for
# selecting first items is maximum Fisher information
Start <- list(nrItems = 1, theta = 0, startSelect = "MFI")

# use weighted likelihood, select items through MFI (see previous comment)
Test <- list(method = "WL", itemSelect = "MFI")

# stopping rule by classication, meaning that the test will stop when the
# CI no longer holds the threshold inside it anymore
Stop <- list(rule = "precision", thr = 0.4, alpha = 0.05)

# how estimates of ability are calculated
Final <- list(method = "WL", alpha = 0.05)

# set true ability at 1, calls lists above
res <- randomCAT(trueTheta = rnorm(n=1,mean=0,sd=1), itemBank = Bank,
                 start = Start, test = Test,
                 stop = Stop, final = Final)

# plotting the response pattern
plot(res, ci = TRUE, trueTh = TRUE, classThr = 2)

Extra bit: extracting item parameters from response patterns

From the question, I realized you already had response patterns that you'd like to use to extract the item parameters (discrimination, difficulty, guessing factor etc). In order to do that, I recommend the mirt package, which does just that (and a lot more). You can find examples on how to use this package here and here.

The only extra work I predict you would have to do is convert mirt's output matrix to the input format that catr uses.