IRT Nominal Response Model theta

Question

I'm quite familiar with multiple IRT models; however, I can't seem to understand the Nominal Response Model. How is theta estimated? For example, if we have a science test with multiple-choice items ("Which one of the following is an animal?" A) chair, B) cat, C) owl, D) sunflower), how does the model "know" which answer reflects the greatest knowledge? Do I have to put the answers in "logical" order myself (as 0 - chair, 1 - sunflower, 2 - owl, 3 - cat), so the bigger the category, the "righter" the answer? Or do I have to add the key to the model?

It's quite confusing because I've read everywhere that the answers can be unordered (that's why it's called nominal for sure). However, if I do not add the key or reorder the answers myself in a logical way, then I really can't understand how the ability/theta is estimated.

Answer 1

Although much of the terminology regarding IRT models suggests that IRT is about "abilities" and therefore wrong or correct (more or less correct) answers, this is not always how it is applied. The $\theta$ parameters may also point to a certain trait and to what extent an answer expresses this trait, or, equivalently (see below), its opposite.

The nominal response model will model the response probabilities for the different options assuming a one-dimensional person parameter $\theta$ without assuming how the individual choices are related to it. The choices will have numerical scores (an assumption is that there is a true score for every choice but not directly observed), and these scores are estimated from the data (as are the person parameters). This means that there is the assumption of an underlying one-dimensional "trait", and that in principle the choices can be ordered, but that the order is unknown. In particular this also means that the direction of the order is not identified. If you multiply all the scores by -1, you have an equivalent model. There is no well defined "correct/ability" side of the scale vs. "wrong/inability".

For example (I'm making things up here, maybe rather nonsensical) you may have a number of questions that you suspect are related to a one-dimensional psychological trait somehow, like "what's your favourite colour", "which of the following sports do you prefer practicing", "what is your favourite music genre", and then the data may show that you tend to find "blue", "cycling", "ambient electronic" together on one side of the scale, "red", "high jump", "heavy metal" together on the other, other answers somewhere in between. Obviously that there is a one-dimensional trait underlying these is an assumption that may well be very wrong, but fitting the model and diagnosing it may help you to see that. Like often in model fitting and model selection, it is an assumption that reduces complexity, and one can look at how well the fitted model explains the data to see whether this is efficient.

In any case I think that this is more suitable for situations in which it's not a well defined ability that is measured, where one side is "ability" and some answers are well defined as "correct", others as "wrong", because in such cases we would usually want to put more information into the analysis than the nominal response model uses. (I have read that this is also applied in situations in which a researcher may have an idea about how answers may be ordered and connected, maybe even in ability measurement, and isn't 100% sure about it, for testing whether the nominal model will recover the order assumed by the researcher even without being put explicitly into the model as assumption.)

Answer 2

I'm quite familiar with multiple IRT models; however, I can't seem to understand the Nominal Response Model. How is theta estimated? For example, if we have a science test with multiple-choice items ("Which one of the following is an animal?" A) chair, B) cat, C) owl, D) sunflower), how does the model "know" which answer reflects the greatest knowledge? Do I have to put the answers in "logical" order myself (as 0 - chair, 1 - sunflower, 2 - owl, 3 - cat), so the bigger the category, the "righter" the answer? Or do I have to add the key to the model?

No, you don't have to add the key. Adding a key to multiple-choice data like you describe would be equivalent to fitting the two-parameter logistic (2PL) model if "0" were to be assigned to all response options except for the correct response.

It's quite confusing because I've read everywhere that the answers can be unordered (that's why it's called nominal for sure). However, if I do not add the key or reorder the answers myself in a logical way, then I really can't understand how the ability/theta is estimated.

So, if I understand this part of your question correctly, you are asking how the IRT estimation algorithm "knows" which response option is indicative of the highest $\theta$ level. To understand how this is done, it is first important to realize that not all nominal data is appropriate for the Nominal Response Model (NRM; Bock, 1972; Thissen, Cai, & Bock, 2011). Revuelta, Maydeu-Olivares, & Ximénez (2020) make this clear by differentiating first choice data - a type of nominal data appropriate for the NRM, from other types of nominal data.

We need to distinguish between two kinds of nominal data. Usually, when we think of nominal data we think of purely unordered data such as country of residence (1 = US, 34 = Spain, etc.). In these cases, factor analysis is not applicable. However, in some cases, we can conceive of the existence of an underlying order among the response alternatives, with the observed outcome being the result of a decision-making process. For instance, if we ask “in what country would you like to retire?”, we could use the same numeric codes as before, but the obtained data would be different; it would be first choice data. In first choice data, we assume that the respondent orders the alternatives in her mind but only provides her top choice to follow the instructions received (Bock, 1997; Maydeu-Olivares & Böckenholt, 2005, 2009). First choice data is still nominal data, as the response alternatives are unordered. However, because of the assumed underlying order, one of the major tasks in applications is to ‘uncover’ the ordering of the alternatives presented.

Now, regarding how this is done, I suggest @Christian Hennig's answer as they address this very well. Though, in a few words, the NRM does not make any assumptions regarding the "correctness" of a response.

References

Darrell Bock, R. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29-51.

Revuelta, J., Maydeu-Olivares, A., & Ximénez, C. (2020). Factor analysis for nominal (first choice) data. Structural Equation Modeling: A Multidisciplinary Journal, 27(5), 781-797.

Thissen, D., Cai, L., & Bock, R. D. (2011). The nominal categories item response model. In Handbook of polytomous item response theory models (pp. 43-75). Routledge.