Using IRT when most of the responses are null values

Question

Greetings Cross Validated,

I feel like I need to go into a little background before I bring up the specific issue I'm having. A few years ago Congress passed the FAST act, Part of this act mandated a review of the FMCSA Safety management system methodology (PDF can be found here if you are curious). The review found the current system defensible but not statistically based, and recommended switching to a system based on Item Response Theory. Fast forward a few years, and the FMCSA is getting ready to roll out their new IRT model in Q4 of 2019. The problem is that no one in the industry knows anything about how this is going to work, or how it will affect Motor carriers.

Based on presentations and such, I've got a good idea on how they are going to model the data. They will use total violations from road side inspections for each carrier, with the violations broken out into 66 different categories, using the number of inspections as an indicator of exposure. I have that data together, however I'm having an issue due to the fact that an inspection will only yield a small number of violations, so the collected data has many more null results than values even when looking at two years worth of inspection data. I've been using the MIRT library in R-studio, using the Partial credit model to analyze the data, but due to the large number of nulls (which I replaced with zeros and that didn't work out either) that hasn't been working out.

Is there a different model I should be using, or some way to prep the data that I'm missing? If it will help, I'm happy to provide samples of the data, because it's based on publicly available data sets. Anyway, thank you for taking the time to read this wall of text, and any help at all would be greatly appreciated.

MIRT package in question: https://cran.r-project.org/web/packages/mirt/mirt.pdf Code I'm using so far is:

 setwd("C:\\Data\\IRTData")
 data<-read.csv("sample_IRT_Data.csv", header = TRUE)
 polymodel = mirt(data = data,
                  model = 1,
                  itemtype = "gcpm"
 )

Answer 1

@Jim implied that you shouldn't replace null or NA values with zeros, but didn't explain why. Here's why: zero means that they tried the item and scored a zero, null/NA means that they did not answer the item. In IRT, these would be scored differently.

Imagine a questionnaire with 10 (for example) ordered categorical (i.e. Likert) items. Imagine 0 is the lowest score. It doesn't matter what model you use to score it. If someone answered all 10 items and scored 0, they get the lowest possible ability estimate. If someone didn't answer any of the questions, then your test gives you no added information about their ability - so, if you went and predicted their ability, you'd assign them a zero, which was your prior estimate. Remember that the prior is that abilities are distributed normally, with mean 0 and variance 1.

Side note: I wonder if you are this poster on Reddit?