Questions Regarding Item Response Models – the 3PL model vs. the mixture of 2PL and 3PL model

Question

I currently work on an item analysis for an assessment. The assessment consists of 25 multiple-choice questions. Each question has 4 choices with one correct answer. I used different IRT models to estimate item discrimination (a-parameter) and difficulty levels (b-parameter), along with guessing values (c-parameter). Firstly, I tested a 2PL IRT model and a 3PL IRT model. I compared the results of the two models and determined that the 3PL model was a better model. Second, I generated a few mixed 2PL-3PL models using information from the 3PL model. In the first mixed model, items with guessing value greater than 0.3 were treated as 3PL models and the remaining items were treated as 2PL models. In the second mixed model, items with guessing value greater than 0.2 were treated as 3PL models and the remaining items were treated as 2PL models. In the third mixed model, items with guessing value greater than 0.1 were treated as 3PL models and the remaining items were treated as 2PL models. I usually test the third mixed model only if the 3PL model is better than the 2PL model. If opposite is true, I just test the first and second mixed model to get the best model. I tested and compared all mixed models and selected the best mixed model. After that, I compared the 3PL model with the best mixed model to get my final best model. Here are my questions:

1. Is my procedure correct for selecting the best IRT model?
2. What is the cut-off score for c-parameter (guessing value) in the 3PL model in order to consider an item as a 3PL model?

With these questions in mind, I conducted an online search. I found a paper on “Fixing the c parameter in the three-parameter logistic model” (Han, 2012). According to Han (2012), “neither the a-, b-, or c parameters of 3PLM can accurately reflect the discrimination, difficulty, and guessing properties of an item, respectively”, and the three parameters are heavily influenced by each other in the 3PL model. He also discussed that computer programs will estimate a guessing value when c-parameter cannot be estimated, making estimation of the 3PL model uninterpretable and the a- and b- parameters incomparable across items. So one salutation is to fix the c- parameter as (1/the number of options) for each item.

Because I am not very familiar with the field of IRT, my questions become:

1. How do I determine which mixed 2PL and 3PL model to be tested?
2. Should I let 3PL model estimate the guessing values or should I fix the guessing value for stable and reliable a- and b- parameters? Thanks so much for your time and help.

EDIT:

Additional Information:

I am using mirt to conduct my analysis. The main goal is to estimate the item difficulty levels and discrimination values for a given assessment(i.e., Language skills, or computational skills). I try to identify the best model that gives more accurate estimations. I don't have any theory about the assessment. Basically I explore different models to identify the best model. When I tried to explain the reason why I decided to compare two models using the guessing values from 3PL models, my co-worker did not get it. It suggests to me that my rational may be incorrect. I did some research and still felt puzzled about the use of guessing value, especially after I found the paper written by Han (2012).

I think I am lost because I am unsure why I am using guessing values as criteria to determine the mixed models to be tested. If not using guessing values, what other ways I can explore different models to find the best model that gives more accurate estimation of item difficulty levels and discrimination values for an given assessment? Or Can I test the mixed 2PL and 3PL model at all? (p.s., I have found that people used either 2PL models or 3PL models during my research).

And I am worried the method I am using to estimate a, b, c parameters will cause problems in linking/equating/calibration items later. Yet, I don't understand how they may influence these processes. Sorry I have so much questions.

Answer 1

1) That seems like an okay way to test the item response functions, though it's unclear what the details are (how are they compared, likelihood ratio statistics, Wald tests, etc, or via model fitting routes where the 2PL item fit is poor, but the 3PL is okay?). It's also important to know the software you are using, because different software will automatically impose prior distributions in the 3PL model that can bias results.

2) I don't believe there is a specific cut-off for the guessing parameters, and even then its not necessarily the value itself you should care about; i.e., what is the parameters 95% confidence interval? You might have to use a profiled likelihood approach to answer that question, which is available in software such as mirt in R.

3) You determine which items should be tested in theory. Automatic selection procedures are purely exploratory in nature, and therefore you'll have to worry about Type I error issues. If you are interested in simply testing all the items yourself, be aware that you are doing exploratory searches in your analysis.

4) That's a very difficult question, as even fixing the value can cause issues with the other parameters. If your sample size is very large, then there probably is no harm in estimating them, but smaller sample sizes this can become a huge issue. Use of prior distributions can also help here, even in larger samples. Ultimately, if the purpose is to eventually score your test, then you should do what makes sense to maximize the precision in that area and probably pick a more theoretically suitable model.