I currently work on an item analysis for an assessment. The assessment consists of 25 multiple-choice questions. Each question has 4 choices with one correct answer. I used different IRT models to estimate item discrimination (a-parameter) and difficulty levels (b-parameter), along with guessing values (c-parameter). Firstly, I tested a 2PL IRT model and a 3PL IRT model. I compared the results of the two models and determined that the 3PL model was a better model. Second, I generated a few mixed 2PL-3PL models using information from the 3PL model. In the first mixed model, items with guessing value greater than 0.3 were treated as 3PL models and the remaining items were treated as 2PL models. In the second mixed model, items with guessing value greater than 0.2 were treated as 3PL models and the remaining items were treated as 2PL models. In the third mixed model, items with guessing value greater than 0.1 were treated as 3PL models and the remaining items were treated as 2PL models. I usually test the third mixed model only if the 3PL model is better than the 2PL model. If opposite is true, I just test the first and second mixed model to get the best model. I tested and compared all mixed models and selected the best mixed model. After that, I compared the 3PL model with the best mixed model to get my final best model. Here are my questions:
- Is my procedure correct for selecting the best IRT model?
- What is the cut-off score for c-parameter (guessing value) in the 3PL model in order to consider an item as a 3PL model?
With these questions in mind, I conducted an online search. I found a paper on “Fixing the c parameter in the three-parameter logistic model” (Han, 2012). According to Han (2012), “neither the a-, b-, or c parameters of 3PLM can accurately reflect the discrimination, difficulty, and guessing properties of an item, respectively”, and the three parameters are heavily influenced by each other in the 3PL model. He also discussed that computer programs will estimate a guessing value when c-parameter cannot be estimated, making estimation of the 3PL model uninterpretable and the a- and b- parameters incomparable across items. So one salutation is to fix the c- parameter as (1/the number of options) for each item.
Because I am not very familiar with the field of IRT, my questions become:
- How do I determine which mixed 2PL and 3PL model to be tested?
- Should I let 3PL model estimate the guessing values or should I fix the guessing value for stable and reliable a- and b- parameters? Thanks so much for your time and help.
I am using mirt to conduct my analysis. The main goal is to estimate the item difficulty levels and discrimination values for a given assessment(i.e., Language skills, or computational skills). I try to identify the best model that gives more accurate estimations. I don't have any theory about the assessment. Basically I explore different models to identify the best model. When I tried to explain the reason why I decided to compare two models using the guessing values from 3PL models, my co-worker did not get it. It suggests to me that my rational may be incorrect. I did some research and still felt puzzled about the use of guessing value, especially after I found the paper written by Han (2012).
I think I am lost because I am unsure why I am using guessing values as criteria to determine the mixed models to be tested. If not using guessing values, what other ways I can explore different models to find the best model that gives more accurate estimation of item difficulty levels and discrimination values for an given assessment? Or Can I test the mixed 2PL and 3PL model at all? (p.s., I have found that people used either 2PL models or 3PL models during my research).
And I am worried the method I am using to estimate a, b, c parameters will cause problems in linking/equating/calibration items later. Yet, I don't understand how they may influence these processes. Sorry I have so much questions.