Questions regarding interpretation of item parameters: inconsistent results between classical item analysis and item response analysis

Question

I was not trained as a psychometrician, so my questions may be concept-related. If you know different resources, books, or articles that can clear my confusions, please let me know. Thank you very much for helping. I appreciate your time and effort in answering my question. Currently I am working on item analysis for an assessment (i.e., literacy skills) for different groups. I conducted classical item analysis to obtain P values and Point Biserial values. I also conducted analysis of item response modeling to obtain item difficulty and discrimination. The sample size was between 600 and 800 participants. For IRT analysis, I conducted 3PL model with fixing guessing value, as recommended by Han (2012). However, I have found that the results I got from classical item analysis were inconsistent with the results I got from IRT analysis. See a few examples below.

TestName Question CorrectAnswer Pvalue  PBvalue Discrimination  Difficulty

Pink Assessment Q13 D   0.87    0.48    2.29    -1.18
Pink Assessment Q17 B   0.91    0.47    3.12    -1.38
Pink Assessment Q18 C   0.21    0.34    6.78    1.49
Pink Assessment Q22 AB  0.10    0.34    2.31    1.94
Pink Assessment Q26 C   0.28    0.29    2.89    1.67
Pink Assessment Q32 C   0.44    0.17    0.13    8.61

As you see, for Q32, classical item analysis considered the item as medium difficulty (almost ideal), where IRT analysis considered the item as super difficult. Both PB values suggest that the item had low discrimination, which was consistent with the results of IRT analysis. For Q22, classical item analysis considered the item as super hard and had acceptable discrimination. However, IRT analysis considered the item was high discriminative. The common assumption is that hard or easy item usually has low discrimination. Thus, I feel puzzled about the results. I searched online for answers, but I was unable to clear my confusion. I feel it may be due to the fact that I set the guessing value in 3PL IRT models. So I compared the 3PL IRT models with fixed C parameter and the one without. The 3PL IRT models without fixed C parameter had similar issues but for different items. My questions, why the results were inconsistent? What criteria/rule of thumbs I can use when determining which results were more reliable or trustworthy? Why items with high difficulty level were highly discriminative? Again, thank you so much for helping.

Answer 1

In CTT, very difficult and very easy items usually do have poor discrimination because subjects are forced into a very narrow range of scores. The discrimination in IRT refers to the slope of the curve in the area of interest. In you example, if only 10% of the subjects get a question correct and only the subjects with the highest amount of the attribute you are measuring get the answer correct, then the slope of the curve could be quite high. The use of "discrimination" in both field means different things.