I was presented with a requirement of comparing two sources of diagnoses on the same subjects, resulting in two 4-level categorical variables. I offered to calculate Cohen's Kappa, but the interested party would prefer a p-value as a result.

a) Is there a method that natively does this? b) Does it make sense to do one McNemar test for each of the four variable levels, making binary variables like 1 and non-1, then 2 and non-2 etc. ?


closed as unclear what you're asking by StatsStudent, Michael Chernick, kjetil b halvorsen, mdewey, Jeremy Miles Feb 21 at 1:19

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ Can you tell us more about these sources / diagnoses and how these result in a 4-level factor? Also what exactly are you trying to compare? $\endgroup$ – user2974951 Feb 19 at 8:53
  • $\begingroup$ The sources are clinical and histopathological diagnoses, and the 4-levels are differential diagnoses of skin cancer. The idea is to show the agreement between the two columns. Since the percentage of agreement is rather high (70+%) it is clear to me that permutation tests will show that the agreement is statistically significant. It would be great to show that the one method is not worse than the other. $\endgroup$ – Cindy Almighty Feb 19 at 12:00
  • $\begingroup$ But you can derive a p-value from Cohen's $\kappa$ so I am not sure what the issue is. $\endgroup$ – mdewey Feb 19 at 14:30
  • $\begingroup$ The issue is that rejecting the null hypothesis in that case would mean that the agreement is not due to chance, while I need to show that one method is not worse than the other. It is expected that two diagnostic methods will give statistically significant Cohen's kappa, but I would prefer a conclusion that one method is or isn't worse than the other. $\endgroup$ – Cindy Almighty Feb 19 at 23:17