1
$\begingroup$

I was presented with a requirement of comparing two sources of diagnoses on the same subjects, resulting in two 4-level categorical variables. I offered to calculate Cohen's Kappa, but the interested party would prefer a p-value as a result.

a) Is there a method that natively does this? b) Does it make sense to do one McNemar test for each of the four variable levels, making binary variables like 1 and non-1, then 2 and non-2 etc. ?

Improve this question
$\endgroup$
4
  • $\begingroup$ Can you tell us more about these sources / diagnoses and how these result in a 4-level factor? Also what exactly are you trying to compare? $\endgroup$
    – user2974951
    Commented Feb 19, 2019 at 8:53
  • $\begingroup$ The sources are clinical and histopathological diagnoses, and the 4-levels are differential diagnoses of skin cancer. The idea is to show the agreement between the two columns. Since the percentage of agreement is rather high (70+%) it is clear to me that permutation tests will show that the agreement is statistically significant. It would be great to show that the one method is not worse than the other. $\endgroup$
    – Cindy Almighty
    Commented Feb 19, 2019 at 12:00
  • $\begingroup$ But you can derive a p-value from Cohen's $\kappa$ so I am not sure what the issue is. $\endgroup$
    – mdewey
    Commented Feb 19, 2019 at 14:30
  • $\begingroup$ The issue is that rejecting the null hypothesis in that case would mean that the agreement is not due to chance, while I need to show that one method is not worse than the other. It is expected that two diagnostic methods will give statistically significant Cohen's kappa, but I would prefer a conclusion that one method is or isn't worse than the other. $\endgroup$
    – Cindy Almighty
    Commented Feb 19, 2019 at 23:17

0

Reset to default

Browse other questions tagged or ask your own question.