1
$\begingroup$

I want to measure the inter-agreement rate between two physicians that rated the same set of images with two different methods (rater1 used method1, rater2 used method2) BUT

  • Rater1 rated the the images with method1 : 0 (absence of the lesion) or 1 (presence of the lesion)

  • Rater2 rated the images with method2 : 0 (absence of the lesion), 1 (small lesion), 2 (little bigger lesion), 3 (intermediate size of lesion), 4 (big lesion) or 5 (very big lesion)

Like you can see my rating variables have different set of catergories (2 categories for rater1 and 6 for rater2). how can I deal with that? is there a way to change the scores 1,2,3,4 and 5 (of rater2) to 1 (so we find our self in the classic setting 2 categories vs 2 categories) but still give some sort of weights to them (because if rater1 with method 1 scored 0 when the rater2 with method 2 scored 5 it's more serious than if rater1 scored 0 when rater2 scored 2 for example)

I want to know if method 1 is comparable to method 2 based on this scores (i.e. method 2 is the gold standard). Basically how method 1 perform compared to method 2 based on scoring variables/ how is method 1 reliable compared to the gold standard method 2?

PS: if you have another way then KAPPA to answer the latter question I take it !

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In order to compute Cohen's alpha or any other of agreement measure, you'd have to recode both variables to the same set of categories.

However, you could first cross-tabulate the ratings of both raters to indicate where they disagree and in how many cases, for example, a small lesion (1 in coding scheme 2) is already coded as a lesion (1 in CS 1).

In this cross-table you would also spot strong disagreement (e.g.: 5 in CS2 and 0 in CS1), where you could look into cases in detail.

These cross-tables are a good way to get a feeling of rater reliability before computing actual scores. You can then proceed to recode the second coding scheme to 0 (=0) and 1 (=1,2,3,4,5) or any other cutting point and compute kappa.

Depending on how interested you are in the performance of the two raters and the compatibility of scales, you could also recode CS2 with different cut-points to see where the agreement would be maximal. For example, it might be that 0 in CS1 actually corresponded to 0 or 1 in CS2. If this was the case, you would have to concede that the first rater did not consider small lesions to be actual lesions.

As in most cases, an interrater agreement analysis goes beyond just computing a score.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you. Actually instead of using a threshold method (cutting points). what about do it separately? Meaning in this new coding we would have : 1st 0 (=0) and 1(=1), 2nd 0 (=0) and 2(=1), 3rd 0 (=0) and 3(=1), 4th 0 (=0) and 4(=1); 5th 0 (=0) and 5(=1) and for each 'scheme' we include a weight matrix e.g. for the 5th 'scheme' weight matrix would be w5=matrix(c(0,5,1,0),ncol=2,byrow=TRUE). We can include the weight w5 with 'cohen.kappa' function (from psych package in R)..... we would have: cohen.kappa(contingency _table,w5). $\endgroup$
    – learners
    Commented Aug 27, 2021 at 9:16
  • 1
    $\begingroup$ I would not weight the data for an interrater reliability test, unless there is a really strong theoretical reason to do so (e.g.: upweigh difficult cases to make it harder). Of course, you could use dummies for all the categories on the second coding scheme, but that won't help you much. It just gives you an extremely low agreement for each single category as all lesser AND worse lesions are coded 0. I don't see how this might work. $\endgroup$
    – Martin Wettstein
    Commented Aug 27, 2021 at 9:31
  • $\begingroup$ Thank you. But why do you think "all lesser AND worse lesions are coded 0"? they are actually coded '1' not '0'. 0 is only for absence of lesion? (maybe i am missing something) $\endgroup$
    – learners
    Commented Aug 27, 2021 at 9:55
  • 1
    $\begingroup$ Ah, I thought you want to create dummies. If you recode only two values, however, you get too many missing values for any meaningful pairwise comparison. Cutting at a given value would be more robust. $\endgroup$
    – Martin Wettstein
    Commented Aug 27, 2021 at 10:17
  • $\begingroup$ Yes indeed by doing so I lower my sample size! Also a problem with including weight matrices is how to fill up these weight matrices, which weights Wij are best to choose in order to demonstrate the level of disagreement ?... $\endgroup$
    – learners
    Commented Aug 27, 2021 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.