Weighted Kappa with rating variables that have different set of categories I want to measure the inter-agreement rate between two physicians that rated the same set of images with two different methods (rater1 used method1, rater2 used method2) BUT

*

*Rater1 rated the the images with method1 : 0 (absence of the lesion) or 1 (presence of the lesion)


*Rater2 rated the images with method2 : 0 (absence of the lesion), 1 (small lesion), 2 (little bigger lesion), 3 (intermediate size of lesion), 4 (big lesion) or 5 (very big lesion)
Like you can see my rating variables have different set of catergories (2 categories for rater1 and 6 for rater2). how can I deal with that? is there a way to change the scores 1,2,3,4 and 5 (of rater2) to 1 (so we find our self in the classic setting 2 categories vs 2 categories) but still give some sort of weights to them (because if rater1 with method 1 scored 0 when the rater2 with method 2 scored 5 it's more serious than if rater1 scored 0 when rater2 scored 2 for example)
I want to know if method 1 is comparable to method 2 based on this scores (i.e. method 2 is the gold standard). Basically how method 1 perform compared to method 2 based on scoring variables/  how is method 1 reliable compared to the gold standard method 2?
PS: if you have another way then KAPPA to answer the latter question I take it !
 A: In order to compute Cohen's alpha or any other of agreement measure, you'd have to recode both variables to the same set of categories.
However, you could first cross-tabulate the ratings of both raters to indicate where they disagree and in how many cases, for example, a small lesion (1 in coding scheme 2) is already coded as a lesion (1 in CS 1).
In this cross-table you would also spot strong disagreement (e.g.: 5 in CS2 and 0 in CS1), where you could look into cases in detail.
These cross-tables are a good way to get a feeling of rater reliability before computing actual scores. You can then proceed to recode the second coding scheme to 0 (=0) and 1 (=1,2,3,4,5) or any other cutting point and compute kappa.
Depending on how interested you are in the performance of the two raters and the compatibility of scales, you could also recode CS2 with different cut-points to see where the agreement would be maximal. For example, it might be that 0 in CS1 actually corresponded to 0 or 1 in CS2. If this was the case, you would have to concede that the first rater did not consider small lesions to be actual lesions.
As in most cases, an interrater agreement analysis goes beyond just computing a score.
