0
$\begingroup$

There are 3 subjective methods to measure the severity of a condition. Each method measures it on an ordinal scale, but the scales are different:

method 1 scale is 1-3

method 2 scale is 1-4

method 3 scale is 1-5

Evaluation of a condition according to each method is performed by 3 raters.

Krippendorff's alpha seems like an optimal approach to calculate the agreement between those 3 rates.

How could I compare the 3 resulting Krippendorff's alphas for each method and estimate if one method gives significantly better agreement than the other 2?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Assuming zero bias among the raters, the "1-4" and "1-5" will have the best agreement. This is actually a simple math problem. When you round a number to fit in a scale of "1-3," you lose more information than you lose with bigger scales.

The bias issue may be a problem. If you ask person A to rate using both scales: (disagree, agree, strongly agree) and (strongly disagree, disagree, agree, strongly agree), would person B do it the same way? People have different intuitive thresholds for "strongly." And then there's the standard bias issue. Seeing things differently.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I would expect the opposite: the bigger the number of categories is, the less is the chance for raters to agree. However, it does answer the question: how can I compare if one method gives significantly better agreement than the other 2. Can I test it or shall I bootstrap and look at CIs? $\endgroup$
    – Gregory
    Jul 20, 2020 at 11:17
  • $\begingroup$ How are you measuring bias if you reduce the available scale, hoping for more consistency? You may not actually get that. -- If you are just looking for absolute agreement percent between the two methods, run two tests on two different groups. Group A gets 3 choice answers. Group B gets 5 choice answers. Then count the number of exact matches in both groups. $\endgroup$
    – user255758
    Jul 21, 2020 at 2:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.