Inter-rater reliability for Ratio data

Question

Which inter-rater method is most suitable for data which is measured in interval scale? I have data which are measured in percentages. e.g.

Rater 1

A: 50% B: 20% C: 11% D: 9% E: 10%

Rater 2

A: 50% B: 20% C: 10% D: 10% E: 10%

I have two raters rating how much A B C D E..... are accounted for in a study. In the example above, there are small disagreement on C and D among the two raters. I understand that the differences are small and I want to prove that the two raters have statistically significant agreement (or, in another case, their scoring are so different that it is statistically different)

In the study each rater are rating 30 categories so there will be decimals or even rounding for the percentage numbers.

Answer 1

The easiest way would be to use the intraclass correlation coefficient (ICC) for each category. Alternatively, you can use any chance-adjusted agreement index (e.g., S score, kappa coefficient, or pi coefficient) with interval or ratio weights for each category, as described on my website. However, I would caution you not to rely on null hypothesis significance testing for this. Rather than showing that the raters have significantly more than 0.0 agreement (which is obvious and unhelpful), it would be better to quantify the extent of their agreement and provide a confidence interval for this estimate.