There are 10 raters giving a value on the ordinal scale for 20 instruments. I applied Krippendorff's alpha to calculate the overall agreement between the raters, that is equal to 0.7 (tentative agreement). How can I identify the rater who disagrees more as compared to the rest? What would be the right method?

  • 2
    $\begingroup$ That's a fairly small dataset, so you could post it here. Otherwise, the wording here is ambiguous. Do you know that one rater is very different from the others or is your question broader in terms of identifying one or more raters who are some kind of minority or at least quite different from most others. Graphical possibilities include profile (parallel coordinate) plots and correspondence analysis. $\endgroup$
    – Nick Cox
    Commented Feb 11, 2021 at 13:47
  • $\begingroup$ Unfortunately, I am not allowed to share the dataset. It does not seem that any rater deviates from the rest, but it is a theoretical consideration. If this would be a group of raters (>1) then I could calculate K-alpha for the groups and check if CIs overlap. I also took your advice to look at the plots, but did not really see any obvious deviations. $\endgroup$
    – Gregory
    Commented Feb 18, 2021 at 9:51
  • 1
    $\begingroup$ You could post a mock-up dataset with similar properties! $\endgroup$ Commented Feb 18, 2021 at 13:48
  • $\begingroup$ The need for confidentiality is appreciated, but we are distinctly limited without sight of real(istic) data. Overall measures of agreement can't possibly help much in identifying who is anomalous. You can plot all the data in one graph, but the issue is getting a graph that is easy to think about. Presumably the raters don't come in any order, although that could be true, e.g. if your sample ranged from supposed experts to newcomers. Possibly the instruments can be arranged in some kind of order, or it's one purpose of correspondence analysis to find that out. $\endgroup$
    – Nick Cox
    Commented Feb 18, 2021 at 14:00
  • $\begingroup$ The kind of plot shown at stats.stackexchange.com/questions/190152/… may also help. In Stata circles, I've called them "front-and-back plots" and the name is available. $\endgroup$
    – Nick Cox
    Commented Feb 18, 2021 at 14:01


Browse other questions tagged or ask your own question.