1
$\begingroup$

If one wants to see how much variance there is in the responses that various subjects give to a single rating measure, why are measures any more sophisticated than a mere box plot, or error bar with SEMs needed? What do measures such as the Inter-rater reliability tell us, above and beyond simple descriptive statistics of the response distribution?

Am I right that the difference lies in the assumption we make about the "true" value of the thing being rated, in other words that if we use the Inter-rater Reliability we assume that "ideal" subjects would agree and their scores would come close to an objectively-true/measurable value, whereas if using e.g. a box plot, that asuumption need not be made?

Is a high inter-rater reliability not simply (qualitatively) equivalent to a low standard deviation/error of the distribution?

Improve this question
$\endgroup$
10
  • $\begingroup$ It's not clear how you might try to go about assessing inter-rater reliability using a box plot. Are you thinking of generating one plot per measure/person being rated, with the box and whiskers representing the range of scores raters give? $\endgroup$
    – Ian_Fin
    Commented Aug 12, 2016 at 11:05
  • 2
    $\begingroup$ It's also worth noting that inter-rater reliability is a concept, rather than a statistic. That concept is multifaceted (e.g. consistency vs. agreement), using a variety of different statistics. You may want to make clear which facet you're referring to $\endgroup$
    – Ian_Fin
    Commented Aug 12, 2016 at 11:08
  • $\begingroup$ Sorry, I meant to ask how is looking at inter-subject variability with a box-plot (whiskers=variance across subjects), or alternatively computing the SD/SEM of the distribution, different from computing a measure of inter-rater reliability. $\endgroup$
    – z8080
    Commented Aug 12, 2016 at 11:08
  • 1
    $\begingroup$ imagine that I have two raters, rating two subjects. Rater 1 gives subject 1 a score of 1 and subject 2 a 5. Rater 2 gives subject 1 a score of 5 and subject 2 a score of 1. The mean score given by both raters is the same, 3, but rater 1 considers subject 2 to be higher than the mean, while rater 2 considers subject 1 to be higher than the mean. So they agree on the mean, but not who is higher/lower than the mean. $\endgroup$
    – Ian_Fin
    Commented Aug 12, 2016 at 12:36
  • 1
    $\begingroup$ second example: Rater 1 gives subject 1 a score of 1, and subject 2 a 3. Rater 2 gives subject 1 a 3 and subject 2 a 5. While both agree that subject 2 is better, their means differ (2 for rater 1, 4 for rater 2). $\endgroup$
    – Ian_Fin
    Commented Aug 12, 2016 at 12:38

1 Answer 1

Reset to default
3
$\begingroup$

If one wants to see how much variance there is in the responses that various subjects give to a single rating measure, why are measures any more sophisticated than a mere box plot, or error bar with SEMs needed?

If that's what you want to look at, then you don't need inter-rater reliability. In fact, you don't even need multiple raters.

Measures of inter-rater reliability tell you something completely different; namely, they are about inter-rater association. Box plots tell you nothing about this. You can have perfect inter-rater reliability with big dispersion in each rater; you can have low IRR with small dispersion in each rater, or any other combination.

The answer to your question in the middle paragraph is "No, that's not it". The answer to your question in the last paragraph is also "No, that's not it".

EDIT: It appears I misunderstood. The question was about a boxplot of the differnces between raters. That makes more sense. However, it's still quite different from a measure of interrater reliability, just as a (regular) boxplot is not the same as the standard deviation; one is a number, the other a graph.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I think I might not have been clear enough about the box plot and how it might give overlapping information to IRR. The box plot would just represent between- (not within-)subject variability, whereby each data point that goes into the boxplot represents one subject's (single, or averaged) rating. Therefore, I hope I am correct to say that I do in fact need multiple raters as my sample, and that it is hard to find a sample that has high IRR but also a large SD of its distribution. Am I wrong? $\endgroup$
    – z8080
    Commented Aug 12, 2016 at 12:10
  • $\begingroup$ So you are after a boxplot of the difference in ratings? OK, I will edit my answer $\endgroup$
    – Peter Flom
    Commented Aug 12, 2016 at 12:40
  • $\begingroup$ Thanks yes I think we are now on the same page :) But could you please expand on why IRR and SD would be completely independent measures in this case? I still intuitively feel that high IRR should somehow be equivalent to a low SD, but please correct me if I am wrong $\endgroup$
    – z8080
    Commented Aug 12, 2016 at 12:52
  • $\begingroup$ There would certainly be some relation, but I'm not sure what. The IRR is more directly what most people want when they are asking about how well two raters agree: It's like correlation. $\endgroup$
    – Peter Flom
    Commented Aug 12, 2016 at 12:55
  • 1
    $\begingroup$ Pairwise IRR would be a special case - easier to see that this question is completely separate from that of the distribution dispersion. But for a generalised IRR, that shows how much agreement there is between all subjects as a group rather than between any two of them, I still don't understand what this measure could mean other than "low dispersion", i.e. low SD $\endgroup$
    – z8080
    Commented Aug 12, 2016 at 14:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.