2
$\begingroup$

I have a dataset of 80 samples, with each containing two measurements on a 0-10 ordinal scale (Rating_1 and Rating_2) as well as their true values on the same scale (Rating_True). I'm now looking for a way to visualize the deviation of each rater from the truth, allowing me to judge if raters tend to over- or underestimate the value.

Example dataset in R:

df <- structure(list(ID = 1:20, 
Rating_1 = c(9,8,6,9,8,6,7,3,9,4,5,5,0,1,6,3,2,1,0,7), 
Rating_2 = c(7,10,10,10,10,8,5,9,7,6,8,6,3,4,7,2,0,3,1,2), 
Rating_True = c(10,10,9,9,8,8,7,7,6,6,5,5,4,4,3,3,2,2,1,0)), 
class = "data.frame", row.names = c(NA, 
-20L))

I was thinking about Bland-Altman plots, comparing the mean deviation of each of the two raters to the truth separately (example below).enter image description here

Another option would be comparing BoxPlots of both raters for each level of the scale (example below).

enter image description here

Would you judge one of the approaches (or a unmentioned third approach) superior? If so, why? Could you point me to additional resources for finding a good approach?

Thanks a lot!

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Can you post a small, example dataset for people to work with? It doesn't have to be your full dataset n=20 might be fine, & you could post fake data, if you prefer. $\endgroup$
    – gung - Reinstate Monica
    May 6, 2021 at 14:04
  • 1
    $\begingroup$ Do you want to compare these two raters to each other, or get a sense of how raters perform in general? $\endgroup$
    – gung - Reinstate Monica
    May 6, 2021 at 14:06
  • 1
    $\begingroup$ You say the data are ordinal, are you assuming the values are approximately equal interval (ie, can be treated as roughly continuous)? $\endgroup$
    – gung - Reinstate Monica
    May 6, 2021 at 14:07
  • 1
    $\begingroup$ Thanks for your comment! I posted an example dataset for R in the edited question, hope that helps. Rater comparison as well as rater performance would be of interest, but the second (rater performance vs. truth) is the focus. Yes, intervals can be treated as equal. $\endgroup$
    – Adrian Mak
    May 6, 2021 at 15:32
  • 1
    $\begingroup$ Similarity of distribution does not imply similarity of values: it is easy to think of examples that disagree strongly but have identical distributions. Box plots don't work well with data like this where ties are very common. $\endgroup$
    – Nick Cox
    May 6, 2021 at 15:59

1 Answer 1

Reset to default
2
$\begingroup$

One approach is a scatter plot matrix with jittering of points given over-plotting of identical values.

enter image description here

Refinements might include, depending partly on taste:

  1. Bubble plots, where bubble size encodes frequency.

  2. Hinton- or Bertin-type twoway bar charts, where bar height encodes frequency.

In this case visual patterns can be supported by measures of agreement (not correlation) such as Lin's concordance correlation:

Rating_1 Rating_2       0.633 
Rating_1 Rating_True    0.630
Rating_2 Rating_True    0.839
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you for your reply! The scatter plot matrix seems promising, would you mind sharing the R function (if used) that you used to create these? Also thanks for showing the weakness of BoxPlots for this application. $\endgroup$
    – Adrian Mak
    May 6, 2021 at 16:58
  • $\begingroup$ Nevermind, a short google gave the answer :) I also try out the other visualizations you mentioned. $\endgroup$
    – Adrian Mak
    May 6, 2021 at 17:07
  • $\begingroup$ I used Stata but the R code for this should not be esoteric. $\endgroup$
    – Nick Cox
    May 6, 2021 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.