7
$\begingroup$

I am trying to visualize the process of two raters who have each rated the same dataset. Each row (line in the plot) represent a heart beat (or an error) on an ecg. Every disagreement was discussed and commen rating was agreed upon. The agreed ratings are stored in the variable common.

I want to show frequent patterns in this agreement process.

My dataset contains ~1000 disagreements(from ~20000 ratings). Each ratings is one of 6 categories. The categories are not ordered, but d, e and f represent different types of heart beats (incl. unknown), while a, b and c are other ECG patterns.

My initial idea was a parallel plot connecting each rating in for rater1, rater2 and common:

This gives the general idea, that there are som major patterns, but it is not easy to interpret.

Im am hoping someone can recommend a better solution.

Slightly modified sample of the data:

rater1,rater2,common
f,d,e
c,b,b
f,a,a
d,e,e
d,f,f
d,f,e
f,d,c
f,d,e
b,c,c
d,e,e
c,b,b
d,b,b
d,f,e
d,e,e
f,e,e
f,e,e
b,c,c
f,e,e
d,f,e
f,d,e
b,c,c
d,e,e
f,d,e
c,f,c
f,e,e
f,d,f
f,e,e
f,e,e
d,f,e
d,f,f
f,d,e
f,e,e
c,f,c
f,e,e
c,f,c
f,d,e
f,d,f
c,f,c
d,f,e
d,e,e
f,e,e
b,c,c
c,f,c
f,e,e
f,d,e
f,e,e
b,c,c
f,e,e
f,d,f
e,f,e
Improve this question
$\endgroup$
7
  • 4
    $\begingroup$ I will admit to being confused by this data. It might help to provide more information about what is being rated, e.g., is it really a classification into one of 6 types or is it a scale of some kind? Is it possible to convert it to an ordinal scale? Also, the "common" rating is pretty ambiguous. It seems like it's obvious to you what it means and refers to, but may not be as obvious to us, the readers. Clarification of these things should facilitate obtaining suggestions. $\endgroup$
    – Mike Hunter
    Nov 19, 2015 at 11:53
  • 3
    $\begingroup$ I think this visualization is really good! (What is the distinction between red and white lines?) I see patterns of meeting in the middle, changes from rater 1 to meet rater 2 (f -> c), and a cluster the other way (e -> f). I also see there is a cluster where each gave a d but them in common switched to a f! $\endgroup$
    – Andy W
    Nov 19, 2015 at 12:06
  • 2
    $\begingroup$ There may be simpler tables or aggregations to present the data in a more diluted form, but I always appreciate when one just shows all of the data, and IMO your parallel coordinate plot here is hard to beat in terms of being informative. $\endgroup$
    – Andy W
    Nov 19, 2015 at 12:07
  • $\begingroup$ @AndyW That is exactly what i want to show. I'm also adding a table of the frequency of the most common patterns. Red is where the raters choose a common rating, which was not choosen by either initially. This means that the last pattern you described is not a correct interpretation, as what you describe would have appered red. $\endgroup$
    – JohannesNE
    Nov 19, 2015 at 12:17
  • 2
    $\begingroup$ I agree with @DJohnson in that I don't follow the rules here. For example d,d,c (if I understand it correctly) is a case where the raters agreed on d and then changed their minds to c. Please explain. Was every agreement discussed too? I have to disagree on the parallel coordinate plot: it is too noisy to help much and the jittering that is a good idea for scatter plots seems less successful here. I would start with a tabulation of the triples. In principle there are $6^3 = 216$ distinct possibilities, but in practice there could be many fewer. $\endgroup$
    – Nick Cox
    Nov 19, 2015 at 12:21

1 Answer 1

Reset to default
1
$\begingroup$

This isn't as sexy as your plot, but it might make it easier to read off actual frequency data.

ratings

Simulated data -- color indicates the number of raters who initially agreed with the eventual common rating (i.e. 0, 1 or 2).

Code:

library(ggplot2)

theme_set(theme_bw())
theme_update(strip.background=element_rect(colour="white"))
theme_update(panel.border=element_blank())

## Simulated data with similar format
n <- 10^3
df <- data.frame(common=sample(letters[1:6], size=n, replace=T, prob=c(1, 2, 3, 4, 4, 4)),
                 stringsAsFactors=F)
df$rater1 <- ifelse(runif(n) < 0.5, df$common, sample(letters[1:6], size=n, replace=T))
df$rater2 <- ifelse(runif(n) < 0.5, df$common, sample(letters[1:6], size=n, replace=T))
for(var in c("common", "rater1", "rater2")) {
    df[, var] <- factor(df[, var], levels=letters[1:6])
}
df$rater1_label <- sprintf(ifelse(df$rater1 == "a", "rater1 = %s", "%s"), df$rater1)
df$rater1_label <- factor(df$rater1_label, levels=c("rater1 = a", letters[2:6]))
df$rater2_label <- sprintf(ifelse(df$rater2 == "a", "rater2 = %s", "%s"), df$rater2)
df$rater2_label <- factor(df$rater2_label, levels=c("rater2 = a", letters[2:6]))
df$agree_with_common <- as.character(1*(df$rater1 == df$common) + 1*(df$rater2 == df$common))
p <- (ggplot(df, aes(x=common, color=agree_with_common)) +
      scale_color_manual("", guide=F,
                         values=c("0"="#D55E00", "1"="#0072B2", "2"="#009E73")) +
      xlab("common rating") + ylab("count") +
      geom_histogram(fill="white") +
      facet_grid(rater1_label ~ rater2_label) +
      ggtitle("Histogram of common rating conditional on individual ratings"))
p
ggsave("ratings.png", p, width=10, height=8)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.