Interpreting Kendall's W (KendallW of descTools) with respect to very similar ratings I have 3 raters that have rated 9 subjects. Similar tests have been done with different raters and different subjects, and I need to compare in how far the different raters have agreed on their respective subjects. I have chosen Kendall's W since I am using ordinal data (1 = do not agree at all, 5 = agree completely). Now, I find it very difficult to understand why the results look like that when I compare them to the violin diagrams that I have created of the ratings.
library("DescTools")
rtr1 <- c(5,5,5,5,5,5,4,5,5)
rtr2 <- c(5,5,5,5,5,5,5,5,5)
rtr3 <- c(3,3,3,4,4,4,4,4,4)
ratings <- cbind(rtr1, rtr2, rtr3)
KendallW(ratings, test=TRUE, correct=TRUE)

Paket ‘DescTools’ wurde unter R Version 3.4.1 erstellt
    Kendall's coefficient of concordance Wt

data:  ratings
Kendall chi-squared = 6.1538, df = 8, subjects = 9, raters = 3, p-value = 0.63
alternative hypothesis: Wt is greater 0
sample estimates:
       Wt 
0.2564103 

In the violin diagram, however, ratings are close to each other.
Now, when I look at another set, the ratings seem to be less close to each other:
rtr1 <- c(3,4,5,3,3,5,3,2,5)
rtr2 <- c(1,3,2,1,3,2,1,1,1)
rtr3 <- c(3,2,3,3,3,3,2,2,2)

However, in this case W is 0.4064815. In other words, there is more agreement in the ratings. How can this be the case given the data? Is Kendall's W a good choice at all here given that ratings are not ranked?
 A: Violin plots won't give you a complete picture of what Kendall's W is looking at.  As I understand it, the concordance among raters has to do with their ratings of each subject relative to their scores for other subjects.
For example, the following data yield a W statistic of 1.  All raters rate the first three subjects as lowest, the next three as middle, and the final three as highest.
library("DescTools")
rtr1 <- c(1,1,1,2,2,2,3,3,3)
rtr2 <- c(2,2,2,3,3,3,4,4,4)
rtr3 <- c(3,3,3,4,4,4,5,5,5)
ratings <- cbind(rtr1, rtr2, rtr3)
KendallW(ratings, test=TRUE, correct=TRUE)

# W = 1

In the following example, Raters 1 and 3 are concordant, and Rater 2 is not, yielding a W of 0.67
library("DescTools")
rtr1 <- c(1,1,1,2,2,2,3,3,3)
rtr2 <- c(3,3,3,3,3,3,3,3,3)
rtr3 <- c(3,3,3,4,4,4,5,5,5)
ratings <- cbind(rtr1, rtr2, rtr3)
KendallW(ratings, test=TRUE, correct=TRUE)

# W = 0.67

In the following example, Raters 1 and 2 are entirely discordant, and Rater 3, with all ties, doesn't change this discordance, yielding a W of 0.
library("DescTools")
rtr1 <- c(1,1,1,3,3,3,5,5,5)
rtr2 <- c(3,3,3,2,2,2,1,1,1)
rtr3 <- c(3,3,3,3,3,3,3,3,3)
ratings <- cbind(rtr1, rtr2, rtr3)
KendallW(ratings, test=TRUE, correct=TRUE)

# W = 0

