I have a cupcake factory. There are an infinite number of cupcake machines. We also have 5 taste testers. These taste testers claim to be in complete agreement over how they rate cupcakes. They claim that given a cupcake, they will be able to judge how good the cupcake machine is and not disagree with each other.

We will take 100 machines and pull 3 random cupcakes from each. Then each tester will rank the cupcake on a scale of 1 to 4:

MachineID, CupcakeSampleID (1,2,or 3), TasterID (1 - 5), Rating

What is the best way to quantify how much in agreement these taste testers are? I was thinking wilcoxon for 2 paired samples, but after doing some more reading I am afraid there are too many ties with a 1-4 scale.

Any advice would be appreciated. I am only concerned about the objectivity of the tester and not the quality of the cupcake machines.

I am sorry that this is a made-up analogy representing a real-world situation, and I do not in fact have any cupcakes. I feel bad for suckering you into reading this. Not really. But seriously, thanks in advance.

  • 1
    $\begingroup$ You have trivialized your real problem by couching it in this form. According to your hypothesis of perfect agreement among testers, any diffference in ratings should cause you to reject the null; otherwise, you cannot reject the null. If you would like to increase your chances of getting answers that are appropriate, useful, and correct, then please explain what your problem actually is. $\endgroup$
    – whuber
    Commented Jul 8, 2016 at 13:52
  • $\begingroup$ Of course I trivialized the real problem. The real problem is conveyed at its simplest form which makes it much easier to talk about without getting confused by specifics which don't matter. $\endgroup$
    – Josh
    Commented Jul 9, 2016 at 18:24
  • 2
    $\begingroup$ The problem is that you appear to have simplified away some essential aspects of your actual problem, to the point where correct answers to the question you have posted (such as the trivial one I gave in my comment) will have little use in the actual situation. As it stands, if you are not satisfied with rejecting the null upon observing any difference in ratings, then you need to edit your question to include the reasons why that would not be a good procedure in your case. $\endgroup$
    – whuber
    Commented Jul 11, 2016 at 15:17
  • $\begingroup$ I am more interested in quantifying the disagreement between the human rankings, as that way I can compare an alternative ranking if necessary. I was considering the MSE between all the pairs but surely some error is expected due to randomness. Does that help clarify? More interested in Quantifying than rejecting the null hypothesis. Or perhaps the hypothesis isn't that they are exactly the same but just similar. $\endgroup$
    – Josh
    Commented Jul 11, 2016 at 16:16
  • $\begingroup$ If you want to quantify agreement, you can look into Cronbach's alpha. $\endgroup$ Commented Mar 20, 2017 at 15:18

1 Answer 1


You may want to consider using an intraclass correlation (ICC) for this purpose. At its heart, this is an analysis of variance, like an ANOVA. Basically, the idea is to see what proportion of the total variance can be attributed to the CupCake ID. To the extent that there are other sources of variance in your data (such as differences between raters), the ICC will drop. It can also be interpreted as the correlation among ratings of the same cupcake --- if your raters always perfectly agreed, this would be 1.

The psych package in R provides a nice ICC function, and excellent help documentation that briefly describes not only the function, but a more general explanation of how ICCs work and what they mean.

An example in R

First, here's some made-up data matching the structure you describe. I've created a set of "true" ratings for each cupcake. The ratings from each rater mostly match the true ratings, but for each rater, I replaced a random 1/3 of their ratings with a new random rating, so that there is some disagreement.


true_ratings <- sample(1:4, 300, replace = TRUE)

Rater1 <- true_ratings
Rater1[sample(1:300, 100)] <- sample(1:4, 100, replace = TRUE) # replace 1/3 with random ratings
Rater2 <- true_ratings
Rater2[sample(1:300, 100)] <- sample(1:4, 100, replace = TRUE) # replace 1/3 with random ratings
Rater3 <- true_ratings
Rater3[sample(1:300, 100)] <- sample(1:4, 100, replace = TRUE) # replace 1/3 with random ratings
Rater4 <- true_ratings
Rater4[sample(1:300, 100)] <- sample(1:4, 100, replace = TRUE) # replace 1/3 with random ratings
Rater5 <- true_ratings
Rater5[sample(1:300, 100)] <- sample(1:4, 100, replace = TRUE) # replace 1/3 with random ratings

data <- data.frame(MachineID = rep(1:100, each=3),
                   CupCakeID = 1:300,
                   Rater1 = Rater1,
                   Rater2 = Rater2,
                   Rater3 = Rater3,
                   Rater4 = Rater4,
                   Rater5 = Rater5)

Here's what the first 6 lines of that look like:

> head(data)
  MachineID CupCakeID Rater1 Rater2 Rater3 Rater4 Rater5
1         1         1      2      2      2      2      3
2         1         2      1      1      1      1      3
3         1         3      4      3      3      3      3
4         2         4      4      4      4      4      4
5         2         5      1      3      2      3      3
6         2         6      2      3      4      2      2

I am only concerned about the objectivity of the tester and not the quality of the cupcake machines.

If you really don't care about the machine and you're confident the machines are all interchangeable, you can discard it from the model completely and just look at cupcakes and raters.

ICC(data[, 3:7]) # just using the rater columns, each row is one cupcake

This provides the output for 6 different versions of the ICC.

Call: ICC(x = data[, 3:7])

Intraclass correlation coefficients 
                         type  ICC   F df1  df2 p lower bound upper bound
Single_raters_absolute   ICC1 0.43 4.8 299 1200 0        0.38        0.49
Single_random_raters     ICC2 0.43 4.8 299 1196 0        0.38        0.49
Single_fixed_raters      ICC3 0.43 4.8 299 1196 0        0.38        0.49
Average_raters_absolute ICC1k 0.79 4.8 299 1200 0        0.75        0.83
Average_random_raters   ICC2k 0.79 4.8 299 1196 0        0.75        0.83
Average_fixed_raters    ICC3k 0.79 4.8 299 1196 0        0.75        0.83

 Number of subjects = 300     Number of Judges =  5

The correct ICC to interpret for your situation depends on how you're thinking about the data generating process. If you want to consider your raters as a random effect, then the ICC2 (either Single_random_raters or Average_random_raters) makes the most sense in your case.

The ICC function is handy, but it really is just a ratio of variances, so it's not hard to calculate by hand if you want to allow for a different model specification. For example, you may decide you want to allow for the possibility of variance in ratings due to machine. To flexibly calculate ICC for whatever model structure you need, use the variance estimates from a mixed effects model. Here is an accessible discussion of ICCs calculated from mixed effects models, a useful answer here on SE, and several examples of ICCs calculated from mixed effects models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.