# Wine tasting: are point totals significant?

193 participants in 19 groups of varying sizes tasted 6 wines. Each participant then distributed 6 points among the wines in any way she or he liked, e.g., $(0,0,6,0,0,0)$ or $(1,1,1,1,1,1)$ or $(0,3,0,2,0,1)$.

Point totals in groups a-s for wines A-F are as follows (N denotes each group's size):

   N  A  B  C  D  E  F
a  6  4  3  0 10 10  9
b 28 62 19 21 22  2 42
c 12 24  5 19  3 12  9
d  9  8  5  9 11  6 15
e  6  1  0  5  9 21  0
f  6 10  0  6 11  2  7
g  6  5  7  4  6 14  0
h 17  8 33  1 23 17 20
i  9  2 11 10  3 23  5
j 17 12 16 22 12 15 25
k  2  7  0  0  0  2  0
l  4 11  0  0  8  0  5
m 13 12  2 30 10 12 12
n 10 11  6 13  2 25  3
o  6  0  0 15 18  3  0
p  7  2 20 17  0  3  0
q 14 22 20 17  7 13  5
r 16  7 11 35 16 25  2
s  5 21  2  0  0  6  1


Overall, wine A has the most points:

  A   B   C   D   E   F
229 160 224 171 211 160


However, even if people assigned their points totally at random, some wine would simply by chance have the highest point total. How can we assess whether wine A's total is significantly larger than would be expected by chance?

Note: this question came to me via a mailing list, and I found it interesting enough to post it here along with the answer I came up with. Any other takes on this are appreciated. And yes, I know that there are 3 points missing, but I don't think this makes all that much of a difference. Data in R format below.

wine <- structure(list(
N = c(6, 28, 12, 9, 6, 6, 6, 17, 9, 17, 2, 4, 13, 10, 6, 7, 14, 16, 5),
A = c(4, 62, 24, 8, 1, 10, 5, 8, 2, 12, 7, 11, 12, 11, 0, 2, 22, 7, 21),
B = c(3, 19, 5, 5, 0, 0, 7, 33, 11, 16, 0, 0, 2, 6, 0, 20, 20, 11, 2),
C = c(0, 21, 19, 9, 5, 6, 4, 1, 10, 22, 0, 0, 30, 13, 15, 17, 17, 35, 0),
D = c(10, 22, 3, 11, 9, 11, 6, 23, 3, 12, 0, 8, 10, 2, 18, 0, 7, 16, 0),
E = c(10, 2, 12, 6, 21, 2, 14, 17, 23, 15, 2, 0, 12, 25, 3, 3, 13, 25, 6),
F = c(9, 42, 9, 15, 0, 7, 0, 20, 5, 25, 0, 5, 12, 3, 0, 0, 5, 2, 1)),
.Names = c("N", LETTERS[1:6]), row.names = letters[1:19], class = "data.frame")


The null hypothesis is that all wines taste subjectively equally well, i.e., participants assigned their points at random.

Now, we could in principle get a handle on the null distribution of the largest (and the second largest etc.) point total analytically via . However, we would still need to account for the dependency in the data, since a point someone assigns to wine A is not available for wines B-F any more.

It's easier to simulate. ("Computers are cheap, and thinking hurts", which I have seen ascribed to Uwe Ligges.) We simulate the tasting many times, say 2,000 times. In each case, we randomly assign each participant's points to each wine, and we note the point totals of the top, second etc. place wine. Which wine this is in each case is irrelevant under the null hypothesis.

Finally, we insert the observed 229 points for the top wine into this simulated distribution of the first order statistic and get a p-value: the proportion of simulated first order statistics exceeding 229. We can do this for all six wines, calculating one-sided tests in each case.

Here is a beanplot of the simulated point totals for the first, second etc. wine, along with a horizontal line for the unconditional expected score of 193, and with the actual sorted point totals inserted as red dots:

It turns out that each total lies in the tails of the corresponding empirical distribution and is significantly larger (or smaller) than what we would expect under the null hypothesis.

1 : p = 0.0135
2 : p = 0
3 : p = 0.0015
4 : p = 0
5 : p = 5e-04
6 : p = 0.022


Actually, this simulation approach also allows us to test other quantities of interest. For instance, here is a histogram of the difference in point totals between the first and the second placed wine, along with a red vertical line indicating the actually observed difference of 229-224=5:

We see that this difference is not significantly different from the null, p = .59, so we cannot reject the null hypothesis that the first- and second-placed wines taste equally well.

n.sims <- 2000
n.participants <- 193
results <- sort(colSums(wine)[-1],decreasing=TRUE)
n.wines <- length(results)

set.seed(1) # for reproducibility

results.sim <- matrix(NA,nrow=n.sims,ncol=n.wines)
pb <- winProgressBar(max=n.sims)
for ( ii in 1:n.sims ) {
setWinProgressBar(pb,ii,paste(ii,"of",n.sims))
# randomly allocate as many points as there are wines:
points <- t(sapply(1:n.participants,function(xx)sample(1:n.wines,n.wines,replace=TRUE)))
# tabulate these points after sorting (because we are interested in order statistics!)
results.sim[ii,] <- sort(table(points),decreasing=TRUE)
}
close(pb)

library(beanplot)
beanplot(data.frame(results.sim),what=c(0,1,0,0),col="lightgray",border=NA,xaxt="n")
abline(h=n.participants,lty=2)
axis(1,1:n.wines)
points(results,pch=19,col="red")

for ( jj in 1:n.wines ) {
cat(jj,": p =",
if(jj<=n.wines/2){1-ecdf(results.sim[,jj])(results[jj])}else{ecdf(results.sim[,jj])(results[jj])},
"\n")
}

diff.1.2.sim <- results.sim[,1]-results.sim[,2]
hist(diff.1.2.sim,breaks=seq(min(diff.1.2.sim)-.5,max(diff.1.2.sim)+.5),col="lightgray",xlab="",main="")
abline(v=results[1]-results[2],lwd=2,col="red")
1-ecdf(diff.1.2.sim)(results[1]-results[2])


I would say there is no statistical difference in the average scores of the wines. But it must be said (discussion in the last alinea) that this does not need to mean that the wines are not different. Different wines score (significantly) very different in different groups (see for instance the score of wine F in group b and the score of wine B in group h).

What worries me is the discrepancy between the groups 'a' to 's', and I imagine that the scoring has not been consistent. In that case, the comparison of the high scores of wines A, C, and E with a model where they get random scores may not be entirely fair. This way of scoring may not reflect the actual distribution of the scores when the wines are equal.

• Possibly the discrepancy is large because, even when the wines are similar the wines will not get random independent points. In the extreme case a judge assigns all of the six points to a random single wine. In this case the inconsistency is only apparent and the large differences between groups might be due to the random behaviour of the judges being very coarse.

points <- t(sapply(1:n.participants,function(xx)sample(1:n.wines,1,replace=TRUE)))
results.sim[ii,] <- 6*sort(table(points),decreasing=TRUE)


With such scoring the observed extremes are not so 'special' anymore.

• Another reason for the inconcistency could be that the scoring of the judges in the same groups is correlated. For instance the judges have somehow similar taste or possibly the wines score different among different groups due to different bottles or other variations that occur from serving to serving.

A more realistic simulation might be to use the actual data from either the 193 judges or (since we do not have that data) the grouped data in order to simulate the distribution of the scores. Then the simulation is done by some resampling method.

n.groups <- length(wine[,1])
results.sim <- matrix(NA,nrow=n.sims,ncol=n.wines)
for ( ii in 1:n.sims ) {
points <- t(sapply(1:n.groups,function(xx) sample(as.numeric(wine[xx,-1]))))
results.sim[ii,] <- sort(colSums(points),decreasing=TRUE)
}


Then analysis will look more like:

The observed variations of the max and min scores are consistent with there being no differences (on average) between the wines.

There might actually be large differences between the wines (considering the lack of consistency between the groups this might be likely even when the average scores shows no big difference). It is only the average score that has no difference. This is in my opinion a problem with many competitions that are based on esthetics. There is no good way to define a ranking when wines score differently among different people, and the only score that gets 'published' is are very simplistic order or score that is greatly influenced by the subjectivity of the judges and may not have any bearing on the tastes of the general public and especially subgroups and individuals.