Let's say there is an experiment of determining the best cookie recipe with 4 categorical factors. There are $S$ subjects that eat and rate cookies made using $R$ different recipes. There are $R$ batches of cookies - one per recipe. The thing is that all subjects rate all recipes - each of $S$ subjects eat the total number of $R$ cookies and so we end up with $R*S$ ratings. Also note that all subjects sample from the same batches.

My question is: does this setup violate ANOVA independence assumption since there is no random assignment and everyone samples from the same batches? And if so, what needs to be done to use independent samples ANOVA here AFTER the results are collected?

  • $\begingroup$ If data are normal, this could be analyzed according to a standard ANOVA design, see Answer. I would want to make sure there is no social connection between recipe authors and tasters. Tasters should be given guidelines on how to do scoring, especially if they are not professional tasters. $\endgroup$
    – BruceET
    Jan 11, 2021 at 20:14
  • $\begingroup$ This is an example of a complete randomized block design experiment, a common experimental design. The random part refers to the fact the subjects should sample the recipes in a random order. $\endgroup$
    – Dave2e
    Jan 11, 2021 at 22:57

1 Answer 1


Here are data from a tasting, conducted by a newspaper, of seven brands of chocolate pudding, by five different tasters. As in your Question, each taster rated each brand, resulting in 35 scores. In this case, each score was said to be the sum of several partial scores, so there is some reason to believe scores may be normal.

x = c(7,12,18,10,11,6,5, 15,14,8,6,2,0,6,
      20,10,20,4,13,0,7, 18,16,12,10,8,9,3,
matrix(x, byrow=T, nrow=5)
     [,1] [,2] [,3] [,4] [,5] [,6] [,7]
[1,]    7   12   18   10   11    6    5
[2,]   15   14    8    6    2    0    6
[3,]   20   10   20    4   13    0    7
[4,]   18   16   12   10    8    9    3
[5,]   15   16    9   13    3    5    5

This amounts to a two-way ANOVA with one replication in each of 35 cells (so that there can be no interaction term). Looking at brands of interest as a fixed effect and tasters as randomly chosen, it might be considered a complete block design. (Due to possible quirky taste preferences of various tasters, there may be interaction. But with one observation per cell, we have no way to test for interaction.)

An ANOVA procedure in R finds that there are significant differences among brands but not significant differences among tasters.

If you look at both effects as fixed, the ANOVA model would be:

$$Y_{ij} = \mu + \beta_i + \tau_j + e_{ij},$$ for $i=1,\dots,7$ brands, $j = 1,\dots,5$ tasters and $e_{ij} \stackrel{iid}{\sim}\mathsf{Norm}(\mu=0,\sigma),$ with $\sigma^2$ to be estimated by MeanSq(Resid).

brand=as.factor(rep(1:7, times=5))
tastr=as.factor(rep(1:5, each=7))

anova(lm(x ~ brand + tastr))
Analysis of Variance Table

Response: x
          Df Sum Sq Mean Sq F value   Pr(>F)    
brand      6 580.80  96.800  5.8113 0.000748 ***
tastr      4  55.83  13.957  0.8379 0.514653    
Residuals 24 399.77  16.657                     
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

With appropriate protection against 'false discovery' (e.g., Bonferroni) paired t tests might reveal which brands differ in significant and important ways.

You can look at various help pages online to see how to test whether residuals from the ANOVA model are nearly normally distributed. If you feel that residuals are too far from normal, you might use a nonparametric Friedman test to look for differences among brands.

friedman.test(x, brand, tastr)

        Friedman rank sum test

data:  x, brand and tastr
Friedman chi-squared = 19.321, df = 6, p-value = 0.003654

Paired Wilcoxon (signed rank) tests might be used ad hoc to explore differences among brands.

  • $\begingroup$ Fantastic, thank you! So a complete randomized design would also require tasters to eat and rate cookies in a randomly ordered sequence just as (@Dave2e pointed out)? Or is it enough that the tasters were chosen randomly from the population? I don't really understand the requirements of randomized design $\endgroup$
    – firemelon
    Jan 12, 2021 at 3:00
  • $\begingroup$ Random order of tasting recipes would be ideal. That's usually the way evaluations of wine are done. $\endgroup$
    – BruceET
    Jan 12, 2021 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.