I am trying to compare the efficacy of two devices. Three different operators tested each device for a parameter of interest (the same 3 operators for each device) and yielded two data sets for each device. My first thought was to perform a simple two sample t-test on the data sets; however, it occurred to me that this would be ignoring which data was collected by which operator. It is suspected that the devices perform differently depending on who is using them and I would thus like to be able to mitigate this additional source of variability by blocking the data based on which operator collected it. After doing some research it appears that ANOVA can take into account such blocking. My two questions are:

  1. Am I correct that ANOVA is the appropriate test to use in this case?

  2. Is ANOVA always going to be more powerful than a two-sample t-test in this case or would I be just as well off simply performing the t-test?

Just to be extra clear, my goal is not to assess the degree of variability introduced my using multiple operators but simply to remove it as a confounding variable and only compare the efficacy of one device to the other.

  • $\begingroup$ I don't quite follow this. If you have 3 different operators who use each device, why don't you have 3 datasets instead of 2? $\endgroup$ Commented Dec 7, 2015 at 18:37
  • 2
    $\begingroup$ @gung Data set 1 contains data for device 1 which was collected by operators 1, 2, and 3. Data set 2 contains data for device 2 which was also collected by operators 1, 2, and 3. I could further divide the data so that I have 6 data sets for each device and each operator and then perform 3 two sample t-tests (one for each operator), but I believe that would increase the likelihood of type I error, correct? That would also be undesirable as it would be possible for one test to reject the null and the other two not reject it or vice versa. $\endgroup$
    – David Webb
    Commented Dec 7, 2015 at 19:29

1 Answer 1


Using terminology from design of experiments, you seem to have a randomized block experiment (assuming suitable randomization was done) where the blocks are defined by the operators, and you have two treatments (devices). If you ignore the blocks, then the one-way anova will be equivalent to a t-test. Your two questions:

  1. Anova seems to be the appropriate analysis, yes.

  2. If there are differences between the operators (as would often be expected), anova will be more powerful. If there are no differences between the operators, not ... but simply a t-test would be seen as a risky analysis, as the conclusions would be conditional on there not being differences between operators.

See also Understanding the advantages of CRD experiments.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.