Let's have four different running shoes, I want to test which ones are the fastest in terms of the average speed of the runner. I "hire" four different groups of runners (each getting one version of the shoes) of the same size, sampled randomly from the population for an experiment where I let them run and measure their speed.

Question 1: What is the appropriate testing strategy?

(i) Pairwise difference in means of speed between groups with one of the available p-value corrections such as Bonferroni (6 pairs)

(ii) Taking one shoe as the baseline and check whether the others are faster or slower, also applying a p-value correction (3 pairs)

Question 2: How do I tell the winner in case I choose strategy (ii)?

Question 3: How do I determine the neccessary sample size for my experiment?

R-examples are also welcome.

  • $\begingroup$ This looks like a homework problem to me. $\endgroup$ – SmallChess May 4 '17 at 15:03
  • 1
    $\begingroup$ Is this a self-study question? $\endgroup$ – GeoMatt22 May 4 '17 at 15:07
  • $\begingroup$ Haha, I wish I had homework problems like that. Maybe my fault because I thought an example like that will be more understandable. The true interest is general: how to do correct statistical inference in experiments where I have independent groups with different treatments and need to find the winning treatment. $\endgroup$ – František Kaláb May 4 '17 at 15:09
  • $\begingroup$ I would recommend (i) using Tukey hsd if you stick with your between-subject design. However, I suspect the power of a between-subjects design here would be very low and a within-subjects design would be preferable. In that case, use a version of the Bonferroni correction. $\endgroup$ – David Lane May 4 '17 at 18:00
  • $\begingroup$ @David,That is very helpful! Any idea on how to determine sample size correctly? This article says that you can basically just use pwr.anova.test from the pwr package in R for Tukey's HSD test, but it seems to me like that is incorrect. Under ANOVA, I am looking for a sample size which allows me to compare the means for a given alpha, beta and MDE, however, under a procedure where p-values are corrected for family-wise error I assume I need higher sample size to get the same guarantees? $\endgroup$ – František Kaláb May 10 '17 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.