What is the correct way to choose tests for pairwise comparison after ANOVA?

Question

I am measuring secretion of insulin upon glucose stimulation of pancreatic islets after various treatments. So I have two independent variables/factors:

1. Treatment (three different compounds and a control treatment)
2. Glucose concentration (low and high)

My sample size is only 3 to 4 individuals per experimental group.

I analyzed the data by two-way ANOVA (using Prism for Mac), confirming that the effects of the two factors are not independent. Now I am unsure about what test I should choose to do pairwise comparisons of each treatment vs. the control.

Prism allows for including Bonferroni's post-hoc tests, after quickly checking the Wikipedia article about ANOVA, I believe Dunnett's test might be appropriate, but I have seen publications using Duncan's test, Tukey's test as well as Student's t test for pairwise comparisons in similar setups.

Having no suitable textbook at hand, and since I didn't find a good descriptive example from the life sciences/biology field online, I am not sure what test is appropriate.

Can someone point me to some material (or recommend a good textbook) that guides the experimenter which (post-hoc/a priori) test to use, and ideally explains why?

Answer 1

When choosing a test you have to consider two important things: A: is the test reliable when the ANOVA assumption have been violated; the question is if the test performs well when the group sizes are different and when the population variances are very different or you have not normally distributed data; B: does the test control over the Type I/Type II error rate; statistical power of a test and type I error rate are very related e.g: you can opt for a more conservative test, aiming small probability of Type I error, but you will loose statistical power. It is a trade-off.

Furthermore, Bonferroni and Tukey test are conservative - high control over type I eror rate bat low statistical power; Games-Howell is powerful but not appropriate for small sample. Games-Howell is accurate when sample sizes are unequal. For all of the you should be careful ANOVA assumptions;

Moreover you said: "My sample size is only 3 to 4 individuals per experimental group." but I think this is not enough when it comes to test ANOVA assumptions.

This is a detailed book on the topic. Andy Field is a great teacher and here has a nice video on post-hoc. Also there and there are relevant documents on your question.

Regarding your question in comment: I can say use this test or this one, but the main idea is that you have to know them well, the difference between them and the trade-off; after this you have to decide for one, two or more, and you have to be able to motivate and explain your decision and all of these in relation with your research and data not with the test 'per se'. Moreover, usually 'to assume' is not ok in statistics...therefore you have to test the normality and all ANOVA assumptions. Further, IMHO ANOVA it's ok but the group size is not ok. Considering your exigencies (in terms of significance level, power, no of groups etc.) you can compute a needed sample size per group ( using R, or using many other free resources on the web). I would like avoid to give you a 'cooked dish' because you wont gain anything, but to not make your life harder I say: if I were you I would use ANOVA, 30 individuals per group (for a 2X3 design you need n~180 individuals), I would use Tukey, REGWQ, and Bonfferoni.

Answer 2

I will recommend you to have a read at BIOMETRY of Sokal and Rholf (old one, but with clear concepts for starting) and Experimental Design and Data Analysis for Biologists by Quinn and Keough. This last one is available in pdf on the web.