I collected data on 20 groups (of 30 elements each). A multiple comparison procedure (pairwise t test with Holm correction) shows that in general there are three set of groups: the high with 4 groups, the low with 2 groups, and the middle with the remaining 14 groups. Each set is not significantly different for the groups within but it is significantly different with the groups in the other sets. (This is a simplification, because there are some other significative and non-significative results for the extremes of each set, but I have to make a simplification of the results so I can write a concise summary of the experiment both to you and to the readers of the paper).
If this result is going to be used for decision making, for example treating members of groups the middle set as equivalents, one must be sure that the results are "real" and not just due to the small sample size.
Thus I need to calculate some measure of power (power = 1- the probability of accepting H0 when it is false) or some measure of sample size to show that either a new experiment with larger sample size is needed, or that indeed the differences are "probably true".
But statistical power of WHAT?
a) it is not of the whole 20 groups ANOVA, since that one rejected H0.
b) should I run the ANOVA of the 14 groups in the middle set and calculate the power of that? But that seems will overestimate the power (or underestimate the needed sample size) since the extreme groups in the middle set are "almost" different.
c) should I calculate the power for the worse pairwise t-test in the middle group (with a Bonferroni corrected alpha)? But that will terribly underestimate the power since the two most similar groups are very likely "really" not different.
Any ideias? Any references I can follow?
What I know so far:
A) the R package pwr calculates the power or sample size for t-test, one way ANOVA, and other tests
B) On the relative sample size required for multiple comparisons, by WITTE, ELSTON AND CARDON discusses the used of the Bonferroni corrected alpha values in the calculations of sample size for multiple comparisons