How to calculate power (or sample size) for a multiple comparison experiment?

Question

CrossValidated Folks

I collected data on 20 groups (of 30 elements each). A multiple comparison procedure (pairwise t test with Holm correction) shows that in general there are three set of groups: the high with 4 groups, the low with 2 groups, and the middle with the remaining 14 groups. Each set is not significantly different for the groups within but it is significantly different with the groups in the other sets. (This is a simplification, because there are some other significative and non-significative results for the extremes of each set, but I have to make a simplification of the results so I can write a concise summary of the experiment both to you and to the readers of the paper).

If this result is going to be used for decision making, for example treating members of groups the middle set as equivalents, one must be sure that the results are "real" and not just due to the small sample size.

Thus I need to calculate some measure of power (power = 1- the probability of accepting H0 when it is false) or some measure of sample size to show that either a new experiment with larger sample size is needed, or that indeed the differences are "probably true".

But statistical power of WHAT?

a) it is not of the whole 20 groups ANOVA, since that one rejected H0.

b) should I run the ANOVA of the 14 groups in the middle set and calculate the power of that? But that seems will overestimate the power (or underestimate the needed sample size) since the extreme groups in the middle set are "almost" different.

c) should I calculate the power for the worse pairwise t-test in the middle group (with a Bonferroni corrected alpha)? But that will terribly underestimate the power since the two most similar groups are very likely "really" not different.

Any ideias? Any references I can follow?

What I know so far:

A) the R package pwr calculates the power or sample size for t-test, one way ANOVA, and other tests

B) On the relative sample size required for multiple comparisons, by WITTE, ELSTON AND CARDON discusses the used of the Bonferroni corrected alpha values in the calculations of sample size for multiple comparisons

Thanks

jacques

Answer 1

I came upon your post and I really didn't know if my answer will be of any help to you because it would actually require to reconsider the whole analysis. However: your problem appears to be purely related to data-mining since you are actually trying to uncover the number and centers of latent clusters of individuals within your data without supervision (i.e. without an outcome variable that would easily allow you to separate these categories of individuals). There are several methods you might want to consider for unsupervised classification. Most of these will allow you to estimate the number of groups, the centroid as well as a measure of robustness and uncertainty. You can look at k-means methods and hierarchical clustering methods for a start. You may also consider visualizing your data using principal components before you consider statistical inference.

Answer 2

If you have already done the experiment then there is little point in doing any power analyses. Where the P-values are small the power for the observed effect size and variability was large enough. Where the P-values are large then the power was small for the observed effect size and variability. Power analysis is useful for planning experiments, but not useful after the fact. See this paper by Hoenig & Helsey: http://www.tandfonline.com/doi/abs/10.1198/000313001300339897#preview

Your desire for a power analysis appears to be based on this statement "one must be sure that the results are 'real' and not just due to the small sample size", and so it is useful to consider it closely. Firstly, statistical analysis cannot tell you about the reality of a result--something that you probably know, given that you put the 'real' in quotes. Second, you imply that a small sample is more likely to yield a false positive result, when the reality is that a small sample is exactly as likely to do that as a large sample. The small sample is more likely to yield a false negative result.

If you want to be confident that the results yield reliable conclusions then you have to consider their nature in light of what is known about the system and, ideally, replicate the parts of the study that are most interesting or surprising. (I acknowledge that a well-judged statistical analysis is more helpful here than a poorly judged one: see Julien Sturnemann's answer for some suggestions.)