I collected data on 20 groups (with 30 elements each). A multiple comparison procedure (pairwise t-test with Holm correction) shows that in general there are three sets of groups: the high with 4 groups, the low with 2 groups, and the middle with the remaining 14 groups. Each set is not significantly different for the groups within but it is significantly different from the groups in the other sets. (This is a simplification, because there are some other significant and non-significant results for the extremes of each set, but I am making a simplification of the results so I can write a concise summary of the experiment both to you and to the readers of the paper.)
If this result is going to be used for decision making, for example treating members of groups the middle set as equivalent, one must be sure that the results are "real" and not just due to the small sample size.
Thus I need to calculate some measure of power (power = 1- the probability of accepting H0 when it is false) or some measure of sample size to show that either a new experiment with a larger sample size is needed, or that indeed the differences are "probably true".
But statistical power of WHAT?
- It is not of the whole 20 groups ANOVA, since that analysis rejected the null.
- Should I run the ANOVA of the 14 groups in the middle set and calculate the power of that? But that seems will overestimate the power (or underestimate the needed sample size) since the extreme groups in the middle set are "almost" different.
- Should I calculate the power for the least significant pairwise t-test in the middle group (with a Bonferroni corrected alpha)? But that will terribly underestimate the power since the two most similar groups are very likely "really" not different.
Any ideas? Any references I can follow?
What I know so far:
- The R package pwr calculates the power or sample size for t-test, one way ANOVA, and other tests.
- On the relative sample size required for multiple comparisons, by Witte, Elston AND Cardon discusses the use of the Bonferroni corrected alpha values in the calculations of sample size for multiple comparisons.
EDIT - Aug 2013
There has been some upvote movement in this question, so I decided to add some more information, or better clarification regarding this topic.
I did not quite agree with the two answers posted. I do not think it is a data-mining/clustering problem. But probably I did not phrase the question correctly. That paper is published so I can not only point to it, here, but also discuss what I needed.
In the paper I (and colleagues) discuss the differences between productivity and citations among different computer science subareas, based on a random sample of 30 researchers in each area. The paper includes a compact letter display that shows the significant differences between any two of the 20 CS subareas. But I wanted to show significant equivalences between the areas. That is when it is very likely that two areas have the same productivity or the same citations per paper, given the 30 sample points for each area.
I know of equivalence tests (or Two One Sided Tests - TOST) - there have been some discussions in CV on that, but nowhere did I see multiple equivalence tests!
My idea was to use power was that the definition of power = 1- the probability of accepting H0 when it is false is exactly what I need to state that two areas have the same productivity - I make the statement that they have the same productivity (H0) and that statement is true with "power" confidence level!
I still do not know how to do that, and the paper has no statement of probable equivalence between some CS areas, which is in fact the more interesting result!
I would again appreciate any comments or help.