Power analysis for post hoc test of ANOVA with many groups

In both G*Power and the R pwr package, estimated sample size required per group decreases as the number of groups increase. This seems somewhat counter-intuitive.

For a toy example, assume that I have two groups with a meaningful difference in their mean estimates (Group A and Group B). If I add several additional groups that have means identical to the grand mean of both groups (Groups C1, C2, C3, ...), the power analysis suggests smaller samples from Group A and Group B are needed -- which should make my ability to detect differences in those two groups weaker. At an extreme, if I enter 1500 levels of a single factor (f = .25, b = .8, a = .05), both programs effectively tell me to have group sizes of 2-3.

My understanding is that the power analyses from both programs helps you assess the power of the ANOVA overall. Thus, in the toy example, I'm more likely to pick up a difference between Group A or B and one of the Group Cs . However, this seems like it's a result of an increase in the number of comparisons and the likelihood that some of the Group C samples include mean estimates that are outliers. That doesn't seem like the type of difference I want to pick up.

What is the recommended approach in these circumstances -- or is the "low" sample size per group correct? Since I'm concerned with post-hoc comparisons of the group means, are there a priori power analyses available for those tests?

Because there will be many more error degrees of freedom, you should see an increase in the $A$ vs $B$ rejections as well as $A$ or $B$ vs $C_i$ rejections, because observed differences of a given number of standard errors in size are much less likely to be due to noise in measuring the standard deviation.

For example, imagine that the common error variance, $\sigma^2=1$.

Then the distribution of the estimate of $\sigma^2$ is quite skewed (and spread out) when there's just $A$ and $B$, but as you add more $C$ groups you get a very much stronger idea of the variance, and this will on average improve your ability to tell A and B apart: (This assumes half the groups have 2 observations and half have 3 observations)

That bulge in the left tail of the green density below 1 means you get large F's when $H_0$ is true quite often (because you're dividing by a small number more often). As a result, you need a big F to be confident that it's not just random variation.

That's why the 5% critical value for an F(2,3) (i.e. the A vs B alone comparison) is 9.55, while that for an F(2,150) (i.e. only considering A vs B with 98 "C" groups helping to determine $\sigma^2$) is 3.06.

That effect is part of why you don't need many observations per group.

You should further note that if the $C$ groups have population mean intermediate between the $A$ and $B$ groups, then you should reject the null because of B-C and A-C differences. You seem to think that shouldn't happen. That's simply untrue. It ought to happen (though much less often for any particular $A-C_i$ or $B-C_i$ than for $A-B$).

Simulation is a useful tool to see which rejections occur more often as you add groups.

I imagine that with many groups and only a few observations per group, A vs B rejections will eventually become a relatively small proportion of the total rejections, but it's only $C_j$ vs $C_k$ rejections that are incorrect decisions.

• +1, this is a good answer. Another couple of points to note is that the total N is growing in this scheme (which is generally associated w/ more power intuitively), and you have more group means, so you should get an increasingly clear picture of the b/t group variance as well. – gung May 13 '14 at 15:38
• Thanks! If I'm reading your response correctly, by adding C groups, we should increase our power to distinguish A and B specifically through more precise estimates of common variance; although we lose some (maybe more) power through increased error degrees of freedom. My continued concern is that these significant pairwise differences with small group sizes are likely to be based off point estimates that are "far from the truth." I suppose that's just part of hypothesis testing vs. parameter estimates. I'll think more deeply about your response, but I wanted to send my thanks! – Power Smurf May 13 '14 at 18:06
• @PowerSmurf, you don't lose power from increased error degrees of freedom. Increasing error df increases power. – gung May 13 '14 at 19:23
• Indeed, the point in gung's comment is pretty much the purpose of the analysis I gave above - adding df to the error makes the estimate of the error variance much more certain (less variable), which increases power by making the differences that do exist harder to see as explainable by random variation. – Glen_b May 13 '14 at 22:52