Minimum sample size for 1 way ANOVA?

Question

I have tested a group of participants over various time points and I want to investigate whether mean score on each subsequent time point is significantly different to the mean score at baseline (eg. score on day 0 versus day 1, score on day 0 versus day 2 etc). My plan is to perform a repeated measures 1 way ANOVA and the Dunnett's posthoc test to compare the mean scores at day with the basline day score. However, the n number for the group is 6. Is a one way ANOVA appriate with a low n number like this? Is there a minimum sample size for a 1 way ANOVA? If so, is there a better alternative statistic to use?

Answer 1

Described below are three approaches to estimating sample size for completely randomized designs. Note that the procedures differ in terms of the information you must provide.

Approach #1 (requires most information)

To calculate sample size, the researcher first needs to specify:

1) level of significance, α (alpha)

2) power, 1-β

3) size of the population variance, σ2

4) sum of the squared population treatment effects.

In practice, 3 and 4 are unknown. However, you can estimate both from a pilot stud. Alternatively, you might estimate these parameters from previous research.

As an example, let's assume we conducted a pilot study and estimated the population variance and sum of squared population treatment effects. If we let α = 0.05 and 1-β = 0.80, then we can use trial and error to calculate the required sample size. The test statistic you calculate is phi (Φ), where:

Φ = (n^0.5)[(average of squared treatment effects/population variance)], where n is a sample size value. The Φ test statistic can then be used to look up power that corresponds to the sample size in Tang's Charts (citation below).

Approach #2

If accurate estimates of #3 and #4 are not available from a pilot study or previous research, then one can use an alternative approach that requires a general idea about the size of the difference between the largest and smallest population means relative to the standard deviation of the population standard deviation:

μmax - μmin = dσ, where d is a multiple of the population standard deviation. In other words, this approach allows you to calculate the sample size if you wanted to detect a difference between the highest and smallest means that would equal to some multiplier of the pop. standard deviation (whether it's one half, or 1.5, or anything else). For the math for this approach, see Kirk (2013) [I have a PDF].

Approach 3

If you know nothing about #3 and #4 from Approach 1, and are unable to express μmax - μmin as a multiple of pop. standard deviation, then you can use strength of association or the effect size to calculate the sample size. This approach also requires the researcher to specify the level of significance, α, as well as the power, 1 - β.

Remember that the strength of an association indicates the proportion of the population variance in the dependent variable that is accounted for by the independent variable. Omega squared is used to measure the strength of association in analysis of variances with fixed treatment effects, whereas intraclass correlation is used in analysis of variance with random treatment effects.

Based on Cohen (1988), we know that (for strength of association):

ω^2 = 0.010 is a small association

ω^2 = 0.059 is a medium association

ω^2 = 0.138 or larger is a large association

And for effect size:

f = 0.10 is a small effect size

f = 0.25 is a medium effect size

f = 0.40 or larger is a large effect size.

Back to the Approach 3: if we have a completely randomized design with p treatment levels, then we can calculate the sample size necessary to detect any magnitude of strength of association OR any magnitude of effect size (the mathematics are equivalent). Again, if you are interested for working out the math for this approach, I refer you to Kirk (2013).

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum

Kirk, R.E. (2013) Experimental Design: Procedures for the Behavioral Sciences

Tang, P. C. The power function of the analysis of variance test with tables and illustrations of their use.Statist. res. Memoirs. 1938,2, 126–149.

Answer 2

The absolute minimum possible sample size for a one way ANOVA F-test would be one more than the number of groups. It's unclear in your question, but it sounds like you have 6 per group.

(It's not advisable to have so few, but it's possible to do ANOVA in that situation and still have the theory work when the assumptions hold -- though you can't check them.)

If I correctly understand Dunnett's procedure, the minimum would be the same for that, since its pariwise comparisons appear to be based off the same common estimate of $\sigma$ as the ANOVA.

The biggest concerns at really small sample sizes would be very low power and higher-than-usual sensitivity to the normality assumption at very low sample size; if you specify power you may then be able to calculate a minimum sample size for that; similarly I can't really decide how much impact (on significance level or power) from sensitivity to non-normality you can bear.

Answer 3

There is not a minimum sample size for ANOVA, but you might have problems with statistical power which is your ability to reject a false null hypothesis. If the effect size differences between baseline and the other measures is not large enough you may not be able to reject the null. If you reject the Null then no worries. If you fail to reject but you think that there are meaningful differences, try collecting more data to increase your statistical power. If you cannot collect more data and want to know if more data would have made a difference, copy and paste your data thus doubling its size and then re-run the analysis. If the results are significant then you know the problem is low power due to small sample size. You can then report that your results would have been significant if you had more cases, but do not report the results you get from artificially doubling your sample size - this would not be appropriate.