# Comparing means across two groups and over four time points when group sample sizes are very small

I have two groups (experimental, N=6, and control group, N=20). For each participant I measured a score (let say mean reaction time) 4 times. I would like to check:

1. whether these groups differed in the beginning (Time 1)
2. whether the score changes in time (for control group)
3. compare the change in time for both groups

I use R to analyze the data. What statistical tests can I use, given the small group size? I would be very happy for any advice or link. Thank you in advance.

## 1 Answer

### Your analyses

One strategy is to use the same techniques as you would with larger sample sizes.

• You could do a 2 by 4 mixed ANOVA with appropriate contrasts to test your effects of interest.

Or you could split your analyses up into discrete tests

1. T-test for group differences at time 1
2. Repeated measures ANOVA, possibly with linear and perhaps also quadratic contrasts for the effect of change in control group
3. Interaction effect of the 2 by 4 mixed ANOVA (or perhaps a linear by group interaction) for whether the change in time differed between the two groups

### General considerations regarding small sample sizes

• You need to be particularly careful with outliers; and for this reason I have heard some researchers recommend using non-parametric tests with small sample sizes. I'm not completely convinced that this is necessary; I think if you are careful about looking for outliers, then standard tests may be okay. You also need to rely more on prior knowledge when doing assumption tests, because the data itself may be insufficient, for example, to assess whether residuals are normally distributed in the experimental group.
• Do an actual power analysis based on your expected effect sizes. In your case, the comparisons between groups at time 1 is likely to have very little statistical power unless the effect size is huge. So, you should just acknowledge the fact that you may not be able to answer your research question with your data. In contrast, changes over time in reaction time with 20 people might still be quite powerful because of the increased power of repeated measures effects and the fact that you are ignoring the tiny n=6 group.
• Do things that focus your question. For example, a test of the linear effect of the four time points may be more powerful than if you just do ANOVA with time as factor. This is because the linear effect only uses 1 degree of freedom where as treating it as a factor would use k-1, i.e., 3 degrees of freedom. Of course, this is trade off because you might assume that the effect of time is monotonic, but not quite linear.
• Keep the number of statistical tests to a minimum, which it sounds like you are. Tiny sample sizes combined with testing heaps of hypotheses can lead to some awful data dredging. Or if you are being conservative and apply bonferroni corrections or other adjustments, the minimal power that was present gets even lower.
• Try to get more participants.
• If you know that you can't get more participants, consider at the time of study design whether the research is worth doing if it will be insufficiently powered to answer your research question.
• thank you very much for your answers. My preliminary results suggest, there are some differences despite the small group size. Do you think some other techniques like mixed-effect models (with p-values based on MCMC) or some sampling technique (bootstrapping) should be used, or you I stay with ANOVA? I wondered whether these techniques helped to smooth the results from the N=6 group. Jun 7, 2011 at 9:57