I have read up on a number of threads discussing this, but none address my exact situation so I thought I would try my luck with outlining what I'm working with and getting more relevant insight.

I am doing a pilot study using just one group of subjects. This group has undergone several weeks of an intervention, and I've taken outcome measures pre- and post-treatment.

There are quite a number of outcome measures I'm looking at. What I'm trying to determine is whether the treatment has caused a change in these outcome measures from pre- to post-treatment.

What would be the most appropriate test to use here? It seems to me that a repeated measures ANOVA is used for multiple groups (I do not have this). Paired t-tests would do the trick, but I would be running quite a few of them - I imagine there is some problem with running multiple t-tests? Is there a way to correct for them (i.e., would I need to use something like the Holm method to adjust the p-values coming out of all the t-tests)?

Is there some better way to analyse this data that I may not be seeing? Appreciate any insight.


1 Answer 1


You have the right of it. You can do paired t-tests (which are not that different from RM-ANOVAs, actually), or if you have confounder variables you want to control, mixed-effect models with a random intercept for participant ID. A standard way to control for the number of tests is to apply multiple test correction - BH-FDR correction being one of the more popular ones. Is there a better way? Well, it depends on your data and your question.

One approach would be to a priori identify primary and secondary outcomes of interest. First run your analyses on just the primary outcomes, correcting for multiple comparisons. Then run your analyses on all outcomes, including secondary, and correct for multiple comparisons. Probably you aren't equally interested in all outcomes, so this is the approach I'd most strongly recommend.

Alternatively, if some of your outcomes are related, you could think about applying a data reduction procedure. Ideally this would be hypothesis-driven, so you could do a confirmatory factor analysis (CFA) to reduce related outcomes into one. Or you could use exploratory factor analysis (PCA, or one of it's variations) to reduce the data, if you aren't sure which outcomes are similar. But since you have repeated measures, you will want to be careful that you do it correctly.

  • 1
    $\begingroup$ Thank you very much! This is incredibly helpful. $\endgroup$
    – A.R.
    Feb 21, 2023 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.