Perhaps I am missing something obvious here, but I can't find an answer to this. Let's say we run 10 individual repeated measures models with factor(time) as fixed effect with 5 separate timepoints and 10 different outcome variables. You would get p-values of time 0 vs 1, 0 vs 2, etc. for each outcome variable. Since you have used 10 models how would you correct for multiple testing? And how would that affect p-values between the individual time points?

  • $\begingroup$ In general you need to correct for multiple testing, with, for example, Bonferroni. But please provide more information about how these data were generated and what you are trying to achieve with your analysis $\endgroup$ – Robert Long Nov 29 '18 at 20:41

Strictly speaking you will need to correct for both the multiple comparison within and between outcomes. However, this may be an overkill at the end. Often what is done it is done in such situations is the following:

  • For each outcome you first test with an omnibus test whether there is any different between the time points. If this omnibus test is significant, then you perform the post-hoc tests to see which time points different with each, correcting for multiple testing within the outcome.
  • Then for the different outcomes, you typically pre-select a couple as your primary outcomes and the rest as secondary outcomes. To be fair, this selection needs to be done a-priori and without looking at the results first. Then you only correct for any multiple testing you have in the primary outcomes (i.e., if you have selected more than one). For the secondary, you report the results without correction. You need to be clear and explicit in your text that you have not corrected for multiple testing in the secondary outcomes. And hence, you need also to be more careful on the formulation of statements based on the results from these outcomes.
  • $\begingroup$ Thank you Dimitris. I have a priori selected one outcome as primary and the rest as secondary. If I then do an F-test I would suspect that P-value should also be corrected. For example with Bonferroni do I correct it with a factor 10 (for 10 F-tests)? Or do I have to increase the factor because of the individual time points? $\endgroup$ – User LG Nov 30 '18 at 10:08

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.