# P-value correction for multiple Mann-Whitney tests, some of them being dependent

I have performed multiple comparisons using Mann-Whitney U tests, and want to correct the p-values to know which results are worth reporting. The structure of the data is as follows :

2 experiments, in which X different indicators were measured over time (Y time steps) in Z individuals.

The goal is to know if individuals from the 2 experiments differ in their indicators. As measures in each group need to be independent for Mann-Whitney, a test is run per time step. That's why I wonder if the correction should take into account that some tests are linked (different time steps of a same measure), and if yes, how can I do that?

I intend to plot the data when the tests indicate something significant, to check the effect is really there, and will probably only make strong conclusions if several time steps of a same indicator are detected as different between the groups.

I'd also like to know if the preliminary steps I performed are valid :

• The indicators measured sometimes fall below a limit of detection (LOD), so these will all be treated as ties by Mann-Whitney. So I excluded a priori comparisons for which there was less than 5 measures above limit of detection in each group (if one group had only measures at LOD and the other many measures above LOD, I think it's informative enough to run a test).
• Among the remaining comparisons, I test for equal variance using a Levene's test. I exclude comparisons with unequal variance (threshold of 5%, I don't correct p-values at this step). Here, cases where one group has only measures at LOD and the other shows variations end up being excluded...
• I run the Mann-Whitney tests on the remaining comparisons.
• Alternatively you can consider fitting an appropriate regression for each Xi with predictors experiment (binary indicator) and time (continuous or ordinal). One challenge would be how to model the association between time and Xi (linear? smooth?). But you'll learn much more from one regression model than from multiple tests. Jun 6 at 20:22
• I see what you mean. As the data is very noisy (immune response), I think I'll indeed struggle to find an appropriate type of regression... Jun 7 at 8:12
• Separate tests for each time point won't help you deal with the noise while making sense of the data unless the difference between experiments is "significant" at the same time steps and in the same direction [or insignificant at all time steps]. But it's likely you get significance at some time steps for some metrics and that could be hard to be meaningfully summarised. You'll also have to think about how to correct the p-values for multiple tests, which you won't have to do if you have one model for all the data. Jun 7 at 8:29
• @dipetkov is this the kind of thing you were thinking about ? middleprofessor.com/files/applied-biostatistics_bookdown/_book/… In my case I do have a baseline level then indicators followed over time after a "treatment" so I'm thinking I could go with that. But I do have a silly question : are these models appropriate if the measures post-treatment are not linear over time? Jun 9 at 11:59
• Yes, this book suggests to fit a model to all the data rather than to run tests on subsets of the data. It's possible to let the effect of time be smooth (non-linear) using splines for example. If it makes sense for your problem, start simple with a pre-post design to investigate whether there is a difference at the final time point, ie, in the long-term effect of the treatment. Jun 9 at 18:51