1
$\begingroup$

I have performed multiple comparisons using Mann-Whitney U tests, and want to correct the p-values to know which results are worth reporting. The structure of the data is as follows :

2 experiments, in which X different indicators were measured over time (Y time steps) in Z individuals.

The goal is to know if individuals from the 2 experiments differ in their indicators. As measures in each group need to be independent for Mann-Whitney, a test is run per time step. That's why I wonder if the correction should take into account that some tests are linked (different time steps of a same measure), and if yes, how can I do that?

I intend to plot the data when the tests indicate something significant, to check the effect is really there, and will probably only make strong conclusions if several time steps of a same indicator are detected as different between the groups.

I'd also like to know if the preliminary steps I performed are valid :

  • The indicators measured sometimes fall below a limit of detection (LOD), so these will all be treated as ties by Mann-Whitney. So I excluded a priori comparisons for which there was less than 5 measures above limit of detection in each group (if one group had only measures at LOD and the other many measures above LOD, I think it's informative enough to run a test).
  • Among the remaining comparisons, I test for equal variance using a Levene's test. I exclude comparisons with unequal variance (threshold of 5%, I don't correct p-values at this step). Here, cases where one group has only measures at LOD and the other shows variations end up being excluded...
  • I run the Mann-Whitney tests on the remaining comparisons.
$\endgroup$
5
  • 1
    $\begingroup$ Alternatively you can consider fitting an appropriate regression for each Xi with predictors experiment (binary indicator) and time (continuous or ordinal). One challenge would be how to model the association between time and Xi (linear? smooth?). But you'll learn much more from one regression model than from multiple tests. $\endgroup$
    – dipetkov
    Jun 6 at 20:22
  • $\begingroup$ I see what you mean. As the data is very noisy (immune response), I think I'll indeed struggle to find an appropriate type of regression... $\endgroup$
    – alpagarou
    Jun 7 at 8:12
  • 1
    $\begingroup$ Separate tests for each time point won't help you deal with the noise while making sense of the data unless the difference between experiments is "significant" at the same time steps and in the same direction [or insignificant at all time steps]. But it's likely you get significance at some time steps for some metrics and that could be hard to be meaningfully summarised. You'll also have to think about how to correct the p-values for multiple tests, which you won't have to do if you have one model for all the data. $\endgroup$
    – dipetkov
    Jun 7 at 8:29
  • $\begingroup$ @dipetkov is this the kind of thing you were thinking about ? middleprofessor.com/files/applied-biostatistics_bookdown/_book/… In my case I do have a baseline level then indicators followed over time after a "treatment" so I'm thinking I could go with that. But I do have a silly question : are these models appropriate if the measures post-treatment are not linear over time? $\endgroup$
    – alpagarou
    Jun 9 at 11:59
  • 1
    $\begingroup$ Yes, this book suggests to fit a model to all the data rather than to run tests on subsets of the data. It's possible to let the effect of time be smooth (non-linear) using splines for example. If it makes sense for your problem, start simple with a pre-post design to investigate whether there is a difference at the final time point, ie, in the long-term effect of the treatment. $\endgroup$
    – dipetkov
    Jun 9 at 18:51

1 Answer 1

0
$\begingroup$

I don't think running a test per time step is a nice way of doing this. I would instead, in each experiment, average the value of your indicators over all the time steps. Now each individual has 1 value per indicator which is independent as the sample are individuals out of the population. I dont understand why you would test for equal variance as equal variance is not a requirement for Mann-Whitney U, it is nonparametric.

$\endgroup$
1
  • $\begingroup$ For equal variance, I followed a paper by Kasuya 2000 (doi:10.1006/anbe.2001.1691), which shows type I error are inflated when variances are not equal. I don't wish to average indicators over all time steps, as I would lose an important driver of variability. $\endgroup$
    – alpagarou
    Jun 7 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.