4
$\begingroup$

I've conducted a four-phase experiment with physiological measures on two groups (Controls vs. Patients). After applying a Mann-Whitney test to examine differences between groups for each dependent variable at each phase, I found some significant results.

However, when I attempted to build a linear mixed model (LMM) in R using the formula model <- lme(Dep_Variable ~ TIME * Group, random = ~ 1|ID , data= data), the LMM results are substantially different from the Mann-Whitney test. Specifically, significant differences observed with Mann-Whitney at certain time points are not replicated in the LMM, and vice versa.

Are there specific considerations in interpreting results obtained from a Linear Mixed Model (LMM) compared to non-parametric tests? What factors could contribute to differences between Mann-Whitney and LMM results?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
8
$\begingroup$

Mann-Whitney tests "the null hypothesis that, for randomly selected values X and Y from two populations, the probability of X being greater than Y is equal to the probability of Y being greater than X". You probably use your LME model to test hypotheses about explained variance, or about parameters being zero. These are very different things, so the answers that different tests give to different questions will also differ.

Best to first formulate your research question, then choose an appropriate test.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The thing is that, as a first step, I would like to see if there are any differences between the two groups for each phase of the experiment. Subsequently, I would look into those differences to see which group has higher values. That's the purpose of building a model with all the four phases, and where I found results to be contrasting $\endgroup$
    – Ed9012
    Commented Dec 10, 2023 at 20:41
  • 2
    $\begingroup$ You will need to be more specific than "any differences between the two groups". Two groups can have the same mean (which is what t-tests test), but if you draw a random sample from each group, it may still be more likely that the observation from group A is greater than that from group B (which is what Mann-Whitney tests). Or the other way around. If you just want to test whether the CDFs differ, then a two-sample Kolmogorov-Smirnov test may be appropriate. Yes, this is complicated, and there is a lot of imprecise and misleading information out there. $\endgroup$
    – Stephan Kolassa
    Commented Dec 11, 2023 at 7:03
6
$\begingroup$

In addition to the differences between Mann-Whitney and mixed linear models, as noted by Stephan Kolassa, part of your problem might be:

After applying a Mann-Whitney test to examine differences between groups for each dependent variable at each phase, I found some significant results.

That sounds like you did multiple comparisons over multiple outcome variables at multiple experimental phases. That leads to a potential multiple-comparison problem: if you do enough comparisons, some will pass a given threshold of "significance" even if there aren't any true differences. If you didn't correct the Mann-Whitney tests for the multiple comparisons you might well have found nominally "significant" differences that aren't real.

Note that the Mann-Whitney test can be considered a special case of ordinal logistic regression, a method that allows you to avoid the multiple pairwise testing you thought that you needed to perform. See Chapter 13 of Frank Harrell's Regression Modeling Strategies.

Finally, if a parametric linear model is appropriate then these data might not require mixed modeling. It sounds as if a classic multivariate (multi-outcome) linear model might be OK, or other approaches like those described in Chapter 7 of Regression Modeling Strategies.

Improve this answer
$\endgroup$
5
  • $\begingroup$ after I correct for FDR or Bonferroni, none of the results are significant. However, the reason why I would use a linear mixed model is to look at the effect of time between the two groups over each dependent variable (so that i build many different models with the same predictors). I don't understand your suggestion on how the MW test can be used to avoid the multiple pairwise testing, could you please write more details? $\endgroup$
    – Ed9012
    Commented Dec 7, 2023 at 21:16
  • 1
    $\begingroup$ @Ed9012 what I meant to convey is that ordinal logistic regression, which allows for complex linear multiple regression models without assumptions about error distributions, can be used instead of MW, which can only compare two groups at a time. See Chapter 13 of the Harrell reference and this resource. If you have complete data, multivariate linear models might provide an alternative to mixed models. That said, there's nothing necessarily wrong with your mixed models; if the assumptions are met I'd trust those results. $\endgroup$
    – EdM
    Commented Dec 7, 2023 at 22:38
  • $\begingroup$ I did not mention though that my dependent variables are continuos. I see how I can use multivariate linear models, building models like the following: model<-lm(Dep_Variable~Group,data=TIME1). In this way I can still see if there are any differences between the group for each phase for multiple variables, though I need to split the original dataset several times. Does that sounds right? $\endgroup$
    – Ed9012
    Commented Dec 7, 2023 at 23:38
  • $\begingroup$ I mean, that would still be a simple linear regression... I don't see how I can apply the techniques you mentioned. I did Mann-Whitney because the data are not normally distributed. $\endgroup$
    – Ed9012
    Commented Dec 7, 2023 at 23:53
  • $\begingroup$ @Ed9012 you typically should not do separate analyses for each time point/phase. That tends to have less power than a comprehensive model of all the data. The usual normality assumption in multiple linear regression is about the error distribution estimated by residuals between observations and model predictions, not the raw data. Even that assumption can sometimes be relaxed. See this answer. Ordinal regression doesn't even need that assumption; it just works with the rank-order of observations as Mann-Whitney does. $\endgroup$
    – EdM
    Commented Dec 8, 2023 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.