2
$\begingroup$

I'm running a Mixed effects model ANOVA with two fixed factors (condition, repetition) and one random factor (subject). Subsequently, a Tukey multiple comparisons test is performed.

Now I'd like to plot the means and standard errors (SEMs) of the single conditions in a single error bar plot, and report the p values between the conditions.

The problem: while in the Tukey test, I got significant differences and non-overlapping SEMs between certain means, for my plotted real/observed data the SEM bars overlap.

This is now counterintuitive, since commonly you would assume that in the case of overlapping, the means are not significantly different.

My question is:

  • is the difference between estimated marginal means and observed means due to having a random factor in my model, or what is the reason for the discrepancy?

  • how would you report the data? Would you still plot observed data with the p values and state that the p values are derived from the estimated model? Or would you plot estimated means and standard errors?

Thank you!

EDIT: I'm adding the multiple comparisons result for a sample case as well as the observed means and standard error plot in case this helps.

Observed means and standard errors

Multiple comparisons result

$\endgroup$
2
$\begingroup$

Statistical significance is not transitive. If you want to say how much error there is in estimating the means, show error bars around the means. If you want to compare the means, show results of multiple comparisons. Don't mix those two ideas together. It is quite possible - especially in mixed models - that means can have similar standard errors, but comparisons among the means have radically different standard errors.

$\endgroup$
  • $\begingroup$ Thank you very much for your suggestion, this is helping a lot. In the specific case, would it be a valid choice to first show the observed means along with standard errors or confidence intervals, followed by a table with the comparisons results? I.e. given that you state it is common for mixed models yielding different standard errors for the comparisons. Intuitively I would say showing the "real" data of the experiment succeeded by the model / post hoc results sounds reasonable, but any comment on this is much appreciated. $\endgroup$ – user54643 Sep 8 '14 at 18:36
  • $\begingroup$ Right. Show the means and their SEs or CIs separately from the comparisons. For an easy example of widely differing SEs, think about the cell means for A*B, where A is between-subjects and B is within-subjects. If the design is balanced, then all these means have the same SE but the comparisons of B holding A fixed will have lower SE than comparisons of A holding B fixed. $\endgroup$ – rvl Sep 8 '14 at 20:47
  • $\begingroup$ Thank you a lot. In case you can recommend literature on the theoretical basis of differing means for mixed models, I'd be very happy to read up on it. $\endgroup$ – user54643 Sep 9 '14 at 8:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.