1
$\begingroup$

I have 5 groups with unequal variances. I used a Welch's ANOVA and then a Games-Howell post-hoc analysis to find the pairs that have significant differences. I have a large number of samples in each group (~1000) and therefore almost all differences are significant. Due to such large number of samples, I have observed that even the slightest differences tend to be significant. That is why I am not particularly interested in the pairs that are significant. However, 5 groups result is 10 pairs of groups. I thought of reporting the mean of absolute differences(MAD?). Is this a statistically sound way to summarise these differences?

Additionally, if there are multiple such experiments, could I comparatively present the MAD values from the different experiments?

Thankyou

$\endgroup$

1 Answer 1

1
$\begingroup$

With large sample sizes, trivial differences can lead to small p-values. So you are wise not to focus on that.

But your ANOVA program should be able to report the 95% confidence interval for each of the ten differences between group means. I'd report that. If the CI only contains differences that you'd consider trivial in the context of your work, then you have good evidence that the difference (if any) is trivial.

I don't seem how reporting MAD will be more helpful than reporting difference between means.??

$\endgroup$
2
  • $\begingroup$ Thanks! In my field we work with the assumption (so far untested) that the means of all the groups are not significantly different from each other. For example, consider two groups P and Q. Right now, I am trying to show the magnitude of the difference between groups P and Q. It doesn't matter whether the mean of P>the mean of Q. However, when there is another group, R, there are two differences, right? I thought, as I am only interested in the magnitude of the difference, I can report the mean of the absolute differences. $\endgroup$
    – K_D
    Apr 7, 2022 at 8:27
  • $\begingroup$ The answer to your last question is in my last statement about MAD for multiple experiments. In experiment 1, lets says, mean of P>mean of Q and mean of Q > mean of R. The same pattern need not be true to for experiment 2. It could be mean of P>mean of Q and mean of Q < mean of R. I am reporting the differences. I was only wondering if I could summarise the differences using MAD. $\endgroup$
    – K_D
    Apr 7, 2022 at 8:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.