0
$\begingroup$

I ran an ANOVA on 3 sample sets with means 7.8, 8, and 7.9. The sample with a mean of 8 was statistically different than the other two. The sample size for each mean=200 and 95 confidence level. Why are these results statistically significant when means are so close?

$\endgroup$
  • 2
    $\begingroup$ Because the variability of each sample is small. A standard deviation of 0.28 for each sample with a sample size of 200 translates into a standard error of the sample mean of about 0.02, more than enough for 8 to be significantly different from 7.9. $\endgroup$ – jbowman Jan 13 at 15:09
  • $\begingroup$ Thank you for replying so quickly and helping me understand the reason for the statistical significance :) $\endgroup$ – user270797 Jan 13 at 15:53
2
$\begingroup$

STATISTICAL SIGNIFICANCE $\ne$ PRACTICAL SIGNIFIANCE

You have a pretty large sample size, meaning that you have the power to reject a null hypothesis of equal means, since standard error is a function of both effect size and sample size. You don't dispute that $7.9 \ne 8$, do you? Consequently, if you have that one distribution has a mean of $7.9$ and another has a mean of $8$, you don't dispute that they have different means. End of discussion;$^{\dagger}$ the groups have different means.

Whether or not that difference is enough to be an interesting finding is a separate question and not one that null hypothesis significance testing claims to do or ever has claimed to do. Whether or not the difference is interesting gets at the practical significance of the finding.

Now you use your subject matter expertise to say if such a difference is interesting enough to publish or change your business strategy or whatever. You are allowed to look at a statistically significant finding and decide that it isn't enough to interest you. This is not bad science.

However, you do have evidence that the groups come from populations with different means.

$^{\dagger}$ Well...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.