Assume we compare two sample means from an experiment (Control and Treatment) using an appropriate statistic test.

According to the test, differences are not significant. H0 cannot be rejected.

Based on this test result, can we infer that the two population means are not different, i.e. that the treatment does not have an effect "in real life"?

Thanks in advance.

  • 1
    $\begingroup$ Counterquestion: Assume myself and Dirk Nowitzki (who has enjoyed the "treatment" of decades of basketball training, as well as talent) each throw three free throws, and we both happen to succeed twice. So there is zero difference between the sample means, which hence cannot be significant. Would you be willing to infer that I am no worse than Dirkules, i.e. that the treatment had no effect? $\endgroup$ – Christoph Hanck Sep 15 '16 at 13:37
  • $\begingroup$ So, what you mean is that even if we can be reasonably sure (by conducting a a priori power analysis) that our sample size is indeed big enough, but findings still suggest the difference is insignificant, we cannot infer that population means are different? We would need a significant difference that is equal to 0 to be able to infer this? $\endgroup$ – Hendrik Sep 15 '16 at 13:45
  • $\begingroup$ My example was evidently geared to alerting you to the role of sample size - with small samples, power of the test is low. In general, as you point out, tests make mistakes, include type-II errors. Moreovery, there are many threads on this site which discuss how to properly interpret outcomes of a statistical test (e.g., read up on popular threads related to p-values). $\endgroup$ – Christoph Hanck Sep 15 '16 at 13:51
  • 1
    $\begingroup$ Well, your question is related to type II errors: because tests may fail to reject false null hypotheses, we cannot infer from a nonrejection that the null is true. $\endgroup$ – Christoph Hanck Sep 15 '16 at 14:24
  • 1
    $\begingroup$ "Absence of evidence is not evidence of absence!" -- Carl Sagan, Astronomer $\endgroup$ – Michael M Sep 15 '16 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.