6
$\begingroup$

In testing a null vs an alternative, if we don't reject the null (e.g. the p-value is very big, bigger than the significance level), what is our conclusion?

Can we say that we accept the null, can we? I remember I heard that not rejecting the null doesn't mean accepting the null, but I might be wrong.

Or should we simply say that we don't reject the null?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
13
$\begingroup$

"We fail to reject the null" is the correct answer. Rather than say, for example, "there is no difference" we should write "no difference was detected".

Clearly, with not enough replicates or large measurement error you are likely to not be able to detect even large effects. So maybe the difference is there, but you failed to see it?

As Maarten Buis writes in the comment: "Absence of evidence is not evidence of absence". (Personally, I am careful with the word "evidence".)

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I like the phrase "Absence of evidence is not evidence of absence" in this respect. Failing to reject would be a case of absence of evidence. Many null hypotheses are of the form "there is no effect", so accepting such a hypothesis would be a case of finding evidence of absence (of a effect). $\endgroup$
    – Maarten Buis
    Mar 12, 2014 at 14:12
  • $\begingroup$ I like this formulation as well, but I did not want to start a discussion on whether a large p-value is not a kind of evidence in favor of the null (as in, posterior probability of the null increased; as in, Bayes factor). $\endgroup$
    – January
    Mar 12, 2014 at 14:51
3
$\begingroup$

I would include three things:

  1. The phrase "Insufficient evidence to reject". Shows that with more evidence, e.g. more data, or repeating the experiment with a different random selection of data, you might have rejected.

  2. The significance level. At a higher significance level you might have rejected the null hypothesis.

  3. What your conclusion is. "The data is consistent with" is a good phrase here.

As an example, putting these altogether for an experiment to determine the effect of cognitive behavioural therapy (CBT) on addressing insomnia

"There was insufficient evidence to reject the null hypothesis that CBT increased the amount of sleep at the 5% significance level. The data is consistent with CBT having no effect on insomnia."

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I would also include the study power when failing to reject the null. "There is insufficient evidence to reject the null hypothesis at the 5% significance level. The study's power to detect a 1% difference in means was approximately 90%." is an entirely different statement from "There is insufficient evidence to reject the null hypothesis at the 5% significance level. The study's power to detect a 1% difference in means was approximately 15%." $\endgroup$
    – P Schnell
    Mar 12, 2014 at 16:05
  • 1
    $\begingroup$ @PSchnell Yes -- the significance level is the chance of rejecting the null hypothesis when you shouldn't (when $H_0$ true), and it could also be worth reporting the power, the chance of rejecting the null hypothesis when you should (when $H_0$ false). However I've done significance tests when I haven't calculated the power, so I wouldn't have been able to report it... Sometimes you just do as many replicates as time/money allows and you only get to choose the significance level. $\endgroup$
    – TooTone
    Mar 12, 2014 at 16:36
  • $\begingroup$ Pedant mode: "data are consistent". Carry on. :) $\endgroup$
    – Almo
    Mar 12, 2014 at 17:40
  • 1
    $\begingroup$ @Almo I love this site:) $\endgroup$
    – TooTone
    Mar 12, 2014 at 17:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.