12
$\begingroup$

I'm trying to conduct a Student's t-test for a table of values while trying to follow the explanation and details found on this website. I understand that if the p-value is

  • <.01 then it's really significant
  • >.05 it's not significant
  • in between then we need more data

But on that page they seem to accept their null hypothesis no matter what the p-value is. So I'm really not understanding now when to accept or reject the null hypothesis.

  • When you do you accept or reject the null-hypothesis?
  • Is it true that you are never supposed to accept your null hypothesis, but rather reject or fail to reject the null?
Improve this question
$\endgroup$
9
  • 7
    $\begingroup$ You are right: If the $p$-value is >.05, we often say that we fail to reject the null hypothesis or that we don't have evidence to suggest that the means are different. This does not mean that the null hypothesis is true. But they explain it on the website: "In science, when we accept a hypothesis, this does NOT mean we have decided that the hypothesis is correct or that it is probably correct." I doubt if "accept" is the best term in this case as it can lead to confusion. $\endgroup$
    – COOLSerdash
    Commented Jun 2, 2013 at 16:57
  • 2
    $\begingroup$ This article here also advocates that the term "accept" should not be used by scientists. $\endgroup$
    – COOLSerdash
    Commented Jun 2, 2013 at 17:19
  • 3
    $\begingroup$ The explanation on that website is very poorly worded. $\endgroup$
    – Peter Flom
    Commented Jun 2, 2013 at 21:08
  • 2
    $\begingroup$ "fail to reject the null hypothesis" (or something similar) is the way I generally put it on the rare occasions when I formally test a hypothesis and don't reject the null. I almost never think the null has a chance to be actually true so it's more a lack of evidence against the null than in any sense an acceptance that the null is the case. $\endgroup$
    – Glen_b
    Commented Jun 3, 2013 at 1:53
  • 6
    $\begingroup$ All this begs the questions of why we need formal hypotheses vs. estimating a quantity of interest and reporting confidence intervals. There is no bifurcation implied by estimation. $\endgroup$
    – Frank Harrell
    Commented Jul 15, 2013 at 12:09

1 Answer 1

Reset to default
18
$\begingroup$

I would suggest that it is much better to say that we "fail to reject the null hypothesis", as there are at least two reasons we might not achieve a significant result: Firstly it may be because H0 is actually true, but it might also be the case that H0 is false, but we have not collected enough data to provide sufficient evidence against it. Consider the case where we are trying to determine whether a coin is biased (H0 being that the coin is fair). If we only observe 4 coin flips, the p-value can never be less than 0.05, even if the coin is so biased it has a head on both sides, so we will always "fail to reject the null hypothesis". Clearly in that case we wouldn't want to accept the null hypothesis as it isn't true. Ideally we should perform a power analysis to find out if we can reasonably expect to be able to reject the null hypothesis when it is false, however this isn't usually nearly as straightforward as performing the test itself, which is why it is usually neglected.

Update: The null hypothesis is quite often known to be false before observing the data. For instance a coin (being asymmetric) is almost certainly biased; the magnitude of this bias us undoubtedly negligible, but not precisely zero, which is what the H0 for the usual test of the bias of a coin asserts. If we observe a sufficiently large number of flips, we will eventually be able to detect this miniscule deviation from exact unbiasedness. It would be odd then to accept the "null hypothesis" in this case as we know before performing the test that it is certainly false. The test is certainly still useful though as we are generally interested in whether the coin is practically biased.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ And what if the Power (from simulations) for that test is really high. Could I say "accept the Null"? $\endgroup$
    – An old man in the sea.
    Commented Feb 19, 2015 at 13:30
  • 1
    $\begingroup$ That would be better, but there are still a lot that is left implicit, for instance H1 may not be the only alternative to H0 and the relative prior probabilities of H0 and H1 (which only really enter into the test indirectly via $\alpha$ and $\beta$) may be strongly against H0. The real problem is that we really want to know the probability that H0 is true, which a frequentist test can't give us, so it is a good idea to avoid terminology that could be interpreted that way, I like "fail to reject the null hypothesis" as it implies that its meaning is subtle (which it is!). $\endgroup$
    – Dikran Marsupial
    Commented Feb 19, 2015 at 14:13
  • 4
    $\begingroup$ There is also the point that sometimes you know a-priori that the null hypothesis is certainly false (e.g. that a coin is exactly unbiased), and it would be odd to accept something that you know isn't true even before performing the test. $\endgroup$
    – Dikran Marsupial
    Commented Feb 19, 2015 at 14:16
  • $\begingroup$ @dikran-marsupial how can you make an a priori statement about the coin without using some kind of prior though? It seems to me that you want to say that the P(p=0.5) = 0 and P(p!=0.5) = 1 which is true is we treat p as a having a continuous pdf since 0.5 has measure zero. But I don't think we can do this in the frequentist setting since we can't really assign probabilities to H0 and H1 a priori. $\endgroup$
    – Alexandru Papiu
    Commented Jul 27, 2022 at 18:31
  • $\begingroup$ @AlexandruPapiu frequentists do make use of prior knowledge, but they try to hide it. For instance the choice of significance level in NHSTs is an application of prior knowledge (this is the error made by the frequentist in the well known XKCD cartoon stats.stackexchange.com/questions/43339/… ). It is physics that tells us we can be sure the coin isn't exactly fair, rather than a matter of probability/statistics, if we have enough data, we will always identify a statistically significant bias. $\endgroup$
    – Dikran Marsupial
    Commented Jul 27, 2022 at 19:29

Not the answer you're looking for? Browse other questions tagged or ask your own question.