62
$\begingroup$

Traditional statistical tests, like the two sample t-test, focus on trying to eliminate the hypothesis that there is no difference between a function of two independent samples. Then, we choose a confidence level and say that if the difference of means is beyond the 95% level, we can reject the null hypothesis. If not, we "can't reject the null hypothesis". This seems to imply that we can't accept it either. Does it mean we're not sure if the null hypothesis is true?

Now, I want to design a test where my hypothesis is that a function of two samples is the same (which is the opposite of traditional statistics tests where the hypothesis is that the two samples are different). So, my null hypothesis becomes that the two samples are different. How should I design such a test? Will it be as simple as saying that if the p-value is lesser than 5% we can accept the hypothesis that there is no significant difference?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Very related: Does failure to reject the null in Neyman-Pearson approach mean that one should “accept” it? $\endgroup$
    – amoeba
    Commented Dec 9, 2016 at 23:25
  • $\begingroup$ difference of means is beyond the 95% level, we can reject the null hypothesis. The 95% is not a "level" it is here in 95 cases out of 100 cases (comparisons), the differennce in sample-statistic arises due to sampling fluctuations. it means that null is accepted at alpha = .05. Saying 95% level is not correct term. $\endgroup$
    – user10619
    Commented Apr 23, 2019 at 6:07

4 Answers 4

Reset to default
57
$\begingroup$

Traditionally, the null hypothesis is a point value. (It is typically $0$, but can in fact be any point value.) The alternative hypothesis is that the true value is any value other than the null value. Because a continuous variable (such as a mean difference) can take on a value which is indefinitely close to the null value but still not quite equal and thus make the null hypothesis false, a traditional point null hypothesis cannot be proven.

Imagine your null hypothesis is $0$, and the mean difference you observe is $0.01$. Is it reasonable to assume the null hypothesis is true? You don't know yet; it would be helpful to know what our confidence interval looks like. Let's say that your 95% confidence interval is $(-4.99,\ 5.01)$. Now, should we conclude that the true value is $0$? I would not feel comfortable saying that, because the CI is very wide, and there are many, large non-zero values that we might reasonably suspect are consistent with our data. So let's say we gather much, much more data, and now our observed mean difference is $0.01$, but the 95% CI is $(0.005,\ 0.015)$. The observed mean difference has stayed the same (which would be amazing if it really happened), but the confidence interval now excludes the null value. Of course, this is just a thought experiment, but it should make the basic ideas clear. We can never prove that the true value is any particular point value; we can only (possibly) disprove that it is some point value. In statistical hypothesis testing, the fact that the p-value is > 0.05 (and that the 95% CI includes zero) means that we are not sure if the null hypothesis is true.

As for your concrete case, you cannot construct a test where the alternative hypothesis is that the mean difference is $0$ and the null hypothesis is anything other than zero. This violates the logic of hypothesis testing. It is perfectly reasonable that it is your substantive, scientific hypothesis, but it cannot be your alternative hypothesis in a hypothesis testing situation.

So what can you do? In this situation, you use equivalence testing. (You might want to read through some of our threads on this topic by clicking on the tag.) The typical strategy is to use the two one sided tests approach. Very briefly, you select an interval within which you would consider that the true mean difference might as well be $0$ for all you could care, then you perform a one-sided test to determine if the observed value is less than the upper bound of that interval, and another one-sided test to see if it is greater than the lower bound. If both of these tests are significant, then you have rejected the hypothesis that the true value is outside the interval you care about. If one (or both) are non-significant, you fail to reject the hypothesis that the true value is outside the interval.

For example, suppose anything within the interval $(-0.02,\ 0.02)$ is so close to zero that you think it is essentially the same as zero for your purposes, so you use that as your substantive hypothesis. Now imagine that you get the first result described above. Although $0.01$ falls within that interval, you would not be able to reject the null hypothesis on either one-sided t-test, so you would fail to reject the null hypothesis. On the other hand, imagine that you got the second result described above. Now you find that the observed value falls within the designated interval, and it can be shown to be both less than the upper bound and greater than the lower bound, so you can reject the null. (It is worth noting that you can reject both the hypothesis that the true value is $0$, and the hypothesis that the true value lies outside of the interval $(-0.02,\ 0.02)$, which may seem perplexing at first, but is fully consistent with the logic of hypothesis testing.)

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ "Traditionally, the null hypothesis is a point value" - though in some cases we write the null hypothesis as if it were point, yet actually it's compound. I'm curious what implication the argument in your first paragraph therefore has for one-sided tests. (Since we don't - as far as I know - write "accept $H_0$" even for one-sided tests, I'm not sure the first paragraph captures the true reason we don't write "accept $H_0$.) $\endgroup$
    – Silverfish
    Commented Jan 5, 2015 at 14:19
  • 1
    $\begingroup$ @Silverfish, the paragraph ends with: "a traditional point null hypothesis cannot be proven". However, we also don't write "accept $H_0$" for one-sided tests for the same reason. When $H_0: \delta\le 0$, the true $\delta$ can be $>0$, but arbitrarily close & thus non-significant. If you really wanted to show that it was $<0$, then you can flip the direction of the one-sided test. I don't see a problem here. $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 5, 2015 at 14:26
  • 1
    $\begingroup$ I'm not saying what you wrote is wrong and I suspected that was the idea you were trying to communicate. Obviously the reason you have tackled the two-sided test with a point hypothesis in the first two paragraphs of your answer, is that this the case in the question. But if your answer is re-read by someone wondering about why we don't "accept $H_0$" in general, it may not be clear to them that your argument actually extends beyond point null hypotheses. $\endgroup$
    – Silverfish
    Commented Jan 5, 2015 at 14:45
  • 5
    $\begingroup$ The argument "we can never prove that the true value is any particular point value; we can only (possibly) disprove that it is some point value" is a particular case in point - what if the CI had turned out to be (-0.015, -0.005)? To whatever extent we have "proved" $\delta \neq 0$ (I know you don't use "prove" in the literal, mathematical sense - perhaps "demonstrate" or "suggest" are closer to the intended meaning) it seems we have also "proved" $\delta \leq 0$, yet still we would not "accept" $H_0:\,\delta \leq 0$ $\endgroup$
    – Silverfish
    Commented Jan 5, 2015 at 14:48
  • 1
    $\begingroup$ @Silverfish I think your last comment makes a good point. I feel that, philosophically, one-sided testing with $H_0:\delta<0$ is quite a bit different from two-sided with point null $H_0:\delta=0$, even though mathematically they are almost the same. Accepting point null does not make sense; but testing $\delta>0$ against $\delta<0$ can actually lead to accepting one of them (or an inconclusive result). Plus one-sided testing makes more sense from Bayesian perspective. Plus scientific prediction should have a direction. I guess I start thinking that one-sided testing is not appreciated enough. $\endgroup$
    – amoeba
    Commented Dec 5, 2016 at 22:20
33
$\begingroup$

Consider the case where the null hypothesis is that a coin is 2 headed, i.e. the probability of heads is 1. Now the data is the result of flipping a coin a single time and seeing heads. This results in a p-value of 1.0 which is greater than every reasonable alpha. Does this mean that the coin is 2 headed? it could be, but it could also be a fair coin and we saw heads due to chance (would happen 50% of the time with a fair coin). So the high p-value in this case says that the observed data is perfectly consistent with the null, but it is also consistent with other possibilities.

Just like a "Not Guilty" verdict in court can mean the defendant is innocent, it can also be because the defendant is guilty but there is not enough evidence. The same with the null hypothesis we fail to reject because the null could be true, or it could be we don't have enough evidence to reject even though it is false.

Improve this answer
$\endgroup$
1
  • 5
    $\begingroup$ I like the "Not guilty" example. Going one step further, re-opening cases based on DNA evidence that we did not know how to use in the past and having some convictions overturned is a perfect example of how adding more data may be all that's needed to have enough evidence. $\endgroup$
    – Thomas Speidel
    Commented Feb 10, 2014 at 15:21
7
$\begingroup$

Absence of evidence is not evidence of an absence (the title of an Altman, Bland paper on BMJ). P-values only give us evidence of an absence when we consider them significant. Otherwise, they tell us nothing. Hence, absence of evidence. In other words: we don't know and more data may help.

Improve this answer
$\endgroup$
2
  • 11
    $\begingroup$ sigh...yet again I see this quote being stated and yet again i must point out it is a false statement. absence of evidence is not proof of absence. It is evidence of absence though. Think - if I ingest a new substance it might be poisoness. After doing this once and finding no side effects - I have got evidence of absence of poison, from the absence of an effect in the data I observed. But it's not proof (maybe I was lucky), for this would require more data, as you say. $\endgroup$
    – probabilityislogic
    Commented Jul 16, 2020 at 12:12
  • $\begingroup$ A properly powered hypothesis test that fails to reject the null is absolutely evidence of absence. Sufficient power means that you likely would have rejected the null if it were indeed false, the fact that you didn't implies that it is not false. This answer suggests that if you check to see if there's an elephant in your closet and don't see one, it is not evidence that your closet is elephant-free. $\endgroup$
    – Nuclear Hoagie
    Commented Nov 7, 2022 at 13:44
5
$\begingroup$

The null hypothesis, $H_0$, is usually taken to be the thing you have reason to assume. Often times it is the "current state of knowledge" that you wish to show is statistically unlikely.

The usual set-up for hypothesis testing is minimize type I error, that is, minimize the chance that we reject the null hypothesis in favor of the alternative $H_1$ even though $H_0$ is true. This is the error we choose to first minimize because we don't want to overturn common knowledge when that common knowledge is indeed true.

You should always design your test bearing in mind that $H_0$ should be what you expect.

If we have two samples we expect to be identically distributed then our null hypothesis is the samples are the same. If we have two samples that we would expect to be (wildly) different, our null hypothesis is that they are different.

Improve this answer
$\endgroup$
8
  • $\begingroup$ And what if we have no expectations.. it might be that we just don't know. Also, how will the decision rule work if we want to reject the hypothesis that the two samples are different? $\endgroup$
    – ryu576
    Commented Feb 8, 2014 at 21:12
  • $\begingroup$ In the case you have no expectations you want to keep both both types of errors small but this isn't always possible. You need an extra variable (such as increasing sample size) to do it. $\endgroup$
    – SomeEE
    Commented Feb 8, 2014 at 21:31
  • 3
    $\begingroup$ Since we can reject the null but not prove it true the null is usually the opposite of what we want to prove or assume to be true. If we believe that there is a difference then the null should be no difference so that you can disprove that. $\endgroup$
    – Greg Snow
    Commented Feb 8, 2014 at 22:03
  • $\begingroup$ @Greg That is a good approach if you know which one you want to be true which is probably the usual case. $\endgroup$
    – SomeEE
    Commented Feb 8, 2014 at 22:08
  • 2
    $\begingroup$ "What you expect" and "that they are different" cannot be statistical hypotheses at all because they are not quantitative. That gets to the crux of the matter: the asymmetry in roles between the null and alternative hypotheses derives from the ability to determine the sampling distribution of the test statistic under the null, compared to the need to parameterize the distribution by the effect size under the alternative hypothesis. Nor is it the case the we "minimize Type I error": that never happens (the minimum is always 0). Tests seek a balance between Type I and II error rates. $\endgroup$
    – whuber ♦
    Commented Feb 10, 2014 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.