Timeline for If we fail to reject the null hypothesis in a large study, isn't it evidence for the null?
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 2, 2017 at 8:22 | comment | added | Thern | @amoeba I disagree. For a general assumption, one counterexample is enough. I was referring to the thinking behind the assumption and the logic involved, not practical purposes of statistical evaluations. You ask the question "are typical statistical evaluations designed in a way to allow for assisting the null hyp", I ask the question "is missing evidence against the null hyp always evidence for the null hyp". That are two different questions, and since the intention of the question of the OP is a bit ambiguous, it can be read (and answered) in different ways. | |
| May 2, 2017 at 7:38 | comment | added | amoeba | @Nebr I agree that it's a general question. But if you cannot give me a single example of a real statistical test that would have the same property as your monkey example, then I am sorry but I will have to consider your monkey example pretty much irrelevant for this thread. I am not saying that the monkey example has to correspond to a t-test specifically. But it has to correspond to something!! | |
| May 2, 2017 at 7:23 | comment | added | Thern | @amoeba I interpreted the question as a general question without any correspondence to specific test settings. But I agree with you that there are settings where you can get evidence for the null hyp. However, these are specific cases with additional inbaked information. But the assumption "missing evidence against null hyp" = "evidence for null hyp" is not true regardless of the test setting. The monkey example shows a test setting where the assumption is not true. | |
| Apr 30, 2017 at 12:21 | comment | added | amoeba | Wait a second. The OP asked: "If we perform a large study and we don't find statistically significant evidence against the null hypothesis, isn't that evidence for the null hypothesis?" I presume OP was not testing if monkeys were in the cage. Maybe OP had a t-test in mind? Maybe an ANOVA? Maybe a test that correlation is zero? So what is your answer to their question for the tests that are actually used in practice? Is it evidence that the null is true or is it not? | |
| Apr 30, 2017 at 9:12 | comment | added | Thern | @amoeba T-testing is a common testing method, but why do you expect that it is possible to map any hypothesis testing to a t-test? Can you map the concept "Given the null hypothesis is true, what is the probability that we observe our data (or more extreme data)?" to a t-test in general? The question if monkeys are in the cage can't easily be mapped to a µ = 0 situation because it is not clear what µ = 1 or µ = -1 should mean, but the hypothesis "monkeys are in the cage" is still a validly formulated hypothesis (verifiable, and either true or false). Why should it be mappable to a t-test? | |
| Apr 29, 2017 at 19:28 | comment | added | amoeba | If so @Nebr, then I am again very confused about the meaning of your monkey example. T-test is probably the most common hypothesis test; I mentioned it in my comment just because it's such a typical example of a test. If your monkey example is not applicable (as you say) to this -- typical! -- situation, then I am puzzled about its meaning. In fact, if you say that t-test and monkey example are "two different ways of hypothesis testing", then can you give an example of statistical test that follows your monkey example "way"? What exactly is your monkey example an analogy of? | |
| Apr 29, 2017 at 5:37 | comment | added | Thern | @amoeba I am not sure if it is possible to directly translate the monkey example to your t-test scenario. To my knowledge, null hypothesis testing generally means what also Mark White wrote in his answer: "Given the null hypothesis is true, what is the probability that we observe our data (or more extreme data)?". Your t-testing scenario is a specific case of this, but I currently don't see how this scenario can be generalized. From my gut feeling, I would say that your scenario and the monkey example are two different ways of hypothesis testing that can't be mapped to each other directly. | |
| Apr 28, 2017 at 20:23 | comment | added | amoeba | @Nebr I see. So you are saying that if observations are consistent with the null hypothesis, they might still be consistent with something else too. This makes sense. So how does it explicitly translate to the standard hypothesis testing situation? For example, when you do a one-sample t-test with $H_0: \mu=0$ and observe something close to zero with $p=0.2$ so that you cannot reject the null; you are saying that these data are consistent with $\mu=0.0001$ as well as with $\mu=0$ so it does not provide evidence for the null itself? Do I understand your logic correctly? | |
| Apr 28, 2017 at 19:53 | comment | added | Thern | @amoeba In this case, null hyp would be that monkeys are in the cage. Alt hyp would be that no monkeys are in the cage. The samples I gather are the observations "banana gone" and "banana still there" each morning. Making several assumptions about monkeys and their ability to find bananas, I can calculate the probability p that I would have seen the actual result with monkeys in a cage. If bananas are still there often, I will reject the null hyp. If bananas are always gone, this fits to the null hyp, but it does not prove that monkeys are in the cage. | |
| Apr 28, 2017 at 17:42 | comment | added | amoeba | I don't understand this answer (and I suspect that it can be misleading). Can you please explain how exactly your monkey example corresponds to hypothesis testing? What is the null hypothesis, what is the alternative hypothesis, etc. E.g. I am doing a t-test and my null hypothesis is that $\mu=0$. How does this correspond to your monkey example? | |
| Apr 25, 2017 at 15:32 | review | First posts | |||
| Apr 25, 2017 at 15:37 | |||||
| Apr 25, 2017 at 15:30 | history | answered | Thern | CC BY-SA 3.0 |