This post originates from this discussion: Two one-sided hypothesis tests instead of a two-sided test? Say we repeatedly draw a sample of size $n$ and conduct a two tailed t-test against a hypothesized mean $\mu_0$ at a certain significance level $\alpha$. Let us further assume the samples are from one (normal) distribution (it may or may not be the one we hypothesize) that does not change over time and is unknown to us. After a few runs, we may have some hits in both rejection regions of our hypothesized distribution and may reject the null. My question is: If results start to pile up in one rejection region after a lot of runs, how long will we believe in the plausibility of $H_0$ and thus the relevance of $\alpha$?. After all, an actual error only occurs with a chance of $$P(reject\:H_0\cap H_0\:true)=P(H_0\:true)\cdot P_{H_0\:true}(reject\:H_0)=P(H_0\:true)\cdot \alpha$$ Doesn't it just become unreasonable to believe in the possibility of $H_0\:true$ and the realistic chance of a type I error if results repeatedly suggest otherwise? Moreover, would it be reasonable to reject (or accept) $H_0$ after one test (as we often see being done in practice)? I mean, since frequentist probabilities refer to an "infinite" number of runs, isn't one result rather meaningless? On the other hand, if we conduct many runs, hypothesis testing also seems to become quite pointless as I pointed out before...
-
$\begingroup$ If you are willing to model the truth of hypotheses about parameters (H_0 true) and the data (test stat exceeds critical value (=reject H_0)) within a joint probability distribution, as you have done, then you should use Bayes. $\endgroup$– CloseToCCommented Mar 5, 2020 at 17:17
-
$\begingroup$ Indeed I have read before that a Bayesian approach may be more suitable for significance testing. Unfortunately I was never taught anything about the Bayesian way at university... The more I think about, the more I wonder why we are making such a fuss about t-tests and p values. After one run, we know nothing about $P(H_0\:true)$ and the result may just be mere coincidence. After many runs, we will know if $H_0$ is plausible and there should be no more need to think "in subjunctives". $\endgroup$– chicken_gameCommented Mar 6, 2020 at 6:48
2 Answers
Based upon your question, it appears you wish to know when we can forego traditional hypothesis testing given a sufficient number of rejections of $H_{0}$.
To begin, I want to note that frequentist tests cannot tell us anything with certainty. In the post you referenced, you posed a question about a drug's effectiveness on reducing the incidence of a particular disease. It is worth noting that frequentist routines, even if conducted repeatedly, cannot definitively answer this type of question. We must make a declaration of uncertainty.
Inference proceeds with statements regarding how likely, or unlikely, our observed effect is, if $H_{0}$ is true. Suppose a new drug purports to reduce the the onset of chronic heart failure. One group of 30 patients receives a new drug, while another group of 30 patients receives a placebo. After one year, the proportion of patients experiencing heart failure is lower in the treatment group. Suppose 10 out of 30 patients receiving the new drug experience heart failure after the observation period, compared with 20 out of 30 patients in the control group. It appears the drug reduced the onset of heart failure in the treatment group. Now suppose a new sample was drawn and 28 out of 30 patients did not experience any symptoms related to heart failure, compared with only 10 out of the 30 patients taking the placebo. We could concoct many stories from this observed effect. Maybe patients in the former test were more likely to transcend physiological hardship during the observation period. Based upon one test, this is a likely explanation. However, treated patients in the latter test showed even more improvement. It is still possible that the treatment group showed more willingness to improve their health throughout the observation period. But now, this explanation is less likely. I am only using this hypothetical example to illustrate a point.
My question is: If results start to pile up in one rejection region after a lot of runs, how long will we believe in the plausibility of 𝐻0 and thus the relevance of 𝛼?
Because we often work with samples in practice, we can never confirm nor deny the plausibility of $H_{0}$, which is a statement about a population quantity. Note, the p-value is not evidence of the truth of $H_{0}$. Suppose in our second example we obtained a $p$-value of .03. This is the probability of observing a result this extreme, or a result even more extreme, if the null were true. In other words, if the null were true, an observed effect this extreme, or even more extreme, would occur 3 times out of 100. Even in repeated testing, we are not assigning probabilities to the truth or falsity of the stated null.
Moreover, would it be reasonable to reject (or accept) 𝐻0 after one test (as we often see being done in practice)?
Yes. Hypothesis testing involves explicit statements about population parameters. The number of tests is irrelevant. The conclusions we draw from a single test are up to us to decide. Frequentist methods cannot answer questions with respect to how the data is favoring the null. Large p-values are not indicative of the truth of $H_{0}$.
Bayesian approaches may be more applicable in this scenario. See this post for a brief discussion.
-
$\begingroup$ Thank you for the explanation. So what I conclude after all is that by applying traditional frequentist p value statistics, we always act like we knew absolutely nothing about $p(H_0\:true)$ (even after many runs) and that if we would like to include an estimate of $p(H_0\:true)$ based on the results (evidence), we would have to go for a Bayesian approach. $\endgroup$ Commented Mar 10, 2020 at 7:36
-
1$\begingroup$ Bayesian $t$-tests can be implemented fairly easily in R. See the work by Meredith and Kruschke (2018). Also, this previous post may tangentially address your concerns. $\endgroup$ Commented Mar 10, 2020 at 13:56
Wow, really good question. Let me see if I can add something.
My question is: If results start to pile up in one rejection region after a lot of runs, how long will we believe in the plausibility of 𝐻0 and thus the relevance of 𝛼?
I don't think this is a Frequentist question. To review, probability is the long term relative frequency of an event. To quantify plausibility in a hypothesis sounds, at least to me, very Bayesian.
The mechanics of a hypothesis test force you to make an assumption about the world. In reality, H0 is strictly false (no two populations have precisely the same mean) but it can be a useful approximation. It is up to the investigator to determine if that null hypothesis is a useful approximation conditioned on the experiment, the question, past experiments, etc.
So to answer your question, there is no number we can place on the hypothesis as Frequentists. The plausibility of the null would be a scientific question, not a statistical one.
Doesn't it just become unreasonable to believe in the possibility of 𝐻0𝑡𝑟𝑢𝑒 and the realistic chance of a type I error if results repeatedly suggest otherwise?
I suppose this is the intended purpose of replication. A single rejection of the null does not constitute proof that the null is false (else, the type 1 error would be 0). Repeated rejection of the null through replication would likely lead to people believing the difference is real. One can see this happening even today as theories like General Relativity continuously receive empirical support for their theories. I suppose that is a more a concern for philosophers of science, and I'm sure I am making some philosophers role in their grave, but I find this argument compelling.
