Significance and certainty in hypothesis testing This post originates from this discussion: Two one-sided hypothesis tests instead of a two-sided test? Say we repeatedly draw a sample of size $n$ and conduct a two tailed t-test against a hypothesized mean $\mu_0$ at a certain significance level $\alpha$. Let us further assume the samples are from one (normal) distribution (it may or may not be the one we hypothesize) that does not change over time and is unknown to us. After a few runs, we may have some hits in both rejection regions of our hypothesized distribution and may reject the null. My question is: If results start to pile up in one rejection region after a lot of runs, how long will we believe in the plausibility of $H_0$ and thus the relevance of $\alpha$?. After all, an actual error only occurs with a chance of $$P(reject\:H_0\cap H_0\:true)=P(H_0\:true)\cdot P_{H_0\:true}(reject\:H_0)=P(H_0\:true)\cdot \alpha$$ Doesn't it just become unreasonable to believe in the possibility of $H_0\:true$ and the realistic chance of a type I error if results repeatedly suggest otherwise?
Moreover, would it be reasonable to reject (or accept) $H_0$ after one test (as we often see being done in practice)? I mean, since frequentist probabilities refer to an "infinite" number of runs, isn't one result rather meaningless? On the other hand, if we conduct many runs, hypothesis testing also seems to become quite pointless as I pointed out before...
 A: Based upon your question, it appears you wish to know when we can forego traditional hypothesis testing given a sufficient number of rejections of $H_{0}$. 
To begin, I want to note that frequentist tests cannot tell us anything with certainty. In the post you referenced, you posed a question about a drug's effectiveness on reducing the incidence of a particular disease. It is worth noting that frequentist routines, even if conducted repeatedly, cannot definitively answer this type of question. We must make a declaration of uncertainty.
Inference proceeds with statements regarding how likely, or unlikely, our observed effect is, if $H_{0}$ is true. Suppose a new drug purports to reduce the the onset of chronic heart failure. One group of 30 patients receives a new drug, while another group of 30 patients receives a placebo. After one year, the proportion of patients experiencing heart failure is lower in the treatment group. Suppose 10 out of 30 patients receiving the new drug experience heart failure after the observation period, compared with 20 out of 30 patients in the control group. It appears the drug reduced the onset of heart failure in the treatment group. Now suppose a new sample was drawn and 28 out of 30 patients did not experience any symptoms related to heart failure, compared with only 10 out of the 30 patients taking the placebo. We could concoct many stories from this observed effect. Maybe patients in the former test were more likely to transcend physiological hardship during the observation period. Based upon one test, this is a likely explanation. However, treated patients in the latter test showed even more improvement. It is still possible that the treatment group showed more willingness to improve their health throughout the observation period. But now, this explanation is less likely. I am only using this hypothetical example to illustrate a point.

My question is: If results start to pile up in one rejection region after a lot of runs, how long will we believe in the plausibility of 0 and thus the relevance of ?

Because we often work with samples in practice, we can never confirm nor deny the plausibility of $H_{0}$, which is a statement about a population quantity. Note, the p-value is not evidence of the truth of $H_{0}$. Suppose in our second example we obtained a $p$-value of .03. This is the probability of observing a result this extreme, or a result even more extreme, if the null were true. In other words, if the null were true, an observed effect this extreme, or even more extreme, would occur 3 times out of 100. Even in repeated testing, we are not assigning probabilities to the truth or falsity of the stated null.

Moreover, would it be reasonable to reject (or accept) 0 after one test (as we often see being done in practice)?

Yes. Hypothesis testing involves explicit statements about population parameters. The number of tests is irrelevant. The conclusions we draw from a single test are up to us to decide. Frequentist methods cannot answer questions with respect to how the data is favoring the null. Large p-values are not indicative of the truth of $H_{0}$.
Bayesian approaches may be more applicable in this scenario. See this post for a brief discussion.
A: Wow, really good question.  Let me see if I can add something.

My question is: If results start to pile up in one rejection region after a lot of runs, how long will we believe in the plausibility of 0 and thus the relevance of ?

I don't think this is a Frequentist question.  To review, probability is the long term relative frequency of an event.  To quantify plausibility in a hypothesis sounds, at least to me, very Bayesian.
The mechanics of a hypothesis test force you to make an assumption about the world.  In reality, H0 is strictly false (no two populations have precisely the same mean) but it can be a useful approximation.  It is up to the investigator to determine if that null hypothesis is a useful approximation conditioned on the experiment, the question, past experiments, etc.
So to answer your question, there is no number we can place on the hypothesis as Frequentists.  The plausibility of the null would be a scientific question, not a statistical one.

Doesn't it just become unreasonable to believe in the possibility of 0 and the realistic chance of a type I error if results repeatedly suggest otherwise?

I suppose this is the intended purpose of replication.  A single rejection of the null does not constitute proof that the null is false (else, the type 1 error would be 0).  Repeated rejection of the null through replication would likely lead to people believing the difference is real.  One can see this happening even today as theories like General Relativity continuously receive empirical support for their theories. I suppose that is a more a concern for philosophers of science, and I'm sure I am making some philosophers role in their grave, but I find this argument compelling.
