A basic limitation of null hypothesis significance testing is that it does not allow a researcher to gather evidence in favor of the null (Source)
I see this claim repeated in multiple places, but I can't find justification for it. If we perform a large study and we don't find statistically significant evidence against the null hypothesis, isn't that evidence for the null hypothesis?
Failing to reject a null hypothesis is evidence that the null hypothesis is true, but it might not be particularly good evidence, and it certainly doesn't prove the null hypothesis.
Let's take a short detour. Consider for a moment the old cliché:
Absence of evidence is not evidence of absence.
Notwithstanding its popularity, this statement is nonsense. If you look for something and fail to find it, that is absolutely evidence that it isn't there. How good that evidence is depends on how thorough your search was. A cursory search provides weak evidence; an exhaustive search provides strong evidence.
Now, back to hypothesis testing. When you run a hypothesis test, you are looking for evidence that the null hypothesis is not true. If you don't find it, then that is certainly evidence that the null hypothesis is true, but how strong is that evidence? To know that, you have to know how likely it is that evidence that would have made you reject the null hypothesis could have eluded your search. That is, what is the probability of a false negative on your test? This is related to the power, $\beta$, of the test (specifically, it is the complement, 1-$\beta$.)
Now, the power of the test, and therefore the false negative rate, usually depends on the size of the effect you are looking for. Large effects are easier to detect than small ones. Therefore, there is no single $\beta$ for an experiment, and therefore no definitive answer to the question of how strong the evidence for the null hypothesis is. Put another way, there is always some effect size small enough that it's not ruled out by the experiment.
From here, there are two ways to proceed. Sometimes you know you don't care about an effect size smaller than some threshold. In that case, you probably should reframe your experiment such that the null hypothesis is that the effect is above that threshold, and then test the alternative hypothesis that the effect is below the threshold. Alternatively, you could use your results to set bounds on the believable size of the effect. Your conclusion would be that the size of the effect lies in some interval, with some probability. That approach is just a small step away from a Bayesian treatment, which you might want to learn more about, if you frequently find yourself in this sort of situation.
There's a nice answer to a related question that touches on evidence of absence testing, which you might find useful.
NHST relies on p-values, which tell us: Given the null hypothesis is true, what is the probability that we observe our data (or more extreme data)?
We assume that the null hypothesis is true—it is baked into NHST that the null hypothesis is 100% correct. Small p-values tell us that, if the null hypothesis is true, our data (or more extreme data) are not likely.
But what does a large p-value tell us? It tells us that, given the null hypothesis, our data (or more extreme data) are likely.
Generally speaking, P(A|B) ≠ P(B|A).
Imagine you want to take a large p-value as evidence for the null hypothesis. You would rely on this logic:
This takes on the more general form:
This is fallacious, though, as can be seen by an example:
The ground could very well be wet because it rained. Or it could be due to a sprinkler, someone cleaning their gutters, a water main broke, etc. More extreme examples can be found in the link above.
It is a very difficult concept to grasp. If we want evidence for the null, Bayesian inference is required. To me, the most accessible explanation of this logic is by Rouder et al. (2016). in paper Is There a Free Lunch in Inference? published in Topics in Cognitive Science, 8, pp. 520–547.
To grasp what is wrong with the assumption, see the following example:
Imagine an enclosure in a zoo where you can't see its inhabitants. You want to test the hypothesis that it is inhabited by monkeys by putting a banana into the cage and check if it is gone the next day. This is repeated N times for enhanced statistical significance.
Now you can formulate a null hypothesis: Given that there are monkeys in the enclosure, it is very probable that they will find and eat the banana, so if the bananas are untouched each day, it is very improbable that there are any monkeys inside.
But now you see that the bananas are gone (nearly) each day. Does that tell you that monkeys are inside?
Of course not, because there are other animals that like bananas as well, or maybe some attentive zookeeper removes the banana every evening.
So what is the mistake that is made in this logic? The point is that you do not know anything about the probability of bananas being gone if there are no monkeys inside. To corroborate the null hypothesis, the probability of vanishing bananas must be small if the null hypothesis is wrong, but this does not need to be the case. In fact, the event may be equally probable (or even more probable) if the null hypothesis is wrong.
Without knowing about this probability, you can say exactly nothing about the validity of the null hypothesis. If zookeepers remove all bananas each evening, the experiment is completely worthless, even though it seems on first glance that you have corroborated the null hypothesis.
In his famous paper Why Most Published Research Findings Are False, Ioannidis used Bayesian reasoning and the base rate-fallacy to argue that most findings are false-positives. Shortly, the post-study probability that a particular research hypothesis is true depends - among other things - on the pre-study probability of said hypothesis (i.e. the base rate).
As a response, Moonesinghe et al. (2007) used the same framework to show that replication greatly increases the post-study probability of a hypothesis being true. This makes sense: If multiple studies can replicate a certain finding, we are more sure that the conjectured hypothesis is true.
I used the formulas in Moonesinghe et al. (2007) to create a graph that shows the post-study probability in the case of a failure to replicate a finding. Assume that a certain research hypothesis has a pre-study probability of being true of 50%. Further, I'm assuming that all studies have no bias (unrealistic!) have a power of 80% and use an $\alpha$ of 0.05.
The graph shows that if at least 5 out of 10 studies fail to reach significance, our post-study probability that the hypothesis is true is almost 0. The same relationships exist for more studies. This finding also makes intuitive sense: A repeated failure to find an effect strengthens our belief that the effect is most likely false. This reasoning is in line with the accepted answer by @RPL.
As a second scenario, let's assume that the studies have only a power of 50% (all else equal).
Now our post-study probability decreases more slowly, because every study had only low power to find the effect, if it really existed.
If you have a negative, you found evidence against the null - What? The word "negative" has exactly the opposite meaning. A significant p-value is called a "positive" result; a non-significant is a "negative".
$\endgroup$
The best explanation I've seen for this is from someone whose training is in mathematics.
Null-Hypothesis Significance Testing is basically a proof by contradiction: assume $H_0$, is there evidence for $H_1$? If there is evidence for $H_1$, reject $H_0$ and accept $H_1$. But if there isn't evidence for $H_1$, it's circular to say that $H_0$ is true because you assumed that $H_0$ was true to begin with.
If you do not like this consequence of hypothesis testing but are not prepared to make the full leap to Bayesian methods, how about a confidence interval?
Suppose you flip a coin $42078$ times and see $20913$ heads, leading to you saying that a 95% confidence interval for the probability of heads is $[0.492,0.502]$.
You have not said you have seen evidence that it is in fact $\frac12$, but the evidence suggests some confidence about how close it might be to $\frac12$.
It would perhaps be better to say that non-rejection of a null hypothesis is not in itself evidence for the null hypothesis. Once we consider the full likelihood of the data, which more explicitly considers the amount of the data, then the collected data may provide support for the parameters falling within the null hypothesis.
However, we should also carefully think about our hypotheses. In particular, failing to reject a point null hypothesis is not very good evidence that the point null hypothesis is true. Realistically, it accumulates evidence that the true value of the parameter is not that far away from the point in question. Point null hypotheses are to some extent rather artificial constructs and most often you do not truly believe they will be exactly true.
It becomes much more reasonable to talk about the non-rejection supporting the null hypothesis, if you can meaningfully reverse null and alternative hypothesis and if when doing so you would reject your new null hypothesis. When you try to do that with a standard point null hypothesis you immediately see that you can will never manage to reject its complement, because then your inverted null hypothesis contains values arbitrarily close to the point under consideration.
On the other hand, if you, say, test the null hypothesis $H_0: |\mu| \leq \delta$ against the alternative $H_A: |\mu| > \delta$ for the mean of a normal distribution, then for any true value of $\mu$ there is a sample size - unless unrealistically the true value of $\mu$ is $-\delta$ or $+\delta$ - for which we have almost 100% probability that a level $1-\alpha$ confidence interval will fall either completely within $[-\delta, +\delta]$ or outside of this interval. For any finite sample size you can of course get confidence intervals that lie across the boundary, in which case that is not all that strong evidence for the null hypothesis.
It rather depends on how you are using language. Under Pearson and Neyman decision theory, it is not evidence for the null, but you are to behave as if the null is true.
The difficulty comes from modus tollens. Bayesian methods are a form of inductive reasoning and, as such, are a form of incomplete reasoning. Null hypothesis methods are a probabilistic form of modus tollens and as such are part of deductive reasoning and therefore is a complete form of reasoning.
Modus tollens has the form "if A is true then B is true, and B is not true; therefore A is not true." In this form, it would be if the null is true then the data will appear in a particular manner, they do not appear in that manner, therefore (to some degree of confidence) the null is not true (or at least is "falsified."
The problem is that you want "If A then B and B." From this, you wish to infer A, but that is not valid. "If A then B," does not exclude "if not A then B" from also being a valid statement. Consider the statement "if it is a bear, then it can swim. It is a fish (not a bear)." The statements say nothing about the ability of non-bears to swim.
Probability and statistics are a branch of rhetoric and not a branch of mathematics. It is a heavy user of math but is not part of math. It exists for a variety of reasons, persuasion, decision making or inference. It extends rhetoric into a disciplined discussion of evidence.
I'll try to illustrate this with an example.
Let us think that we are sampling from a population, with an intention of test for its mean $\mu$. We get a sample with mean $\bar{x}$. If we get a non-significant p-value, we would also get non-significant p-values if we had tested for any other null hypothesis $H_0:\mu=\mu_i$, such that $\mu_i$ is between $\mu_0$ and $\bar{x}$. Now for what value of $\mu$ do we have evidence?
Also when we get significant p-values, we do not obtain evidence for a particular $H_1:\mu=M$, instead it is an evidence against $H_0:\mu=\mu_0$ (which can be tought as evidence for $\mu\ne\mu_0$, $\mu<\mu_0$ or $\mu>\mu_0$ depending on situation). Nature of hypothesis testing do not provide evidence for something, it does only against something, if it does.
Consider the small dataset (illustrated below) with mean $\bar x \approx 0$, say that you conducted a two-tailed $t$-test with $H_0: \bar x = \mu$, where $\mu = -0.5$. The test appears to be insignificant with $p > 0.05$. Does that signify that your $H_0$ is true? What if you tested against $\mu = 0.5$? Since the $t$ distribution is symmetric, the test would return a similar $p$-value. So you have approximately the same amount of evidence that $\mu = -0.5$ and that $\mu = 0.5$.
The above example shows that small $p$-values lead us away from believing in $H_0$ and that high $p$-values suggest that our data is somehow more consistent with $H_0$, as compared to $H_1$. If you conducted many such tests, then you could find such $\mu$ that is most likely given our data and in fact you would be using semi-maximum likelihood estimation. The idea of MLE is that you seek for such value of $\mu$ that maximizes the probability of observing your data given $\mu$, what leads to likelihood function
$$ L(\mu | X) = f(X | \mu) $$
MLE is a valid way of finding the point estimate for $\hat\mu$, but it tells you nothing about probability of observing $\hat\mu$ given your data. What you did is you picked a single value for $\hat\mu$ and asked about probability of observing your data given it. As already noticed by others, $f(\mu|X) \ne f(X|\mu)$. To find $f(\mu|X)$ we would need to account for the fact that we tested against different candidate values for $\hat\mu$. This leads to Bayes theorem
$$ f(\mu|X) = \frac{ f(X|\mu) \, f(\mu) }{ \int \, f(X|\mu) \, f(\mu) \, d\mu } $$
that first, considers how likely are different $\mu$'s a priori (this can be uniform, what leads to results consistent with MLE) and second, normalizes for the fact that you considered different candidates for $\hat\mu$. Moreover, if you ask about $\mu$ in probabilistic terms, you need to consider it as a random variable, so this is another reason for adopting Bayesian approach.
Concluding, hypothesis test tells you if $H_1$ is more likely then $H_0$, but since the procedure needed you to assume that $H_0$ is true and to pick a specific value for it. To give an analogy, imagine that your test is an oracle. If you ask her, "the ground is wet, is it possible that it was raining?", she'll answer: "yes, it is possible, in 83% of cases when it was raining, the ground become wet". If you ask her again, "is it possible that someone just spilled the water on the ground?", she'll answer "sure, it is also possible, in 100% of cases when someone spilled water on the ground, it become wet", etc. If you ask her for for some numbers, she will give them to you, but the numbers would not be comparable. The problem is that the hypothesis test/oracle operates in a framework, where she can give conclusive answers only for the questions asking if the data is consistent with some hypothesis, not the other way around, since you are not considering other hypotheses.
Both null and alternative hypothesis are models, and as such different from reality and never true. A rejection of the null hypothesis says that the data are not compatible with the null hypothesis, because if the null hypothesis were true, such data would not normally be observed. A non-rejection of the null hypothesis says that the data are compatible with the null hypothesis, meaning that if data are generated from the null hypothesis, they could very well look like the data we have.
But all data is compatible with many models. For example, if our null hypothesis is ${\cal N}(0,1)$ and we observe data with mean 0.01 and sample variance 1.13, data are surely also compatible with a ${\cal N}(0.01,1.13)$ distribution (and all kinds of other distributions "in between"), even though they will (if the sample size is not excessively large) not reject ${\cal N}(0,1)$. Furthermore the data will be compatible with lots of non-normal models with similar expected values and variances, and with all kinds of distributions with dependence or non-identity structures that fit the data well as they are (including a crazy dependence structure that says that if we observe the first observation as it is, all else will happen as it happened with probability 1). In reality nothing is really identical and nothing is really independent, and for sure nothing is really normally distributed, so really the best we can say is that data are compatible with the $H_0$, which means that nobody claiming anything substantially different can use them as argument.
This issue by the way is not specific to tests and null hypotheses, and the only way Bayesian analysis can get around it is by going 100% subjective. Surely if a Bayesian says that the probability that model X is true is 87.5%, that's nonsense. The probability that any model is really true is zero. They are models and not reality after all.
Let's follow a simple example.
My null hypothesis is that my data follows a normal distribution. The alternative hypothesis is that the distribution for my data is not normal.
I draw two random samples from an uniform distribution on [0,1]. I can't do much with just two samples, thus I wouldn't be able to reject my null hypothesis.
Does that mean I can conclude my data follows normal distribution? No, it's an uniform distribution!!
The problem is I have made the normality assumption in my null hypothesis. Thus, I can't conclude my assumption is correct because I can't reject it.
Rejecting $H_0$ requires your study to have enough statistical power. If you're able to reject $H_0$, you can say that you have gathered sufficient data to draw a conclusion.
On the other hand, not rejecting $H_0$ doesn't require any data at all, since it's assumed to be true by default. So, if your study doesn't reject $H_0$, it's impossible to tell which is more probable: $H_0$ is true, or your study simply wasn't large enough.
No, it is not evidence unless you have evidence that it is evidence. I'm not trying to be cute, rather literal. You only have probability of seeing such data given your assumption the null is true. That is ALL you get from the p-value (if that, since the p-value is based on assumptions themselves).
Can you present a study that shows that for studies that "fail" to support the null hypothesis, a majority of the null hypotheses turn out to be true? If you can find THAT study, then your failure to disprove the null hypotheses at least reflects a VERY generalized likelihood that the null is true. I'm betting you don't have that study. Since you don't evidence relating to null hypotheses being true based on p-values, you just have to walk away empty-handed.
You started by assuming your null was true to get that p-value, so the p-value can tell you nothing about the null, only about the data. Think about that. It's a one-directional inference - period.
