Failing to understand how p-value corresponds to significance of evidence against null hypothesis?

I'm trying to understand how the $p$-value defined as $p = P(D\ge d\ |\ H_0)$ where $D$ is a discrepancy statistic, $d$ is the observed discrepancy, and $H_0$ is the null hypothesis.

As I understand it, $D$ gives a measure of how "inconsistent" the observed data is with the null hypothesis. $D = 0$ corresponds for the "best evidence" to support the null hypothesis, whereas larger and values of $D$ indicate that the data is less consistent with $H_0$.

So, in my textbook, we have the following table:

• $p>0.10$ - No evidence against $H_0$.
• $0.05 < p \le 0.10$ - Weak evidence against $H_0$.
• $0.01 < p \le 0.05$ - Evidence against $H_0$.
• $0.001 < p \le 0.01$ - Strong evidence against $H_0$.
• $p \le 0.001$ - Very strong evidence against $H_0$.

My confusion with this correlation between $p$ and the strength of the evidence is that the $p$ value also depends on the observed data $d$.

To give some more context, the $p$-value is the probability that, given we assume the null hypothesis to be true, we observe a discrepancy greater than the initially observed discrepancy.

Edit: As @NuclearWang pointed out, these are all backwards for some reason. I'm not sure why.

Under my interpretation, if $p$ is small and $d$ is small, that's evidence supporting $H_0$, since the probability of even a moderately high discrepancy is very low, meaning discrepancies will generally be near $0$. (This is the opposite from the above list, where if $p$ is small then that's evidence against $H_0$)

Under my interpretation, if $p$ is large and $d$ is large, that's evidence against $H_0$, since if $p$ is large and $d$ is large then we still have a very high probability of discrepancies that are very far from 0, which is very inconsistent with $H_0$. (This is the opposite from the list above, where if $p$ is large then there is no evidence against $H_0$)

If $p$ is large and $d$ is small, then that's evidence against $H_0$ (sorta), since it means the discrepancies are more concentrated away from the origin. But, this could also be confusing because $p$ naturally gets closer to $1$ as $d$ gets closer to $0$ (that is, if we conducted an experiment where our initial data gave a discrepancy of $0$), meaning we could also use this as a lack of evidence against $H_0$.

However, what if $p$ is small and $d$ is large? If $d$ is large, then the probability of getting a discrepancy larger than $d$ is small regardless of $H_0$, since there are just less values that $d$ can take on, so of course $p$ is small. This isn't evidence for or against $H_0$.

I feel like it would be more worthwhile to analyze $p$ as a distribution of the sampling data. For example, we could take $p(\mathbf Y) = P(D > D(\mathbf Y)\ |\ H_0)$, and then determine if $p$ is more concentrated near its tails, or near 0 or whatnot. For example, $p$ would look linear if the probability of getting any discrepancy was equal (that is, $D$ is uniformly distributed), which is strong evidence against $H_0$, right?

Edit: I just realized that there's probably a problem with my "$p(\mathbf Y)$ above, and that's that we're assuming we know the distribution of $\mathbf Y$ beforehand, when instead that's what we're testing (I think...). So the statement $P(D > D(\mathbf Y)\ |\ H_0)$ is kind of meaningless.

So, all in all, am I misinterpretting something? Are my thoughts and ideas ok or are they way off?

• I think you might have "small" and "large" p-values backwards. Small p-values (more significant, closer to 0) are always evidence against the null hypothesis H0. Not sure what you mean about the "values that d can take", d is an observed variable that's fixed by your data. Feb 27, 2018 at 20:52
• @NuclearWang Huh, I do have them backwards. Now I'm even more confused, I thought my interpretation for the first three was alright, but I guess they're all wrong. I know that $d$ is an observed variable, but what if we were to run the experiment and we "accidentally" got a very large value of $d$ even when thats statistically very unlikely, and then we got a very small $p$ value. We would conclude that it's "very strong evidence against $H_0$" when it's not really, right? Is the fault in my interpretation of "very strong evidence against"? Feb 27, 2018 at 20:56
• Take a look at stats.stackexchange.com/questions/31.
– whuber
Feb 27, 2018 at 21:03
• p-values are only small when $d$ is big relative to what you would expect under $H_0$. So it's not clear what you mean by considering cases when the test statistic is small but the p-value is also small. The p-value is a function of the test statistic. Feb 27, 2018 at 21:12
• Another related CV thread is: Are smaller p-values more convincing? and links therein. Jan 17 at 8:28

The P-value depends on the data because it is a summary of the data. A summary of the strength, according to the statistical model, of the discrepancy between the data and the expectations regarding data when the model parameter(s) are set to the value(s) corresponding to the null hypothesis. When the p-value is small an interesting discrepancy is indicated.

The discrepancy may be due to the null hypothesis being far from the correct value of the parameter OR due to the model being badly matched to the real-world data generating process. Textbooks do not usually (ever?) tell you that last bit, but it is really important.

Notice that once the null hypothesis is appropriately linked to a model parameter it becomes logical to think about the evidence in the data as a function of values of the model parameter. That is what you have proposed with your $P(Y)=P(D>D(Y)|H_0)$, and it is an excellent idea.

The function that some feel is the most appropriate expression of the evidence concerning the values of the parameter is a likelihood function, but others have suggested various alternatives including a p-value function. Likelihood functions are rarely discussed in textbooks—an alarming shortcoming, in my opinion—and so you will need a different resource. I don't recommend the Wikipedia page unless you are mathematically adept, so see if your library has a book called Likelihood by Edwards or Statistical Evidence: a Likelihood Paradigm by Royall.

• Oh! I think you just made me understand. So, for example, if $p$ is small, then this is strong evidence against $H_0$ because it's very that we would have gotten an initial reading of $d$ at all! So $H_0$ is probably a false assumption. So, with your help, I think I've pinpointed where the fault in my understanding is. It's not so much about the distribution of all discrepancies, but more so about the probability of observing the initial data $d$ at all, given $H_0$. How is that interpretation? Feb 27, 2018 at 21:29
• I'm glad to have helped with your confusion, and your interpretation is pretty good. However, you seem to have missed the caveat about the possibility that the model is inappropriate, and I have to point out that a statement about the probability of the null being false should be based on a Bayesian analysis because it depends on the probability of the null being true independent of the evidence (that's the 'prior'). It's all much more complicated than beginners (and textbook writers) might think. Feb 28, 2018 at 0:58

where D is a discrepancy statistic, d is the observed discrepancy. As I understand it, D gives a measure of how "inconsistent" the observed data is with the null hypothesis.

No, d is the discrepancy statistic and a measure of inconsistency. D is just a dummy variable used for calculating p.

D=0 corresponds for the "best evidence" to support the null hypothesis, whereas larger and values of D indicate that the data is less consistent with H0.

d=0 is best evidence for the null in a two-tailed test, but for a one-tailed test, the best evidence for the null is as far away from the tail as possible. In this case, since you're testing the right tail, the best evidence for the null would be $-\infty$.

So, in my textbook, we have the following table:

That table's not quite right.

I can understand how, if p is small and d is small, that's evidence supporting H0, since the probability of even a moderately high discrepancy is very low, meaning discrepancies will generally be near 0.

This sort of hypothesis testing either rejects or fails to reject the null hypothesis. One is on shaky grounds claiming that low p provides "evidence" against the null, and even shakier ground claiming that a high p provides evidence for the null.

I can understand how, if p is large and d is large, that's evidence against H0, since if p is large and d is large then we still have a very high probability of discrepancies that are very far from 0, which is very inconsistent with H0.

As @Nuclear Wang said, you have it backwards.

what if we were to run the experiment and we "accidentally" got a very large value of d even when thats statistically very unlikely, and then we got a very small p value. We would conclude that it's "very strong evidence against H0" when it's not really, right?

The p value is a measure of how likely "accidentally" getting a large d is. You can get a large d either from the null hypothesis being false, or from being "unlucky". The smaller the p is, the less chance you have of being "unlucky", and therefore the more confident you can be that you got it from the hypothesis being false.

In Bayesian terms, E is evidence for H if P(E|H)>P(E|~H). So to say whether E is evidence for H, one has to have both the probability if the null hypothesis is true, and the probability if the null hypothesis is false, and with this sort of hypothesis testing the latter is not as well defined. Note, however, that whether something is "evidence" depends on its conditional probabilities. It does not depend on whether the hypothesis is actually true. A high d would be evidence against H0. If you later found out that H0 was true all along, that doesn't mean that the d wasn't evidence against H0. Evidence is about what probabilistic conclusions you can derive from the evidence at hand, not what's true from an omniscient perspective. If we had an omniscient perspective, we wouldn't need to be doing any of this in the first place.

For example, we could take p(Y)=P(D>D(Y) | H0)

What does that mean? What's D? What's D(Y)?

For example, p would look linear if the probability of getting any discrepancy was equal (that is, D is uniformly distributed), which is strong evidence against H0, right

If p is a function, then it does not depend on the observed data (and here I am making a distinction between a function varying, and the value of a function varying), so it's can't possibly be evidence for or against H0.

• My textbook defines $D$ as, quote, "a function of the data $Y$ that measures the 'agreement' between $Y$ and the null hypothesis." It also defines $d = D(\mathbf y)$ to be the observed discrepancy. Also, what do you mean by "that table's not quite right?" It's not right in that its not giving the truest picture? Feb 27, 2018 at 21:52
• Sorry, your last sentence does not make sense. P-values depend on what value is set as the null hypothesis and so they can be expressed as a function of the value set to be the null hypothesis. P-values are calculated from the data (usually via the test statistic) and so they are dependent on the data. Feb 28, 2018 at 0:54
• @Michael Lew You comment utterly fails to articulate anything that contradicts my last sentence. You seem to first be confusing p() and P-value, and second just completely ignoring the part of my sentence in parentheses. Feb 28, 2018 at 15:40
• I don't see it. Would some editing help? Feb 28, 2018 at 20:21