Failing to understand how p-value corresponds to significance of evidence against null hypothesis?

Question

I'm trying to understand how the $p$-value defined as $p = P(D\ge d\ |\ H_0)$ where $D$ is a discrepancy statistic, $d$ is the observed discrepancy, and $H_0$ is the null hypothesis.

As I understand it, $D$ gives a measure of how "inconsistent" the observed data is with the null hypothesis. $D = 0$ corresponds for the "best evidence" to support the null hypothesis, whereas larger and values of $D$ indicate that the data is less consistent with $H_0$.

So, in my textbook, we have the following table:

• $p>0.10$ - No evidence against $H_0$.
• $0.05 < p \le 0.10$ - Weak evidence against $H_0$.
• $0.01 < p \le 0.05$ - Evidence against $H_0$.
• $0.001 < p \le 0.01$ - Strong evidence against $H_0$.
• $p \le 0.001$ - Very strong evidence against $H_0$.

My confusion with this correlation between $p$ and the strength of the evidence is that the $p$ value also depends on the observed data $d$.

To give some more context, the $p$-value is the probability that, given we assume the null hypothesis to be true, we observe a discrepancy greater than the initially observed discrepancy.

Edit: As @NuclearWang pointed out, these are all backwards for some reason. I'm not sure why.

Under my interpretation, if $p$ is small and $d$ is small, that's evidence supporting $H_0$, since the probability of even a moderately high discrepancy is very low, meaning discrepancies will generally be near $0$. (This is the opposite from the above list, where if $p$ is small then that's evidence against $H_0$)

Under my interpretation, if $p$ is large and $d$ is large, that's evidence against $H_0$, since if $p$ is large and $d$ is large then we still have a very high probability of discrepancies that are very far from 0, which is very inconsistent with $H_0$. (This is the opposite from the list above, where if $p$ is large then there is no evidence against $H_0$)

If $p$ is large and $d$ is small, then that's evidence against $H_0$ (sorta), since it means the discrepancies are more concentrated away from the origin. But, this could also be confusing because $p$ naturally gets closer to $1$ as $d$ gets closer to $0$ (that is, if we conducted an experiment where our initial data gave a discrepancy of $0$), meaning we could also use this as a lack of evidence against $H_0$.

However, what if $p$ is small and $d$ is large? If $d$ is large, then the probability of getting a discrepancy larger than $d$ is small regardless of $H_0$, since there are just less values that $d$ can take on, so of course $p$ is small. This isn't evidence for or against $H_0$.

I feel like it would be more worthwhile to analyze $p$ as a distribution of the sampling data. For example, we could take $p(\mathbf Y) = P(D > D(\mathbf Y)\ |\ H_0)$, and then determine if $p$ is more concentrated near its tails, or near 0 or whatnot. For example, $p$ would look linear if the probability of getting any discrepancy was equal (that is, $D$ is uniformly distributed), which is strong evidence against $H_0$, right?

Edit: I just realized that there's probably a problem with my "$p(\mathbf Y)$ above, and that's that we're assuming we know the distribution of $\mathbf Y$ beforehand, when instead that's what we're testing (I think...). So the statement $P(D > D(\mathbf Y)\ |\ H_0)$ is kind of meaningless.

So, all in all, am I misinterpretting something? Are my thoughts and ideas ok or are they way off?

Answer 1

The P-value depends on the data because it is a summary of the data. A summary of the strength, according to the statistical model, of the discrepancy between the data and the expectations regarding data when the model parameter(s) are set to the value(s) corresponding to the null hypothesis. When the p-value is small an interesting discrepancy is indicated.

The discrepancy may be due to the null hypothesis being far from the correct value of the parameter OR due to the model being badly matched to the real-world data generating process. Textbooks do not usually (ever?) tell you that last bit, but it is really important.

Notice that once the null hypothesis is appropriately linked to a model parameter it becomes logical to think about the evidence in the data as a function of values of the model parameter. That is what you have proposed with your $P(Y)=P(D>D(Y)|H_0)$, and it is an excellent idea.

The function that some feel is the most appropriate expression of the evidence concerning the values of the parameter is a likelihood function, but others have suggested various alternatives including a p-value function. Likelihood functions are rarely discussed in textbooks—an alarming shortcoming, in my opinion—and so you will need a different resource. I don't recommend the Wikipedia page unless you are mathematically adept, so see if your library has a book called Likelihood by Edwards or Statistical Evidence: a Likelihood Paradigm by Royall.

Answer 2

where D is a discrepancy statistic, d is the observed discrepancy. As I understand it, D gives a measure of how "inconsistent" the observed data is with the null hypothesis.

No, d is the discrepancy statistic and a measure of inconsistency. D is just a dummy variable used for calculating p.

D=0 corresponds for the "best evidence" to support the null hypothesis, whereas larger and values of D indicate that the data is less consistent with H0.

d=0 is best evidence for the null in a two-tailed test, but for a one-tailed test, the best evidence for the null is as far away from the tail as possible. In this case, since you're testing the right tail, the best evidence for the null would be $-\infty$.

So, in my textbook, we have the following table:

That table's not quite right.

I can understand how, if p is small and d is small, that's evidence supporting H0, since the probability of even a moderately high discrepancy is very low, meaning discrepancies will generally be near 0.

This sort of hypothesis testing either rejects or fails to reject the null hypothesis. One is on shaky grounds claiming that low p provides "evidence" against the null, and even shakier ground claiming that a high p provides evidence for the null.

I can understand how, if p is large and d is large, that's evidence against H0, since if p is large and d is large then we still have a very high probability of discrepancies that are very far from 0, which is very inconsistent with H0.

As @Nuclear Wang said, you have it backwards.

[taken from comments]

what if we were to run the experiment and we "accidentally" got a very large value of d even when thats statistically very unlikely, and then we got a very small p value. We would conclude that it's "very strong evidence against H0" when it's not really, right?

The p value is a measure of how likely "accidentally" getting a large d is. You can get a large d either from the null hypothesis being false, or from being "unlucky". The smaller the p is, the less chance you have of being "unlucky", and therefore the more confident you can be that you got it from the hypothesis being false.

In Bayesian terms, E is evidence for H if P(E|H)>P(E|~H). So to say whether E is evidence for H, one has to have both the probability if the null hypothesis is true, and the probability if the null hypothesis is false, and with this sort of hypothesis testing the latter is not as well defined. Note, however, that whether something is "evidence" depends on its conditional probabilities. It does not depend on whether the hypothesis is actually true. A high d would be evidence against H0. If you later found out that H0 was true all along, that doesn't mean that the d wasn't evidence against H0. Evidence is about what probabilistic conclusions you can derive from the evidence at hand, not what's true from an omniscient perspective. If we had an omniscient perspective, we wouldn't need to be doing any of this in the first place.

For example, we could take p(Y)=P(D>D(Y) | H0)

What does that mean? What's D? What's D(Y)?

For example, p would look linear if the probability of getting any discrepancy was equal (that is, D is uniformly distributed), which is strong evidence against H0, right

If p is a function, then it does not depend on the observed data (and here I am making a distinction between a function varying, and the value of a function varying), so it's can't possibly be evidence for or against H0.