I'm trying to understand how the $p$-value defined as $p = P(D\ge d\ |\ H_0)$ where $D$ is a discrepancy statistic, $d$ is the observed discrepancy, and $H_0$ is the null hypothesis.
As I understand it, $D$ gives a measure of how "inconsistent" the observed data is with the null hypothesis. $D = 0$ corresponds for the "best evidence" to support the null hypothesis, whereas larger and values of $D$ indicate that the data is less consistent with $H_0$.
So, in my textbook, we have the following table:
- $p>0.10$ - No evidence against $H_0$.
- $0.05 < p \le 0.10$ - Weak evidence against $H_0$.
- $0.01 < p \le 0.05$ - Evidence against $H_0$.
- $0.001 < p \le 0.01$ - Strong evidence against $H_0$.
- $p \le 0.001$ - Very strong evidence against $H_0$.
My confusion with this correlation between $p$ and the strength of the evidence is that the $p$ value also depends on the observed data $d$.
To give some more context, the $p$-value is the probability that, given we assume the null hypothesis to be true, we observe a discrepancy greater than the initially observed discrepancy.
Edit: As @NuclearWang pointed out, these are all backwards for some reason. I'm not sure why.
Under my interpretation, if $p$ is small and $d$ is small, that's evidence supporting $H_0$, since the probability of even a moderately high discrepancy is very low, meaning discrepancies will generally be near $0$. (This is the opposite from the above list, where if $p$ is small then that's evidence against $H_0$)
Under my interpretation, if $p$ is large and $d$ is large, that's evidence against $H_0$, since if $p$ is large and $d$ is large then we still have a very high probability of discrepancies that are very far from 0, which is very inconsistent with $H_0$. (This is the opposite from the list above, where if $p$ is large then there is no evidence against $H_0$)
If $p$ is large and $d$ is small, then that's evidence against $H_0$ (sorta), since it means the discrepancies are more concentrated away from the origin. But, this could also be confusing because $p$ naturally gets closer to $1$ as $d$ gets closer to $0$ (that is, if we conducted an experiment where our initial data gave a discrepancy of $0$), meaning we could also use this as a lack of evidence against $H_0$.
However, what if $p$ is small and $d$ is large? If $d$ is large, then the probability of getting a discrepancy larger than $d$ is small regardless of $H_0$, since there are just less values that $d$ can take on, so of course $p$ is small. This isn't evidence for or against $H_0$.
I feel like it would be more worthwhile to analyze $p$ as a distribution of the sampling data. For example, we could take $p(\mathbf Y) = P(D > D(\mathbf Y)\ |\ H_0)$, and then determine if $p$ is more concentrated near its tails, or near 0 or whatnot. For example, $p$ would look linear if the probability of getting any discrepancy was equal (that is, $D$ is uniformly distributed), which is strong evidence against $H_0$, right?
Edit: I just realized that there's probably a problem with my "$p(\mathbf Y)$ above, and that's that we're assuming we know the distribution of $\mathbf Y$ beforehand, when instead that's what we're testing (I think...). So the statement $P(D > D(\mathbf Y)\ |\ H_0)$ is kind of meaningless.
So, all in all, am I misinterpretting something? Are my thoughts and ideas ok or are they way off?