Is there a name for, or interpretation of, very high p values? If I flip a coin 1000 times and get only one head, I may suspect that the coin is biased. One justification for this suspicion is that I am unlikely to get so few heads under the null hypothesis of the coin being fair.
However, if I flip a coin 1,000 times and get precisely 500 heads and 500 tails, I might have the opposite suspicion: some force is intervening to keep the results perfectly in line with the null hypothesis. One realistic scenario we might see this is in circumstances where people try to correct for bias, e.g. with hiring demographics.
Is there a standard way of formalizing this?
One simple thing is to just look at $1 - p$, and if $1 - p <\alpha$ we could say that $H_0$ can be "anti-rejected" at $\alpha$.
A more complicated thing would be to consider some set of alternatives $H_1,\dots, H_n$ and consider something like $\sum_i P (X\gt x | H_i) P(H_i)$. If this is below some threshold, we might consider all of the alternatives to have been rejected, and therefore we should accept $H_0$.
 A: The trouble with this is that, yes, 500 heads and 500 tails is awful evidence against $H_0: p=0.5$. However, that is also awful evidence against $H_0: p=0.50000001$.
Well which is it, $0.5$ or $0.50000001?$ Those numbers aren’t equal. Sure, they’re close, but they’re not equal.
You don’t know which it is, so you don’t really have evidence in favor of $p=0.5$.
(And $0.49999$. And $0.500103$. And $0.500063$. So many other values of $p$ are totally plausible for 500 heads and 500 tails.)
What you can do is something like two one-sided tests: TOST. The gist of TOST is to show that $p>0.501$ is unlikely and $p<0.499$ is unlikely, so you have confidence that $p\in(0.499,0.501)$.
https://en.wikipedia.org/wiki/Equivalence_test
A: I struggle to see the usefulness of this idea. Using your example of flipping a coin, the case where you get a perfect split of heads and tails is the most likely outcome. Likewise, you could argue if you get 604 heads and 396 tails that this is a remarkable outcome because of of the unlikely event of getting exactly that outcome. The p-value gives the probability of getting a result as extreme as the one you did assuming the null is true, so large p-values only tell you that you got an outcome that was very likely to happen.
As a general response to the question of taking the complement of the alpha level, is it not more useful to re-frame the hypothesis such that you can conduct your statistical test as per usual. As in, set the null hypothesis to be that the coin is biased in a certain way and calculate the probability of getting a 50/50 split under that assumption.
A: One option could be to run repeated experiments and test whether the results come from the expected distribution. For example, you can conduct multiple experiments flipping 50 heads, record the number of heads in each, and test whether the distribution of outcomes comes from the expected binomial distribution.
