3
$\begingroup$

If I flip a coin 1000 times and get only one head, I may suspect that the coin is biased. One justification for this suspicion is that I am unlikely to get so few heads under the null hypothesis of the coin being fair.

However, if I flip a coin 1,000 times and get precisely 500 heads and 500 tails, I might have the opposite suspicion: some force is intervening to keep the results perfectly in line with the null hypothesis. One realistic scenario we might see this is in circumstances where people try to correct for bias, e.g. with hiring demographics.

Is there a standard way of formalizing this?

One simple thing is to just look at $1 - p$, and if $1 - p <\alpha$ we could say that $H_0$ can be "anti-rejected" at $\alpha$.

A more complicated thing would be to consider some set of alternatives $H_1,\dots, H_n$ and consider something like $\sum_i P (X\gt x | H_i) P(H_i)$. If this is below some threshold, we might consider all of the alternatives to have been rejected, and therefore we should accept $H_0$.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Some comments on "too good to be true" at stats.stackexchange.com/questions/473390/one-sided-chi2-test may help here. $\endgroup$
    – Nick Cox
    Jun 25, 2020 at 0:33
  • 1
    $\begingroup$ High p-values wouldn't lead you to doubt the null you're presently considering. Such a value might lead you to consider a different hypothesis with an alternative consistent with what caused that unusually high p-value, but you can't test it on the data that caused you to form the new hypothesis. $\endgroup$
    – Glen_b
    Jun 25, 2020 at 2:52

3 Answers 3

Reset to default
2
$\begingroup$

I struggle to see the usefulness of this idea. Using your example of flipping a coin, the case where you get a perfect split of heads and tails is the most likely outcome. Likewise, you could argue if you get 604 heads and 396 tails that this is a remarkable outcome because of of the unlikely event of getting exactly that outcome. The p-value gives the probability of getting a result as extreme as the one you did assuming the null is true, so large p-values only tell you that you got an outcome that was very likely to happen.

As a general response to the question of taking the complement of the alpha level, is it not more useful to re-frame the hypothesis such that you can conduct your statistical test as per usual. As in, set the null hypothesis to be that the coin is biased in a certain way and calculate the probability of getting a 50/50 split under that assumption.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Not sure I agree with the "most likely outcome" bit. We usually don't look at $P(X = x)$ and instead consider (say) $P(| X -\mu |\gt x)$; the complement is $P(| X -\mu |\lt x)$ and that may be very unlikely indeed for small values of $x$. $\endgroup$
    – Xodarap
    Jun 25, 2020 at 0:19
  • $\begingroup$ @Xodarap Personally, I wouldnt interpret "the most likely outcome" of a discrete random variable to refer to anything other than P(X = x). $\endgroup$
    – Ryan Volpi
    Jun 25, 2020 at 2:21
  • 1
    $\begingroup$ About usefulness of very large p-values, I've got p-values near 1 in t-tests that failed to meet the normality of means assumption. In general, a p-value=1 should be seen at least as a warning that something might be wrong. $\endgroup$
    – Pere
    Jun 25, 2020 at 9:19
2
$\begingroup$

The trouble with this is that, yes, 500 heads and 500 tails is awful evidence against $H_0: p=0.5$. However, that is also awful evidence against $H_0: p=0.50000001$.

Well which is it, $0.5$ or $0.50000001?$ Those numbers aren’t equal. Sure, they’re close, but they’re not equal.

You don’t know which it is, so you don’t really have evidence in favor of $p=0.5$.

(And $0.49999$. And $0.500103$. And $0.500063$. So many other values of $p$ are totally plausible for 500 heads and 500 tails.)

What you can do is something like two one-sided tests: TOST. The gist of TOST is to show that $p>0.501$ is unlikely and $p<0.499$ is unlikely, so you have confidence that $p\in(0.499,0.501)$.

https://en.wikipedia.org/wiki/Equivalence_test

Improve this answer
$\endgroup$
0
$\begingroup$

One option could be to run repeated experiments and test whether the results come from the expected distribution. For example, you can conduct multiple experiments flipping 50 heads, record the number of heads in each, and test whether the distribution of outcomes comes from the expected binomial distribution.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.