5
$\begingroup$

I came across this question on cross-validated around bias of a coin. My initial instinct was to go for a chi-square test.

The other answers provided were also correct with binomial probability calculation and normal approximation.

There is a slight variation in the p-values because of continuity corrections.

Briefly, the question asked is "out of 900 trials, we have 490 heads - is the coin biased?"

p-value: binomial: 0.008468
p-value: normal approximation: 0.0078
p-value: chi-square: 0.00766

All these values are close by and would have 'rejected the null hypothesis of the coin being unbiased'

Let's assume we are testing for a significance level of 1% and the coin is moved towards being unbiased.

At some point, the binomial test will say the coin is unbiased, but normal approximation and chi-square will say biased. Because a coin toss is a binomial process, we should trust the binomial test more.

In general, what is the intuition behind picking the right test, when there are two or more hypothesis tests that could work?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Use the binomial first. If it is hard to finish the calculation for binomial, use other approximation. According to current computer speed, there should be no market for approximation. $\endgroup$
    – user158565
    Aug 6, 2019 at 19:01
  • 3
    $\begingroup$ One intuition, unfortunately too common, is to run as many tests as you can and select the one that most favors your previous beliefs, your company's position, or your career prospects. All such sardonic comments aside, I would like to suggest your question isn't sufficiently precise, because there is no circumstance that admits of just one hypothesis test. You therefore seem to be asking "how does one identify and select valid, useful statistical procedures?" That's an extremely broad question that might be difficult or impossible to address in these forums. $\endgroup$
    – whuber ♦
    Aug 6, 2019 at 19:07
  • 1
    $\begingroup$ You can compare the theory underlying the tests. Do they have the same power? What are the maintained assumptions? Everything else equal, more power is better and fewer assumptions is better. Often the theory will be asymptotic and then there can be some doubt whether, in a finite sample, the supposed theoretical superiority of some test materialises. You can do simulations or check the literature for that. $\endgroup$
    – CloseToC
    Aug 6, 2019 at 19:52
  • $\begingroup$ Well! I understand the broad nature of the questions and there is a certain business or process-specific aspect of the study that needs to be considered. The reason behind this question is an attempt to be more correct with the studies - and this in itself is arguable!. But let's say, we see 489 heads out of 900. Binomial p-value: 0.01022 while chi-square p-value: 0.009322. Strictly speaking, we fail to reject for a 1% confidence interval with binomial and reject with the chi-square test. $\endgroup$
    – Suraj Nagabhushana Ponnaganti
    Aug 6, 2019 at 20:51
  • $\begingroup$ @CloseToC, Just to confirm, power = 1-Pr(type II error), right? I like the advice around simulations and assumptions. $\endgroup$
    – Suraj Nagabhushana Ponnaganti
    Aug 6, 2019 at 20:57

1 Answer 1

Reset to default
3
$\begingroup$

The binomial test is the exact one, so if you're near the 5% level--and have a reason to reject only if the P-value $\le 0.05,$ then use the binomial test to be sure.

Suppose you are testing $H_0: p = 1/2$ vs. $H_a: p > 1/2$ based on $n = 900$ tosses. Then, under $H_0,$ the distribution of the number $X$ of Heads observed is $X \sim\mathsf{Binom}(n=500, p=1/2).$ If you observe $X = 490$ Heads, then the P-value of the test is

$$ P(X \ge 490) = 1 - P(X \le 489) = 0.0042$$

1 - pbinom(489, 900, .5) 
[1] 0.00420954

If you are doing a two-sided test $H_0: p = 1/2$ vs. $H_a: p \ne 1/2,$ then you have to include the probability of being just as far below 450 as 490 is above 450. Thus the two-sided P-value is 0.00842.

1 - pbinom(489, 900, .5) + pbinom(410, 900, 1/2)
[1] 0.00841908

Note: Because the binomial distribution is discrete, the actual significance level of this two-sided test cannot be exactly 5%. If we reject when $X \le 420$ or $X \ge 480,$ then the exact significance level is $0.04916.$

1-sum(dbinom(421:479, 900,.5))
[1] 0.04916137

For any practical purpose this is close enough to 5%. (You would have to do a million tests to notice the difference between 0.050 and 0.049.) If you use a normal or chi-squared approximation, the discreteness of the binomial distribution is not 'on display', so people don't usually know or worry about slight inconsistencies.

Improve this answer
$\endgroup$
1
  • $\begingroup$ The first p value can also be computed with pbinom(489, 900, .5, lower.tail=F) to just indicate we're consider the test statistic within the upper tail $\endgroup$
    – Estimate the estimators
    Oct 15 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.