I came across this question on cross-validated around bias of a coin. My initial instinct was to go for a chi-square test.
The other answers provided were also correct with binomial probability calculation and normal approximation.
There is a slight variation in the p-values because of continuity corrections.
Briefly, the question asked is "out of 900 trials, we have 490 heads - is the coin biased?"
p-value: binomial: 0.008468
p-value: normal approximation: 0.0078
p-value: chi-square: 0.00766
All these values are close by and would have 'rejected the null hypothesis of the coin being unbiased'
Let's assume we are testing for a significance level of 1% and the coin is moved towards being unbiased.
At some point, the binomial test will say the coin is unbiased, but normal approximation and chi-square will say biased. Because a coin toss is a binomial process, we should trust the binomial test more.
In general, what is the intuition behind picking the right test, when there are two or more hypothesis tests that could work?
Binomial p-value: 0.01022
whilechi-square p-value: 0.009322
. Strictly speaking, we fail to reject for a 1% confidence interval with binomial and reject with the chi-square test. $\endgroup$power = 1-Pr(type II error)
, right? I like the advice around simulations and assumptions. $\endgroup$