Best way to test a hypothesis that predicts a binary result

Question

Given:

• an experiment with a yes/no result
• no error in measurement - ie a "yes" is definitely a "yes"
• experiment is performed "n" times (n is largish, say 100+)
• a hypothesis that predicts the expected proportion of yes results

What statistical test should be used to test the accuracy of the hypothesis?
How large does "n" need to be to provide reasonable confidence, and what is the confidence for a given "n"?

Note that for me, "best" means "simplest that works".

My guess is the chi-squared is appropriate, ie:

(obs_yes - exp_yes)^2/exp_yes + (obs_no - exp_no)^2/exp_no

but how to interpret the result given "n" to produce a confidence that the hypothesis is correct?

Answer 1

Are you trying to test if a specific proportion of yes's can be rejected? or are you trying to estimate the proportion of yes's with something like a confidence interval (without stating a specific proportion)? It is not clear from your question.

Either way, there are tests and confidence intervals based on the binomial distribution that are probably simpler and easier to understand than the chi-squared approach (and may be more accurate for smaller sample sizes).

See this Wikipedia article for a description of some of the intervals and links to papers comparing them and other resources.

And this article shows how to test a null hypothesis of a specific proportion using the binomial and this article has the formula for the normal approximation (for large sample sizes) to the binomial.

Answer 2

My research has uncovered a really simple approach - the G-Test.

This is a simple way that gives me the answer I want.