I'm conducting A/B tests where outcomes are binary (success or failure in control and experiment populations).
The assessment of the risk of a Type 1 error is relatively straightforward: given an observed success rate Pe
in a population Ne
exposed to a variation, vs. a success rate Pc
in a population Nc
in the control group, we're measuring what is the probability that the difference we observe between Pe
and Pc
is due to random chance, when in reality there is no difference between the two populations. That's alpha.
For beta, I don't understand how to formulate the problem. A Type 2 error occurs if we accept the null hypothesis, when in fact there is a difference between the 2 populations. It's P(accepting Hn | Pe != Pc)
. But I don't see how I can go from that definition, to computing a percentage that would tell me, given the observations I have, how likely I am to commit a type 2 error. How can I relate that to the sizes of my samples and the size of the effect observed?
Edit: conceptually, what I struggle with is why we can measure alpha without beta, but you need to define an alpha to compute beta (as suggested here: How do I find the probability of a type II error?)