# How to measure risk of a Type 2 error in A/B tests

I'm conducting A/B tests where outcomes are binary (success or failure in control and experiment populations).

The assessment of the risk of a Type 1 error is relatively straightforward: given an observed success rate Pe in a population Ne exposed to a variation, vs. a success rate Pc in a population Nc in the control group, we're measuring what is the probability that the difference we observe between Pe and Pc is due to random chance, when in reality there is no difference between the two populations. That's alpha.

For beta, I don't understand how to formulate the problem. A Type 2 error occurs if we accept the null hypothesis, when in fact there is a difference between the 2 populations. It's P(accepting Hn | Pe != Pc). But I don't see how I can go from that definition, to computing a percentage that would tell me, given the observations I have, how likely I am to commit a type 2 error. How can I relate that to the sizes of my samples and the size of the effect observed?

Edit: conceptually, what I struggle with is why we can measure alpha without beta, but you need to define an alpha to compute beta (as suggested here: How do I find the probability of a type II error?)

New contributor
Cystack is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

The traditional way of doing this is to choose a type I error rate, e.g. $$\alpha=0.05$$, and then to specify an assumed target exposed probability $$p_e$$ and a target type II error rate $$\beta$$, which then define your sample size $$N=N_e+N_c$$ (often $$N_e=N_c=N/2$$) given an assumed $$p_c$$. I.e. we do not typically calculate $$\alpha$$ so much as simply fixing it up-front.
You can in theory do this in any other way, e.g. given an available budget that gives me a fixed $$N$$ and given that I want $$\beta=0.2$$, what $$\alpha$$ would I pick assuming specific $$p_e$$ and $$p_c$$. Or you could say, if I observe $$\hat{p}_e\geq 0.6$$ and $$\hat{p}_c=0.5$$, I want to call this significant, what $$\alpha$$ gives me that and then you next fix either $$N$$ or $$\beta$$ and then calculate the one you did not fix.
However, the traditional way of fixing $$\alpha$$ first would be by far the most common way of doing this and often there are strong conventions on what one would require. E.g. to get a new drug approved, you might often - amongst many other things - have to reject the primary null hypothesis with $$\alpha=0.05$$ in two separate confirmatory clinical trials (there are lots of rules/traditions/precedents around under what circumstances one might deviate from that) and a $$\beta \in [0.1, 0.2]$$ is very typical for such trials.