How to measure risk of a Type 2 error in A/B tests

Question

I'm conducting A/B tests where outcomes are binary (success or failure in control and experiment populations).

The assessment of the risk of a Type 1 error is relatively straightforward: given an observed success rate Pe in a population Ne exposed to a variation, vs. a success rate Pc in a population Nc in the control group, we're measuring what is the probability that the difference we observe between Pe and Pc is due to random chance, when in reality there is no difference between the two populations. That's alpha.

For beta, I don't understand how to formulate the problem. A Type 2 error occurs if we accept the null hypothesis, when in fact there is a difference between the 2 populations. It's P(accepting Hn | Pe != Pc). But I don't see how I can go from that definition, to computing a percentage that would tell me, given the observations I have, how likely I am to commit a type 2 error. How can I relate that to the sizes of my samples and the size of the effect observed?

Edit: conceptually, what I struggle with is why we can measure alpha without beta, but you need to define an alpha to compute beta (as suggested here: How do I find the probability of a type II error?)

Answer 1

The traditional way of doing this is to choose a type I error rate, e.g. $\alpha=0.05$, and then to specify an assumed target exposed probability $p_e$ and a target type II error rate $\beta$, which then define your sample size $N=N_e+N_c$ (often $N_e=N_c=N/2$) given an assumed $p_c$. I.e. we do not typically calculate $\alpha$ so much as simply fixing it up-front.

You can in theory do this in any other way, e.g. given an available budget that gives me a fixed $N$ and given that I want $\beta=0.2$, what $\alpha$ would I pick assuming specific $p_e$ and $p_c$. Or you could say, if I observe $\hat{p}_e\geq 0.6$ and $\hat{p}_c=0.5$, I want to call this significant, what $\alpha$ gives me that and then you next fix either $N$ or $\beta$ and then calculate the one you did not fix.

However, the traditional way of fixing $\alpha$ first would be by far the most common way of doing this and often there are strong conventions on what one would require. E.g. to get a new drug approved, you might often - amongst many other things - have to reject the primary null hypothesis with $\alpha=0.05$ in two separate confirmatory clinical trials (there are lots of rules/traditions/precedents around under what circumstances one might deviate from that) and a $\beta \in [0.1, 0.2]$ is very typical for such trials.

On the other hand, if we are talking about A/B testing for e.g. a new feature on a webpage, then of course to some extent it is more or less up to the webpage owner how bad either mistake (a type I vs. a type II error) would be. So, in that case there are unlikely to be regulatory requirements or concerns from peer reviewers at some journal, so it may make sense to make a decision analysis (perhaps even instead of hypothesis testing).