0
$\begingroup$

In AB testing context, if we have a control group and test group (2 groups), and I'd like to calculate the relative difference (Mean test/ Mean control -1) and the confidence interval of this ratio difference, I would need to calculate the standard error for the ratio difference.

I've seen a calculation of the standard error to be: $$ \sqrt{\left [ SE^{2}_{test} + \left ( Mean_{test}/Mean_{control} \right )^{2} * SE_{control}^{2} \right ]/Mean_{control}^{2}} $$

But I can't seem to find information on how this is derived. Does any one have pointers?

$\endgroup$
0

2 Answers 2

2
$\begingroup$

I think the most elegant derivation is for the special case where everything is positive. Write $\mu_C, \sigma^2_C$ for the control mean and variance, and $\mu_T,\sigma^2_T$ for the treatment ones.

Since $d\log x/dx$ is $1/x$, the variance of $\log\hat\mu_C$ is approximately $\sigma^2_C/\mu^2_C$, and similarly for the treatment group. So variance of $\log R=\log\hat\mu_T-\log\hat\mu_C$ is approximately $\sigma^2_T/\mu^2_T+\sigma^2_C/\mu^2_C$. Now, $d\exp(x)/dx=\exp(x)$, so the variance of $R$ is approximately $$\left(\sigma^2_T/\mu^2_T+\sigma^2_C/\mu^2_C\right)R^2=\left(\frac{\sigma^2_T}{\mu^2_T}+\frac{\sigma^2_C}{\mu^2_C}\right)\frac{\mu^2_T}{\mu^2_C},$$ and the standard error of $R$ is the square root of that, which is what you have.

You can also derive it more directly using the quotient rule for derivatives (and you basically have to when the two variables are not independent) but I always find that more difficult to remember

$\endgroup$
0
$\begingroup$

The image from this question might clarify the Delta method for a ratio of distributions

joint distribution along with ratio distribution

In this case the variables have $\mu_X = \mu_Y = 1$ and you get the approximation

$$\frac{X}{Y} \approx 1 + X - Y$$

We could use this result and generalize it by using the parameterization $X = U/\mu_U$ and $Y = V/\mu_V$,then we get

$$\frac{U/\mu_U}{V/\mu_V} \approx 1 + U/\mu_U - V/\mu_V$$

or

$$\frac{U}{V} \approx \mu_U/\mu_V + \frac{1}{\mu_V} (U - V\frac{\mu_U}{\mu_V}) $$

then

$$\sigma_{U/V} \approx \frac{1}{\mu_V} \sqrt {\sigma_U^2 + \frac{\mu_U^2}{\mu_V^2}\sigma_V^2}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.