I randomly assign users to two distinct groups and track their return rate to the website (the return rate on each day for each group is simply $\frac{Number Of ReturningUsers}{TotalUsersOnDayZero}$). I have two groups, $X$ and $Y$, and I'm calculating the $SEM$ of the following relation:

$$ Z = \frac{X-Y}{Y} $$

Since the experiment runs over 7 days in a row, I'm comparing the two groups day by day.

To calculate the $ SEM $ I need first to get the Expected Value and Variance of $Z$ using Taylor Expansions

  1. The Expected Value is calculated with

$$ E[\frac{X-Y}{Y}] = \frac{E[X-Y]}{E[Y]} - \frac{Cov[X-Y,Y]}{E[Y]^2} + \frac{E[X-Y]}{E[Y]^3}Var[Y] $$

  1. The Variance is calculated with $$ Var[\frac{X-Y}{Y}] = \frac{Var[X-Y]}{E[Y]^2} - \frac{2E[X-Y]}{E[Y]^3}Cov[X-Y,Y] + \frac{E[X-Y]^2}{E[Y]^4}Var[Y] $$

Then I calculate the $SEM$ with

$$ {SEM_i} = \frac{\sigma_i}{\sqrt{N_i}} $$

where $i$ represents the day

What is the right sample size ${N_i}$ to use in that case?

  1. is it the number of users that returned on that day ($Nx_i+Ny_i$)?
  2. is it the $max(Nx_i, Ny_i)$?
  3. is it the number of days I run the experiment (7)?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.