I am changing one variable in a piece of software I am writing so that it has two variants. I am able to measure how many users continue to use the software a week after opening it for the first time. From this I can see that one variant of the software has retained more users. How do I test if this is significant?
Note, I'm not recording mean use time, rather, I'm recording the total number of users each of the two groups has retained. Group A has 51% of my users, group B 49%.
If the sample sizes are large enough you do can a $z$ test to test for equality of retention probability between the two groups. That is, you'd be testing whether $p_A = p_B$ or $p_A > p_B$, where $p_A$ and $p_B$ are the retention probabilities for groups A and B. To test this you let $\hat{p}_A$ and $\hat{p}_B$ be the proportion of users retained within the two groups and $\hat{p}$ the overall proportion retained. Your test statistic is then,
$$ z = \frac{\hat{p}_A - \hat{p}_B}{\sqrt{ \frac{\hat{p} (1 - \hat{p})}{n_A} + \frac{\hat{p} (1 - \hat{p})}{n_B} }} $$
which is distributed normal$(0, 1)$ when $p_A = p_B$, and you reject this hypothesis when $z$ is a large positive number (where "large" is measured in terms of how deep the observed $z$ value is into the tail of the standard normal distribution).
