Can someone explain what 'comparative error rate' is?

Question

I have a set of rates of conversion for two different groups of ads. The rates are calculated by dividing conversion/impression (or, more simply, purchased/seen ad). I did a t-test for the impressions and another for the conversions - neither was significant. I wondered if I could do a t-test on the set of rates, but thought that it wouldn't come up as significant if the underlying metrics that were used to calculate the rates weren't significant. Then, of course, another coworker asked what I would do if one of the two metrics was and the other wasn't - I didn't know the answer to that.

Then they sent me the link below and said that this test suggested significance of the difference.

Can someone explain to me what the formula in this link is testing? Bonus points for telling me why I'm dumb for not thinking I can t-test two groups' differing response rates... $$ \text{Comparative Error} = 1.96 \sqrt{\frac{r_1(100-r_1)}{s_1} + \frac{r_2(100-r_2)}{s_2}} $$

Answer 1

Each set of conversions out of total adds seen is a binomial. Each rate is an estimate of the parameter, $\pi$, that controls the probability of a conversion for a given trial / instance where the add was seen by someone. A $z$-test for the difference of two proportions is the appropriate hypothesis test here. (That is, not actually the $t$-test, although they look similar.)

I think the "comparative error" formula is giving you the standard error of the difference of two proportions. Notably, $s_j$ would not be a sample standard deviation (which it might otherwise look like), but the sample size for that condition (more commonly noted $n_j$). Under some basic assumptions (e.g., $n_j(r_j)\ \&\ n_j(100-r_j) > 5$), the quotient of the observed difference of your two rates divided by the "comparative error" would be distributed as a standard normal. That means you can look up your test statistic in a normal distribution table and determine the $p$-value, i.e., the probability of getting rates that differ by as much (or more) than yours do, if there were actually no difference between the conditions.