2
$\begingroup$

I am having trouble trying to determine the error of a conversion rate in A/B testing.

My understanding is that to determine an error of a measurement, you have to perform multiple trials and look at the variance. However, in A/B testing, you do a single experiment (over the course of weeks).

So you have a single conversion rate for A, and a single conversion rate for B. So how is it that you can determine the error or do any sort of hypothesis testing if the experiment is only done once?

The closest answer I've encountered so far is in this website: https://vwo.com/blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/

The website states that you can simply calculate the standard error using: Standard Error (SE) = Square root of (p * (1-p) / n)

Why is this true? And what kind of hypothesis test can I conduct to determine significance? Two-sample test of proportions?

Any help would be appreciated.

$\endgroup$
5
$\begingroup$

You do indeed need multiple trials to calculate variance, and you have them. Each user that arrives during the test is an observation which we model as a Bernoulli random variable. In other words, they either convert (have value 1) with probability p, or they fail to convert (have value 0) with probability 1-p. I think the point of confusion is that you're treating $\hat{p}$, the estimate of p, as an observation, when really it is a statistic calculated from the set of observations. A two-sample difference of proportions is the test you most likely want to do.

One thing to note is that this assumes the conversion rate is calculated as (# users that convert)/(# users). If instead you use (# conversions)/(# users) as the conversion rate, and a user can have multiple conversions, then the formula p*(1-p)/n is not quite right for the variance. In this case you might use a Poisson model, where the variance is p/n (and the standard error is $\sqrt{p/n}$). Or you could just calculate the standard error directly without assuming a particular model for conversions per user.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks a bunch, your insight on my confusion point was pretty spot on. if a user can only have one conversion, then would the formula p*(1-p)/n apply? How would you calculate the standard error directly? $\endgroup$ – somethingstrang Jun 13 '17 at 1:30
  • $\begingroup$ Yes, if users can only convert once then that formula still applies. By 'directly' I just mean using the generic standard error formula $s=\sqrt{\Sigma_{i=1}^{n}(x_{i}-\bar{x})^2/(n-1)}$ then calculating the standard error as $s/\sqrt{n}$. $\endgroup$ – KMcC Jun 13 '17 at 17:29
1
$\begingroup$

You can determine significance properly (conversion rate for A is significantly higher than conversion rate for B) with a $\chi^2$ test of homogeneity : http://stattrek.com/chi-square-test/homogeneity.aspx?Tutorial=AP

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.