A/B testing confusion: finding the error in conversion rate I am having trouble trying to determine the error of a conversion rate in A/B testing.
My understanding is that to determine an error of a measurement, you have to perform multiple trials and look at the variance. However, in A/B testing, you do a single experiment (over the course of weeks).
So you have a single conversion rate for A, and a single conversion rate for B. So how is it that you can determine the error or do any sort of hypothesis testing if the experiment is only done once?
The closest answer I've encountered so far is in this website:
https://vwo.com/blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/
The website states that you can simply calculate the standard error using:
Standard Error (SE) = Square root of (p * (1-p) / n)
Why is this true? And what kind of hypothesis test can I conduct to determine significance? Two-sample test of proportions?
Any help would be appreciated.
 A: You do indeed need multiple trials to calculate variance, and you have them. Each user that arrives during the test is an observation which we model as a Bernoulli random variable. In other words, they either convert (have value 1) with probability p, or they fail to convert (have value 0) with probability 1-p. I think the point of confusion is that you're treating $\hat{p}$, the estimate of p, as an observation, when really it is a statistic calculated from the set of observations. A two-sample difference of proportions is the test you most likely want to do.
One thing to note is that this assumes the conversion rate is calculated as (# users that convert)/(# users). If instead you use (# conversions)/(# users) as the conversion rate, and a user can have multiple conversions, then the formula p*(1-p)/n is not quite right for the variance. In this case you might use a Poisson model, where the variance is p/n (and the standard error is $\sqrt{p/n}$). Or you could just calculate the standard error directly without assuming a particular model for conversions per user.
A: You can determine significance properly (conversion rate for A is significantly higher than conversion rate for B) with a $\chi^2$ test of homogeneity : http://stattrek.com/chi-square-test/homogeneity.aspx?Tutorial=AP
