I am having trouble trying to determine the error of a conversion rate in A/B testing.
My understanding is that to determine an error of a measurement, you have to perform multiple trials and look at the variance. However, in A/B testing, you do a single experiment (over the course of weeks).
So you have a single conversion rate for A, and a single conversion rate for B. So how is it that you can determine the error or do any sort of hypothesis testing if the experiment is only done once?
The closest answer I've encountered so far is in this website: https://vwo.com/blog/what-you-really-need-to-know-about-mathematics-of-ab-split-testing/
The website states that you can simply calculate the standard error using: Standard Error (SE) = Square root of (p * (1-p) / n)
Why is this true? And what kind of hypothesis test can I conduct to determine significance? Two-sample test of proportions?
Any help would be appreciated.