3
$\begingroup$

there is a short introduction to AB Tests in this question or here at 20bits.

We are currently testing different versions of landing pages and are using the conversion rate (e.g. 4% vs. 5%) to track which version performs better. This is working fine so far.

What I would like to do is start calculating whether a version performs better using the sales volume. So I could say the control version sold $\$$10.000 with 100 visitors, while the new version sold $\$$12.000 with 110 visitors. Is the difference statistically significant?

I would appreciate any points in the right direction. Specifically I am having problems understanding how I can calculate 95% and 99% percentile intervals for the data above.

I can share my Excel sheet for calculating "basic" A/B tests, if that helps.

Thank you in advance for any help!

$\endgroup$

1 Answer 1

1
$\begingroup$

When you move from testing a difference in the conversion rate to testing the difference in volume, the characteristics of your underlying data is changing.

The conversion rates you were testing were proportions based on the number of "successes" out of the number of "trials" which meant their intervals were [0,1]. That means those rates can be approximated to binomial distributions. As such, the sample variance/standard deviation required for confidence intervals and hypothesis testing could be simplified to a term related to the probability of a conversion, (p*(1-p)).

On the other hand, comparing two sales volumes which would not lie on the [0,1] interval would mean you would not be using binomials. Therefore you would want to testing the difference between two means rather than two proportions. You would need to define the means you wish to test based on the business question you wanted to answer (for example: is the the test daily average of revenue, visitors, or some other metric statistically significant from the control?).

Because the data you are using is not on the [0,1] interval, your variance will not be limited in the same manner as it is when using proportions. Therefore, you will need to find the sample variances of the two groups using whatever data you were using to construct the means (both the data and the time period). Depending on the circumstance of your data acquisition, you may be able to continue to use excel or it may better to compute your variances through another package.

Assuming the two groups have the same sample sizes, you should be able to average the two variances together to find the variance that would be used in either the confidence interval or the test statistic calculation.

$\endgroup$
1
  • 2
    $\begingroup$ What about an example ? I think the OP might appreciate that (given he/she seems to have only basic knowledge of statistics). An example metric, some artificial data and an exemplary execution of a test ... this would make the answer really complete. $\endgroup$
    – steffen
    Apr 22, 2011 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.