AB Testing other factors besides conversion rate So I've been learning about AB Testing and have used it to examine the form conversion rate of two different forms. However, I'm curious about testing whether a form with ads will generate more revenue than a site with just a form and no ads.
So the two things I'm comparing:
- form with ads
- form with no ads

I'm trying to see which produces more revenue.
So traditionally, a test of conversion rates would involve using a chi-square or fishers exact test to examine the values in a contingency table. I'm just not sure how to approach this question when it comes to revenue.
           Revenue     Conversions
Site 1 -    $500         100 
Site 2 -    $400         70

Is this no different that a tradition ab test of conversion rates?   
Or could I just test wether a form made money or not.
          made money       made no money
form 1       50                200
form 2       5                 250 

 A: Since you're interested in revenue, it makes more sense to compare the revenue from the 2 sites rather than whether or not they made money.  However, this means you are comparing distributions rather than contingency tables, and need different type of statistical test.  One common test of this type is the t-test.
However, the t-test assumes that your data are normally distributed, which probably isn't the case in this situation.  You can do a bit better by using a non-parametric test to compare your distributions, such as the Mann-Whitney U Test (also known as the two-sample Wilcoxon test).
In R, you can do this with the wilcox.test command.
A: To compare this two forms, you will need the number os users who filled this forms. More precisely, you need a sample from each form where the variables is the amount of money made with the user. Considering $\theta_{1}$ the vector of samples for the form with no ads and $\theta_{2}$, the samples for the form with ads, you can then do a hypothesis test on the sample mean:
$$ H_0: \bar{\theta_1} = \bar{\theta_2}$$
$$H_1: \bar{\theta_1} \neq \bar{\theta_2}$$
Where the test statistic is:
$$
z_0 = \dfrac{\theta_1 - \theta_2}{\sqrt{\dfrac{s_{\theta_1}^2}{n_1} + \dfrac{s_{\theta_2}^2}{n_2}}}
$$
$n_1$ and $n_2$ are the sample sizes, $s_{\theta_1}^2$ and $s_{\theta_2}^2$ are the sample variance.
The null hypothesis is reject if $|z_0| > z_{\alpha/2}$, considering an $\alpha$ level of significance.
