So, working in web analytics, it's common to use a chi-square test for A/B tests to test statistical significance. I often use a calculator like this: http://www.usereffect.com/split-test-calculator , so if each variation gets 1000 impressions, and A has 100 conversions and B has 200 conversions, I know that, with a 99.9% confidence interval (assuming randomized samples) that B is the winning variation.

But occasionally, I have values that create percentages over 100%, and seem to throw these calculations off.

So, if I'm testing two donation pages, and I want to use the $ values for conversions, how would I go about that? My goal is to make a statistically informed decision about which page is performing better, in terms of its ability to raise more money.

If I have 1000 impressions of each variation, and one variation raises $2000 off of 200 contributions, and the othe other raises $2500 off of 100 contributions, what calculation can I use to know if the difference is meaningful?

  • 1
    $\begingroup$ 2500 - 2000 = $500. Whether that's "meaningful" or not depends on how much you value this money ;-). $\endgroup$ – whuber Dec 30 '10 at 20:09
  • $\begingroup$ @whuber I can tell that one made more money, but I want to know if how to tell if that difference is likely due to chance, or if its due to the inherent superiority of the other page. $\endgroup$ – Yahel Dec 30 '10 at 20:12
  • 1
    $\begingroup$ You can't answer that with the information given. If your pages are offered randomly (using a random number generator) to all comers during the same period and you have the individual "conversion" values, then you can use simple, standard techniques for testing whether the difference can be attributed to non-chance mechanisms. $\endgroup$ – whuber Dec 30 '10 at 20:15
  • $\begingroup$ I probably misunderstood the question, but where is the >100% value coming from? $\endgroup$ – nico Dec 30 '10 at 21:09
  • 2
    $\begingroup$ @Nico You're right to wonder that. The references to percentages and, indeed, to chi squared are red herrings. In the new situation the observations are total dollar returns on page visits, not a percentage of anything (so of course a chi-squared calculation is wholly invalid). In the old situation the observations are counts of "conversions" and visits for each page. Chi squared enters as one (of several) possible approximations to a test of proportions. (In both situations it's necessary that conversions be serially independent, which is a dubious but often necessary assumption.) $\endgroup$ – whuber Dec 30 '10 at 23:53

I assume you have data on the amount of each donation, not just the two totals for each variation. You don't want to use Pearson's chi-square test; that's only appropriate for counts or proportions, not for amounts. You want an independent (unpaired) two-sample test for equality of location. The classic such 'parametric' test is Student's t-test. As it's quite plausible that the variances differ as well as the means, so I'd suggest you use Welch's t-test, which is a slight variation that allows for unequal variances.

You might also consider 'non-parametric' ('distribution-free') tests, of which the obvious one is the Mann–Whitney U test, but as your outcome of interest is clearly the difference in mean donation and your sample size is reasonably large, you're almost certainly better off sticking with parametric tests such as the above t-tests that use the mean as the location parameter and so directly address the null hypothesis of equality of means of the two samples.

By the way, don't forget to include the people who saw either variation of the donation page but didn't donate anything as donations of value zero.

  • $\begingroup$ With the inclusion of all the non-donators, it seems that 80-90% of the data will be 0. Is the $t$-test still sufficiently robust if there's such a strong deviation from normality? It would also be interesting to know if the data is essentially categorical because people typically donate 5, 10, 15 dollars, ... That would mean a lot of ties, and Mann-Whitney wouldn't be appropriate. $\endgroup$ – caracal Dec 31 '10 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.