A/B testing:



I'm not too familiar with A/B testing, but I was wondering if there were any specific packages/libraries in R or Python that can be used to perform A/B testing.

  • $\begingroup$ A/B testing is mainly about designs of experiments. For the issue of testing a contingency table, yes R have many available packages. $\endgroup$ – Henry Jul 6 '11 at 23:26
  • 1
    $\begingroup$ Its just a simple factorial design. If you look up any resources on conducting statistical analysis in Python or R you would be able to figure out what functions in what libraries are needed. I see in the comments for the 20bits article 'prop.test()' in R may be what your looking for. $\endgroup$ – Andy W Jul 6 '11 at 23:28

Sure, for both python and R, there are a few interesting and usable packages/libraries.

First, for python, i highly recommend reading this StackOverflow Answer directed to a question about A/B Testing in Python/Django. It's a one-page Master's thesis on the subject.

Akoha is fairly recent (a little more than one year old) package directed to AB Testing in Django. I haven't used this package but it is apparently the most widely used Django package of this type (based on number of downloads). It is available on bitbucket.

Django-AB is the other Django package i am aware of and the only one i have used.

As you would expect of Packages to support a web framework, each provides a micro-framework to setup, configure, conduct, and record the results of AB Tests. As you would expect, they both work by dynamically switching the (django) template (skeleton html page)referenced in the views.py file.

For R, i highly recommend the agricolae Package, authored and maintained by a University in Peru. available on CRAN. This is part of the core distribution. (See also agridat, which is comprised of very useful datasets from completed AB and multi-variate tests).

As far as i know, and i have referred to the agricolae documentation quite a few times, web applications or web sites are never mentioned as the test/analytical subject. From the package name, you can tell that the domain is agriculture, but the analogy with testing on the Web is nearly perfect. This package nicely complements the two Django packages because agricolae is directed to the beginning (test design and establishing success/termination criterion) and end (analysis of the results) of the AB Test workflow.


Depending on the approach you want to take to the subject, the below offered two alternatives. The first is traditional Chi-Squared Testing for Split Testing and the second is a Bayesian approach to split testing. Depending on your organizational stakeholders requirements for the analysis, you might as well do both if you have the data.

Chi-Squared Testing (Traditional) A/B Testing with Python: http://okomestudio.net/biboroku/?p=2375

Bayesian A/B Testing with Python: http://www.bayesianwitch.com/blog/2014/bayesian_ab_test.html


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.