Comparing an A/B test using cost per order?

Question

With a traditional A/B test, you have X1, X2 site visitors and Y1, Y2 conversions. I understand how this is generating two binomial distributions with sample parameters (n1, p1), (n2, p2), where n = number of site visitors (X) and p = X/Y.

Intuitively, I can see how you could do a Monte Carlo simulation here to see how often you witnessed a sample parameter of p2 or higher if the population parameter was actually p1. This alone would give you the intuition as to whether your experiment was likely to be significant or within the realm of chance. However, I also know you can simply plug these numbers into A/B test calculators to get the desired result.

My question is- how can I do the same when I want to compare cost per order? For clarity, these are marketing campaigns I'm comparing.

I have two variants, A & B, with X1, X2 in spend, and Y1, Y2 in conversions. I now care about how the spend is converted into orders (i.e. for A we spent X1 and that generated Y1 orders). But, these are no longer binomial distributions and as such I'm confused as to how we compare them.

Any help would be much appreciated!

Answer 1

You should describe more precisely what you mean by this,

I now care about what essentially amounts to the conversion rate from each $ spent to generate the conversions.

You are maybe describing a model where money spent influences the conversion rate? Or you just know exactly which model costs what and want to make a decision? But are the costs total and fixed? Or are there costs for a conversion? And how do the costs compare to a conversion?

I just cleared up the question a bit. Does it make more sense now? "I now care about how the spend is converted into orders (i.e. for A we spent X1 and that generated Y1 orders). "

The question does make sense, but more detail is needed to find a solution.

• First, do you really care about "how" the spend is converted?
• Second, let me go into a direction that you may be getting at

Let's say you know A costs 200 and B costs 400 and you know that B has significantly higher conversion. Then what do you do? You would need to know more about how much you make from a conversion. Also, translating a result on significancy to a decision is not straightforward. Mostly, you need something for effect size; ie. how much better is B over A. This is all never the outcome of a hypothesis test.

You need to be more precise in the wording to find an answer.

All this to say, you are adding money to the question, which probably needs something with expected value and some decision theory, more than just a hypothesis test. Also your model may get more complicated that a simple binomial model. For both reasons, I would suggest looking into bayesian modelling with PyMC3. This may seem like a rabbit hole at first but it will greatly improve your understanding and expand the possibilities.

• https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers.