1
$\begingroup$

I am wondering what the best way is to model an A/B test. Meaning, if I run an a/b test, I want to run a model to determine if A or B is better for each individual.

A/B would be assigned randomly (50% get A, 50% get B for example)

My thinking: Split data in training/validation/test

On training, run a model on A customers and run a separate model on B customers (response variable being conversion yes/no). I would use a logistic regression for this.

Then run both models on the validation/test data to see if A or B estimates a better chance of conversion. Whichever estimates a higher % chance is the predicted treatment (A or B).

Then I would calculate the uplift in conversion by comparing the overall conversion rate compared to the conversion rate of those that got the correct treatment (IE their actual treatment, A or B, matched their predicted treatment)

Does this sound like the right approach?

Improve this question
$\endgroup$
2
  • $\begingroup$ It rather depends on how you selected candidates for A and for B $\endgroup$
    – Henry
    Commented Dec 8, 2016 at 21:43
  • $\begingroup$ True, I edited my post. It is assigned randomly, 50% A 50% B $\endgroup$
    – Adam Sanders
    Commented Dec 8, 2016 at 21:47

2 Answers 2

Reset to default
1
$\begingroup$

This is an old question, but I think it might benefit from another answer (at least in the name of posterity).

First, there is no need to run the sort of modelling you describe. Given a logistic regression model to estimate the propensity of the outcome, the difference under treatment A or treatment B can be determined from the coefficient of the treatment $\beta$. Assuming $\beta$ is the coefficient for a dummy variable indicating exposure to treatment $B$ then $\beta>0$ means that the predicted propensity is larger for $B$ than $A$. Hence, you can use all your data for estimation, leading to more precise (certain) estimates.

The answer will be the same regardless of the who you examine (its a good exercise to show this is the case) unless you specify interactions between the treatment and any other covariates for your customers. One way you might want to do this is to model continuous covairates with splines and specify an interaction between the spline and the treatment. Then you can perform the sort of uplift analysis you describe. But without the interaction, the conclusion will either result in everyone getting A or everyone getting B (which is kind of the point of AB testing).

Improve this answer
$\endgroup$
0
$\begingroup$

If at the conclusion of your A/B experiment, you know the conversion status (yes/no) for each individual then you can take the following simple approach:

Simply conduct a statistical test to determine whether the difference in the average conversion rate in A vs. B is statistically greater than zero (i.e. A is better), less than zero (i.e. B is better) or equal to zero (no difference in conversion rates between the group).

You can use an unpaired t-test (assumes that conversion rates are normally distributed) or a Mann-Whitney U test (does not require normality) to compare the average conversion rates.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I am trying to go beyond the test between A/B and use a model to dynamically assign customers A or B depending on their characteristics. I would know the overall difference between A and B of course, but there are situations in which A might be better and other situations where B would be better, so I want to use a model to determine which situations warrant each. $\endgroup$
    – Adam Sanders
    Commented Dec 8, 2016 at 22:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.