Finding incremental users using A/B testing Let's say you run an A/B test with a new webpage, and a control (the old webpage). Your new webpage has an 11% response rate, and your old one has 10% response rate. Let's assume population in these is high enough that this is significant. What techniques would you use to isolate/model the "incremental" people in the new process - that is, we assume that some people will respond to both the base webpage and the new webpage - how do we tease out the people who would only have responded to the new webpage and not the old one? The purpose of this would be to build a model to classify someone who hasn't been exposed to the experiment as an incremental user or not. Perhaps some sort of unsupervised technique ala clustering? Is there a better way to think about this problem/framework I should be thinking in?
 A: This is a very interesting question, because A/B-Tests tell us only what, not why. 
Why means two questions:


*

*Which types of users have responded to the new version, but would have not responded to the old one ?

*Which types of users have not responded to the new version, but would have responded to the old one ?


Since you have measured an overall improvement, the first group is bigger than the second one. 
My approaches ...
For 1.
Build a classification model with 


*

*class NEW_YES: Users who have responded to the new version

*class OLD_YES: Users who have responded to the old version


A user which is classified as NEW_YES (with high confidence), can be separated clearly from the users who have responded to the old version. This means, that it is a "new" type of user, who would have not responded earlier.
criticism: But such users may normally never respond ! This could be one in a million ! => Yes, but then a) the data is not sufficient and/or b) the differentation power of features will be very small (see note below).
For 2.
Build a classification model with 


*

*class NEW_NO: Users who have NOT responded to the new version 

*class OLD_NO: Users who have NOT responded to the old version


A user which is classified as NEW_NO (with high confidence), can be separated clearly from the users who have not responded to the old version. This means that this one is a "drop out".
In both cases: By using a so called white-box classification model (for example, Gradient Boosted Trees (e.g. in "The Elements of Statistical Learning")), you may gain insight which features are contributing the most to the class separation, assuming that the data is sufficient. 
Sidenote: 
Even if you have reached significance in the A/B-Test, you may not have enough data for separation. Two reasons


*

*A variety of users have responded to the new version (but would have not to the old one), but all for different
reasons. In the case significance may be reached (in total), but not enough data is available to identify each subgroup

*The motivation of users is a shady thing hard to grasp. If you do not have any pre-visit data available (earliers visits which may allow the derivation of preferences or even demographic data), you may end up with finding the "obvious" differentiation features like: "Users who have brought the improvement of revenue in the new version, put more items in the basket".


This may be the reason why I have not yet convinced myself for doing such an analysis, although I am performing A/B-tests regularly. But as a chinese proverb say: 

If you think something cannot be done, don't stand in the way of those
  who try to do it.

On the plus side, even if you may not end up with a good model, you may have found some clues which lead to new hypotheses which can be tested in new carefully crafted tests. Good luck !
A: One approach:


*

*Build and cross-validate predictive models for both treatments, based on whatever features you have available for both sets of users.

*Use each predictive model on the opposite set of users to classify how they're likely to respond to the opposite treatment.

*Compare and further explore this new (uncertain) dataset.


A few advantages to this approach:


*

*Cross-validation error estimates will help you gauge the predictive strength of your models. If you can't build good predictive models for either, then you can be reasonably convinced you can't make good predictions from the available data, and there's no teasing out to be accomplished. An unsatisfying conclusion certainly, but a time saver if it happens early on.

*It gives you predictions for both models. For each person who responded to the new page, you'll have a prediction of how they'd respond to the old one. In particular, you can see which users would only respond to one of the models. (Within the uncertainty of your models' predictions, that is.) Further exploratory analysis might help you identify drivers of response in each.


More plainly, if you can't build decent predictive models from your available data, you may have to be content with, "We're not sure why this page outperforms the old one, but it does."
