Can you A/B test a predictive model?

Let's say I have a developed a model that can predicts purchase behavior of customers in terms of conversion. Furthermore, I am using this model to create a tool to be used in the workflow for the Sales Team. A salesman using the tool will see that a specific customer is going to buy something, so the salesman contacts that customer. The model only predicts probability of conversion and nothing regarding dollar amounts.

Workflow example:

• All customers get a purchase probability score (between 0 and 1)
• Salesman sorts book of business by probability score
• Salesman contacts customers with highest probability scores (closest to 1)

Given the above, how can I show that the tool is useful in terms of increasing sales (not just conversion) for the business?

I am aware of the various ways to evaluate the model's predictions (e.g. AUROC, precision-recall), but I need to evaluate a metric that is different from the model's prediction.

I was thinking about maybe running an A/B test where

• group A uses the tool detailed above
• group B gets a placebo version of the tool where the probability of conversion is random instead of a model prediction

Then I can compare the sales generated by the groups.

Basically, I would like some feedback on whether this testing approach is sound or not.

My concerns:

• It's not a true A/B test: One group should use the tool and the other group should not have access to it.

• The fact that the model isn't 100% accurate means it will be hard to interpret results or attribute the results to the tool

• If the probability isn't close to 0 or 1, that means the model is unsure of conversion and becomes equivalent of random guess.

• I should just use past data (i.e. before tool creation) as control group data, and use data after tool is introduced as treated group data

• This is a nice question!(+1) That said, they are a lot of degrees of freedom here that need to be controlled for... For starters can you please define how this "probability of conversion" is used? That is the point you need to randomise. – usεr11852 Aug 28 '18 at 20:33
• A customer will receive a probability of conversion (between 0 and 1). Salesman will sort their book of business by their probabilities of conversion and focus on the customers with high probability (i.e. closest to 1). – user1964692 Aug 28 '18 at 20:35
• OK, here is the kicker then... what did the salesman due prior to this "probability of conversion" mechanism being in place? Almost certainly it was not a dice-rolling decision. When evaluating a model we need to have a realistic baseline. Your proposal is a worst case scenario of a completely irrational salesman; e.g. salesmen are more likely to focus on location they have succeed in securing prior sales, or avoid areas who are known to have only a few customers. In addition, you need to control for salesman-related ability. (cont.) – usεr11852 Aug 28 '18 at 20:50
• What if all the good salesmen/saleswomen disregard the suggestion of the model cause "they know better" and the only users are "novices"? Do you have a way of controlling that sales-personnel uses this model and does not all cowboy on themselves? – usεr11852 Aug 28 '18 at 20:52
• You need to check how much this is going to be used and ensure random assignment. Probably you will need to stratify the assignment within sales-personnel ability and locale. Then you need to check how much compliance the personnel had with your suggestions. I would not give junk estimates to half of my test-subjects. Just tell them to do as normal. Notice that this might create an odd race-condition where the personnel not given the assignment might work harder etc. etc. especially if their pay is related to sales (check what SUTVA violation is). – usεr11852 Aug 28 '18 at 21:13