Let's say I have a developed a model that can predicts purchase behavior of customers in terms of conversion. Furthermore, I am using this model to create a tool to be used in the workflow for the Sales Team. A salesman using the tool will see that a specific customer is going to buy something, so the salesman contacts that customer. The model only predicts probability of conversion and nothing regarding dollar amounts.
- All customers get a purchase probability score (between 0 and 1)
- Salesman sorts book of business by probability score
- Salesman contacts customers with highest probability scores (closest to 1)
Given the above, how can I show that the tool is useful in terms of increasing sales (not just conversion) for the business?
I am aware of the various ways to evaluate the model's predictions (e.g. AUROC, precision-recall), but I need to evaluate a metric that is different from the model's prediction.
I was thinking about maybe running an A/B test where
- group A uses the tool detailed above
- group B gets a placebo version of the tool where the probability of conversion is random instead of a model prediction
Then I can compare the sales generated by the groups.
Basically, I would like some feedback on whether this testing approach is sound or not.
It's not a true A/B test: One group should use the tool and the other group should not have access to it.
The fact that the model isn't 100% accurate means it will be hard to interpret results or attribute the results to the tool
If the probability isn't close to 0 or 1, that means the model is unsure of conversion and becomes equivalent of random guess.
I should just use past data (i.e. before tool creation) as control group data, and use data after tool is introduced as treated group data