Assessing the performance of a model that has a policy that makes it worse Assuming that a churn model will be used to prevent customer churn using policies that encourage customers with higher churn score (higher likelihood of churn probability) to stay loyal to a company. How should I asses the quality, like AUC score, of my model over time?
For customers with a low churn score where nothing is supposed to be done, the model validation does not change. As for customers that a higher score (biggest chance of churn) the company will act in such a way that they remain loyal therefore when I'll be looking at the model performance in that group it will probably be affected and not be as good as it would without intervention.
I've already seen an approach where a control group is made out of some customers however no real company in the world will be willing to sacrifice some customers in order to make a rigorous statistical validation of their model.
In another words, is there any statistical way to validate the model's performance over time that takes into account policies that would worsen its performance? Or what should I be doing in this situation to asses my model performance?
 A: In order to measure your model's whole performance, you need back testing with the past data, e.g. train up to time X, and test on data after time X. But, this won't solve your problem in the future when retraining, since the data will be somehow biased. The other option is A/B testing, which your company doesn't seem to make sacrifices about. I'd try to persuade them for this option.
On the live, you can reliably measure the performance of false negatives, because no action is performed on them. And, you can associate this number with the total missed opportunity. The true positives can also be partially measured if users don't respond to actions positively, and churn anyway. Of course, this is bad for the company since it's wasted money.
But, you can't reliably know the number of people stayed just because of company actions. However, based on their engagements with the actions, you can say something about how much they've been incentivized. People who didn't engage with the offer but stayed anyway shows how many false positives you have.
Alternatively, it may even be beter to action on people in the somewhat gray area, instead of the ones with high probability of churn because they might go anyway regardless of company's actions. This decreases the wasted money and uplifts the potential lifetime of others.
