Let's say that I score (with, for example, a logit model) a group of 10,000 customers according to their potential of buying a product, and that I decide to contact the top 1,000 with a special offer. I use the customers previous behavior as my training data. This far this is just a normal classification problem. However, my question concerns my long-term plan: how should I incorporate the results from this campaign into my next scoring? Let's say that I want to score customers and send out new offers once every few months.
The feedback from the latest round (= the knowledge of who picked up on the offer and who didn't) is valuable information in order to tune the model. But at the same time, the sample from which this new information is derived is not a representative sample of the whole customer database (I only get feedback from the 1,000 with the highest scores).
If I train a new model based purely on the latest round I will quickly move towards a model that completely neglects the lower scoring customers. Yet, if I only used the original data (and updating the variables linked to the 1,000 customers picked for the campaign) I feel that I don't put enough weight on the new information AND this also introduces a bias: as only the 1,000 picked customers receive updated values, it is more likely that they may (or may not) me picked in the next round. For example, if a central variable in my model is
recency (eg. the shorter period of time since the last campaign, the more likely the customer is to buy again), the recently picked would have an unfair advantage over other, latent potential in the customer database.
What options and solutions are usually implemented in these situations? let's say that this scoring process needs to be repeated dozens of times.