Let's say that I score (with, for example, a logit model) a group of 10,000 customers according to their potential of buying a product, and that I decide to contact the top 1,000 with a special offer. I use the customers previous behavior as my training data. This far this is just a normal classification problem. However, my question concerns my long-term plan: how should I incorporate the results from this campaign into my next scoring? Let's say that I want to score customers and send out new offers once every few months.

The feedback from the latest round (= the knowledge of who picked up on the offer and who didn't) is valuable information in order to tune the model. But at the same time, the sample from which this new information is derived is not a representative sample of the whole customer database (I only get feedback from the 1,000 with the highest scores).

If I train a new model based purely on the latest round I will quickly move towards a model that completely neglects the lower scoring customers. Yet, if I only used the original data (and updating the variables linked to the 1,000 customers picked for the campaign) I feel that I don't put enough weight on the new information AND this also introduces a bias: as only the 1,000 picked customers receive updated values, it is more likely that they may (or may not) me picked in the next round. For example, if a central variable in my model is recency (eg. the shorter period of time since the last campaign, the more likely the customer is to buy again), the recently picked would have an unfair advantage over other, latent potential in the customer database.

What options and solutions are usually implemented in these situations? let's say that this scoring process needs to be repeated dozens of times.


I can be wrong but it seems to me there's no sense in updating model and scores for the subsample that you will be not taking in your future panel. First, it is unlikely that changing behaviours of those 1000 you select for each next wave to continue with may influence behaviours of the rest, whom you don't select, since every one of your sample lives their own lives: you don't mean to model society dynamics (with its mass-media feedbacks etc.). Second, even if you mean it, future reactions of the 1000 should update those past behaviours of the rest of the sample - the behaviours that were predictors of scores in your initial model; but they can't: the rest are all "dead" by the time of follow-up with those 1000 and their "past" is frozen forever. So, I believe you should just go on with selected 1000 and do future re-modelling/re-scoring only with them. Of course, important variables such as "recency" should be properly taken into account.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.