How to avoid selection bias while updating lead scoring (predictive) model with new data We developed a standard lead scoring model using logistic regression on couple of months worth data. The model has been working and we have been pushing only top 1/3 leads to sales team basis that. The model is giving around 40% lift.
This model is already 2.5 months old and we are planning to update / retrain the model after adding new data along with conversion results.
I am concerned that since we were only pushing qualified leads to sales team, we do not have the conversion result for low quality leads and hence they will have to be excluded from the model but this in turn would mean that the model will get trained on a dataset which is systematically different from ground reality - how to fix this?
 A: You could combine the models.
Let the refined/upgraded/updated model only make decisions for the type of data for which it has been trained.

*

*Let the old model still select the top 1/3 (because the new model doesn't now much better about the bottom 2/3)

*Let the new model remove some from that 1/3. (Which it should do well and without selection bias)

*Then have the old model add some extra for the amount that the new model removed.

And you can keep updating the model that works on the top 1/3.
To update the model that looks at all cases and selects the top 1/3 you need to make sure that it can train on more data. You do not have to add so much for this. You want anyway the model to fit the top very well and not the bottom. The model isn't gonna become extremely better in the bottom and suddenly place low cases in the top 1/3.
A: This specific flavor of selection bias is called rejection bias, at least in credit decisions.  Searching for that term should find some useful information.
I know of two ways proposed to mitigate the effects:

*

*Passthroughs.  Randomly select some of the model-rejects to actually approve (and thereby eventually get labels and be able to include in future retraining).  This was mentioned in the comments, and I agree that oversampling/upweighting these passthroughs in the model training makes sense: you want to inform the model that the real distribution has more of "these kind" of datapoints.  However, in a test at my company we found that the optimal weighting was much less than what would bring the passthrough sample back to its true proportion, perhaps because we didn't pass through enough to be fully representative?


*Reject inference models.  Use your old model (or if available, some third data source) to score the goodness of the rejected population, and use that as a proxy for actual performance to include them in the retraining dataset.  (There are several ways to specify the proxy: depending on the model, you can use probabilities instead of hard classes, or you can flip score-weighted coins, or duplicate the samples once with each class and give them weights equal to the scores, etc.)  There are a few papers on this, and the benefits are debated.  I think the balance to strike is between retaining the good information from the old model while forgetting the bad (or no longer relevant) information.  See also What is "reject inferencing" and how can it be used to increase the accuracy of a model?
Passthroughs obviously cost something to gather.  A colleague of mine did an analysis on a historical chunk of our data where we had opened up quite a bit (so he could set a more business-as-usual threshold to simulate what a passthrough would look like).  For us, a reject inference model did improve the retrained model's score, but not nearly as much as a passthrough did, and the cost of the passthrough was dwarfed by the improvement of the retrained model.  But I expect all that depends greatly on business specifics.
The answer by Sextus Empiricus feels like it's similar to a reject inference: you trust the old model to rank-order the whole population well, and in particular you trust its performance on the lower-quality datapoints and incorporate the new information from the higher-quality approved datapoints.
