3
$\begingroup$

We developed a standard lead scoring model using logistic regression on couple of months worth data. The model has been working and we have been pushing only top 1/3 leads to sales team basis that. The model is giving around 40% lift.

This model is already 2.5 months old and we are planning to update / retrain the model after adding new data along with conversion results.

I am concerned that since we were only pushing qualified leads to sales team, we do not have the conversion result for low quality leads and hence they will have to be excluded from the model but this in turn would mean that the model will get trained on a dataset which is systematically different from ground reality - how to fix this?

$\endgroup$
10
  • $\begingroup$ Is it too expensive to train the model from scratch including low quality leads? $\endgroup$ Oct 7, 2021 at 18:51
  • 1
    $\begingroup$ @AdrianKeister: I imagine that kind of defeats the entire purpose of having a model in the first place, which is to detect the high quality leads and to concentrate resources (sales people's time) on them. Still, it may well be that at least a few low quality leads need to be "seeded" into the selection. Ideally without telling the sales team - otherwise, they will ignore the low quality ones. $\endgroup$ Oct 7, 2021 at 18:53
  • 1
    $\begingroup$ @AdrianKeister, Stephan thanks for the comments. I guess the only feasible way is to use few low quality leads in actual sales and then over sample them during retraining - though not ideal, we will have to live with this. If anything else comes to mind - please do share!! $\endgroup$ Oct 8, 2021 at 4:18
  • 1
    $\begingroup$ @DeepakAgarwal: oversampling won't be much help. It simulates knowledge you don't have, and your model will be too sure of its conclusions. Also: Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? $\endgroup$ Oct 8, 2021 at 5:14
  • 1
    $\begingroup$ What is a lead scoring model? $\endgroup$ Oct 8, 2021 at 17:19

2 Answers 2

1
$\begingroup$

You could combine the models.

Let the refined/upgraded/updated model only make decisions for the type of data for which it has been trained.

  • Let the old model still select the top 1/3 (because the new model doesn't now much better about the bottom 2/3)
  • Let the new model remove some from that 1/3. (Which it should do well and without selection bias)
  • Then have the old model add some extra for the amount that the new model removed.

And you can keep updating the model that works on the top 1/3.

To update the model that looks at all cases and selects the top 1/3 you need to make sure that it can train on more data. You do not have to add so much for this. You want anyway the model to fit the top very well and not the bottom. The model isn't gonna become extremely better in the bottom and suddenly place low cases in the top 1/3.

$\endgroup$
1
  • $\begingroup$ Interesting approach! Definitely worth looking into. @Sextus thanks!! $\endgroup$ Oct 10, 2021 at 18:09
0
$\begingroup$

This specific flavor of selection bias is called rejection bias, at least in credit decisions. Searching for that term should find some useful information.

I know of two ways proposed to mitigate the effects:

  1. Passthroughs. Randomly select some of the model-rejects to actually approve (and thereby eventually get labels and be able to include in future retraining). This was mentioned in the comments, and I agree that oversampling/upweighting these passthroughs in the model training makes sense: you want to inform the model that the real distribution has more of "these kind" of datapoints. However, in a test at my company we found that the optimal weighting was much less than what would bring the passthrough sample back to its true proportion, perhaps because we didn't pass through enough to be fully representative?

  2. Reject inference models. Use your old model (or if available, some third data source) to score the goodness of the rejected population, and use that as a proxy for actual performance to include them in the retraining dataset. (There are several ways to specify the proxy: depending on the model, you can use probabilities instead of hard classes, or you can flip score-weighted coins, or duplicate the samples once with each class and give them weights equal to the scores, etc.) There are a few papers on this, and the benefits are debated. I think the balance to strike is between retaining the good information from the old model while forgetting the bad (or no longer relevant) information. See also What is "reject inferencing" and how can it be used to increase the accuracy of a model?

Passthroughs obviously cost something to gather. A colleague of mine did an analysis on a historical chunk of our data where we had opened up quite a bit (so he could set a more business-as-usual threshold to simulate what a passthrough would look like). For us, a reject inference model did improve the retrained model's score, but not nearly as much as a passthrough did, and the cost of the passthrough was dwarfed by the improvement of the retrained model. But I expect all that depends greatly on business specifics.

The answer by Sextus Empiricus feels like it's similar to a reject inference: you trust the old model to rank-order the whole population well, and in particular you trust its performance on the lower-quality datapoints and incorporate the new information from the higher-quality approved datapoints.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.