Binary classification optimized on the top 1000 samples I am creating a binary classification model for marketing purposes: which customers are more likely to buy the product, whom should be contacted by the sales. I don't need a model that is good on the entire test dataset, I only need one that works very good on the top 1000 samples because that is the limitation of the sales agents.
I used cost-sensitive (the data is imbalanced) Xgboost model with ndcg@1000 eval_metric.
But it only has an effect on the number of trees. Do you know any method or model type that can be used to optimize on the top samples?
Top 1000 meaning: I take the prediction scores and sort them according to the given scores. Top 1000 samples are the samples with the 1000 highest scores.
Bonus question: Now I'm facing a problem, where out of 40.000 customers ony 5-8 are interested in the product. Do you think some anomaly detection model could be used here? The problem is that these customers don't vary that much to pick any outlier.
Thank you in advance!
 A: I think that what you want is to optimize the classification calibration. That is, you want the score produced by the classifier to be a good prediction of how likely is the customer to purchase the product. If the score is indeed a good estimate of the probability, you then choose the top 1000  estimates.
One way to optimize for calibration is the use the bier score - which is just the mean square error between your prediction (the score) and the true value (either 1 or 0)
The wikipedia entry for calibration https://en.wikipedia.org/wiki/Calibration_(statistics) You want the 2nd view - calibration in classification.
Edit Jul/11
I forget to add a very good blog on calibration, in Python. The blog addresses calibration after the model is trained - adjusting the calibration plot so that it is straighter (closer to the y=x line) - where as I was referring to adjust the hyperparameters of the classifier using Bier score. The concepts are different but related.
The blog https://wttech.blog/blog/2021/a-guide-to-model-calibration/
