I have collected a binary classification dataset in a somewhat biased way:

  • I have thousands of unlabeled samples.
  • A small percentage of these samples belong to the positive class.
  • I know for a fact that my regressors are strongly correlated with the target.
  • Using this fact I can collect positive samples by picking samples whose regressors are in the top end of the values.
  • This way I can collect enough positive samples, and randomly pick the rest to get negative samples.
  • I don't have the luxury to label many samples, yet I need enough positive samples to get an ok model.

However, I dont know the true percentage of positive samples. When fitting a logistic regression afterwards, I notice that my output is not very well calibrated (i.e. I need to tune the threshold to output a positive classification), probably because I collected the dataset in such a biased way.

I suspect that if I weighed my labeled samples using a density estimate of the regressors on the whole data this could have some sort of "importance sampling" effect, but I am not sure.

What can I do to improve model calibration?

  • 1
    $\begingroup$ Garbage in, garbage out. How does your procedure differ from just arbitrary classifying everything that is strongly correlated with the features as a positive class? $\endgroup$
    – Tim
    Commented Apr 4, 2022 at 22:02
  • $\begingroup$ I still need to set some kind of threshold for them I guess, and I want to combine multiple regressors into a single, hopefully, calibrated score. I don't know if I would call my labeled data garbage - the problem is that it over-represents the positive class. But I don't know by how much. $\endgroup$
    – eloaf
    Commented Apr 4, 2022 at 22:10
  • 1
    $\begingroup$ I realize my question is very similar to stats.stackexchange.com/questions/137893/… $\endgroup$
    – eloaf
    Commented Apr 4, 2022 at 22:26
  • 1
    $\begingroup$ With a tremendous effort you could solve for the optimum sampling strategy that would allow you to estimate a model given the amount of $ you have to spend don labeling. This would involve a well-defined sampling scheme that would allow for recalibration of the intercept in the model. It's not for the faint of heart. $\endgroup$ Commented Apr 4, 2022 at 23:18
  • 1
    $\begingroup$ I don't say this lightly but this seems like a decent application to try semi-supervised learning. It is the "next step" after PU learning (which is related to your issue as you correctly recognised). Also, are you able to label at least "some" negative examples? Just to you can have an anchor from that side too? $\endgroup$
    – usεr11852
    Commented Apr 5, 2022 at 0:19

1 Answer 1


One way to make this work is to have a well-defined set of probabilities. At the simplest level, you have 'high-yield' observations with values of the regressors where you expect a positive result, and 'low-yield' observations where you don't. If you sample randomly from high-yield observations with probability $p_H$ (which could even be $p_H=1$) and randomly from low-yield observations with probability $p_L\ll p_H$, you'll get a sample that's enriched with positive outcomes. A weighted regression using $1/p_L$ as sampling weights for the low-yield observations and $1/p_H$ as sampling weights for the high-yield observations will reconstruct the true population relationships. You can have more complicated setups where there are more than two groups: all you need is that the probabilities really are the sampling probabilities you use, and that the probability is non-zero for every individual in the population. Oversampling higher-yield individuals is a standard design approach in survey sampling.

If your sampling depends only on regressors that are actually in the model, and your model is correctly specified, you don't even need the weights, because whether or not an observation is in the sample is independent of the outcome conditional on the regressors. That's quite a strong set of assumptions, though.

  • $\begingroup$ Thomas I seem to recall an older paper by Anderson who showed how to adjust the intercept without using weights in the analysis, as an alternative. Sorry I don't have the reference. $\endgroup$ Commented Apr 5, 2022 at 3:15
  • $\begingroup$ I actually attempted to do something like this with density estimation (kernel density estimation) of the regressors but did not seem to work in a toy experiment I created. But I will try your suggested approach it seems simpler and more principled. $\endgroup$
    – eloaf
    Commented Apr 5, 2022 at 15:14
  • $\begingroup$ Giving it more thought, the difference between what I tried (weighing my samples by an estimated p(x)) and what you are suggesting is that I did NOT sample the x's to be labeled according to 1/p(x). $\endgroup$
    – eloaf
    Commented Apr 6, 2022 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.