0
$\begingroup$

We are taking anonymised demographic data of n people with a rare medical condition, and trying to train a binary classifier (currently using xgboost) to reach more similar people with let's say a public service announcement.

To train we have been taking a random sample of n non-patients (with their demo data). The classifier helps us reach many people with what we believe is a propensity to have the disease.

Starting to have doubts if an artificially balanced sample like this is a good thing. Should there be more non-patients in the training sample since in real life there is MANY more non-patients and why?

Thanks!

Improve this question
$\endgroup$
5
  • $\begingroup$ you might find my answer here helpful stats.stackexchange.com/questions/488939/… $\endgroup$
    – Dikran Marsupial
    Commented Apr 1, 2022 at 15:29
  • $\begingroup$ Thanks! Going through it! $\endgroup$
    – Gamliel Beyderman
    Commented Apr 1, 2022 at 18:02
  • $\begingroup$ I am puzzled how to post-process as you suggest, will it affect the final result though as we just classify people into one of two classes and get a value of 1 or 0 for each person... $\endgroup$
    – Gamliel Beyderman
    Commented Apr 1, 2022 at 21:02
  • $\begingroup$ You wrote: "Fortunately this bias tends to go away rapidly as the size of the dataset increases, so if you have lots of data, you don't need to do anything." Not to do anyth = keep the sample balanced? How do I quantify how much data is enough to overcome this bias. Let's say is 40 thousand positive case and 40 thousand negative, a lot, a little? $\endgroup$
    – Gamliel Beyderman
    Commented Apr 1, 2022 at 21:06
  • 1
    $\begingroup$ That is probably a lot, but it also depends on the dimensionality of the data, in high dimensions you need more data to characterise the statistical distribution of the minority class. It is important to consider the misclassification costs, and factor those in. For problems like this it is probably best to go for a probabilistic classifier that estimates the probability of class membership rather than a discrete 0/1 classifier, as you can then cope with misclassification costs and differences in priors using the tricks in my other answer. $\endgroup$
    – Dikran Marsupial
    Commented Apr 1, 2022 at 21:30

1 Answer 1

Reset to default
0
$\begingroup$

This depends on the KPI that you want to achieve with your classifier. Depending on different costs involved in false positives and false negatives you want to optimize precision or recall.

Generally, I would recommend to use a very large amount of examples (much more negative than positive) and use class_weight parameter of your ML algorithm to optimize performance based on your necessities.

Improve this answer
$\endgroup$
3
  • $\begingroup$ We want to maximize recall. But how to decide on the "right" imbalance? $\endgroup$
    – Gamliel Beyderman
    Commented Apr 1, 2022 at 20:51
  • $\begingroup$ You want the test set as close to reality as possible. Than you could treat the training set imbalance as a hyperparameter and optimize it using model selection. $\endgroup$
    – user2672299
    Commented Apr 1, 2022 at 21:09
  • $\begingroup$ However, generally I would not recommend to optimize recall using under sampling. You would like to optimize recall by changing the error function of your ML algorithm. $\endgroup$
    – user2672299
    Commented Apr 1, 2022 at 21:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.