1
$\begingroup$

If I want to use an undersampling approach to construct the machine learning model, I am wondering if there are any criteria to determine how many times I should sample the data from the majority group (the minority is 14% and the majority is 86%) and build the ML model? I am working with biological data and I am recommended not to use oversampling approaches. To determine which sampling approach we should use, are there any criteria to determine before constructing the ML model? Is it really dependent on the field of study? 

$\endgroup$
4
  • 3
    $\begingroup$ “If I want to use an undersampling approach to construct the machine learning model”—you probably don’t. Stephen or Dave can explain why soon. If you want to read ahead, then look up “proper scoring rules”. $\endgroup$ Aug 19 at 14:43
  • 3
    $\begingroup$ Class imbalance is less of a problem than you might think. I also like Sycorax's comment about how we got here, which I think is correct. // Frank Harrell has good blog posts, too: 1 2. // There is an interesting example on here where proper scoring rules do not win, granted, under strict assumptions and a specified cost. $\endgroup$
    – Dave
    Aug 19 at 15:16
  • 1
    $\begingroup$ First you need to decide what performance criterion is important for your application, it could well be that there is no need for any over- or under-sampling. We also need an idea of how big the dataset is (more data resolves imbalance problems). Note that there doesn't seem to be a means of determining whether class imbalance is actually a problem for a specific application - it may be that assigning everything to the majority class is the optimal solution stats.stackexchange.com/questions/539638/… $\endgroup$ Aug 19 at 18:34
  • $\begingroup$ Whether proper scoring rules are appropriate depends on the needs of your application, but they are definitely something to investigate before considering up or (especially) down-sampling. You might want to use a classifier that can weight the importance of patterns, rather than resampling, it achieves the same thing, but without throwing away information . The SVM can do this, but so can neural nets and logistic regression etc. $\endgroup$ Aug 19 at 18:36
1
$\begingroup$

I am wondering if there are any criteria to determine how many times I should sample the data from the majority group (the minority is 14% and the majority is 86%) and build the ML model.

If you are interested in accuracy (or expected loss, which is just a sort of weighted accuracy), then I think the short answer is probably "no". Class imbalance problems tend to arise when you have very little data, and so due to the imbalance, there are especially few patterns describing the distribution of the minority class. This can lead to unduly biasing the decision surface towards the majority class. However, the bias is generally fairly small. If you do something as extreme as balancing the dataset so it has a 50:50 distribution, this is likely to way over-compensate, so that the true positive rate goes up, but the false negative rate goes up much faster and your accuracy goes down. So basically you want to resample/reweight just enough to compensate for the bias, but no more, in order to improve test accuracy. But if we have so little data that we have a class imbalance problem, we probably don't have enough data to estimate the required correction either (apart from simple methods like logistic regression), so in practice we may be better of doing nothing.

Fortunately, most parameter estimation problems go away if you collect enough new data, so that is often going to be the most practical solution.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks everyone for the explanation. $\endgroup$
    – tassaneel
    Aug 20 at 1:54
  • $\begingroup$ It is a good question, I'm working on finding a better answer. $\endgroup$ Aug 20 at 6:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.