0
$\begingroup$

I have a Training Set of respiratory disease sounds, so there are 2 classes:

  • 0 for respiratory sounds of healthy patients.

  • 1 for breathing sounds of patients with a disease.

The Training Set is heavily unbalanced, there are many more examples of class 1 than of class 0. So my network architecture has problems learning on it. So I decided to try two strategies:

  • class_weight: which is present in Keras, going to weight class 0 more than class 1 in the cost function.

  • UnderSampling

Question:

How do I choose between these two strategies? is it correct to run the Model Training on the Training Set twice applying the previous two strategies and choose the strategy that performs best on the Test Set? I think this is correct because I am not fine-tuning the hyper-parameters. Or is it wrong and I should use a Validation Set?

$\endgroup$
1
  • $\begingroup$ Make sure that you are dealing with a classification problem and not an anomaly detection problem. Methods of tackling these problems are very different. See the question One class classifier vs binary classifier for more details. $\endgroup$
    – mhdadk
    Commented Jul 24, 2021 at 12:19

2 Answers 2

2
$\begingroup$

Use a probability model, e.g., binary logistic regression model. This automatically handles even extreme "class" imbalance (outcome imbalance). The goal of most analyses is not forced choice classification but is rather the estimation of tendencies, i.e., probabilities.

Any method that requires you to discard valid data is bogus. Stay away from sampling.

$\endgroup$
3
  • $\begingroup$ Note that deep learning models output probability values. People have claimed that deep learning is overconfident, but perhaps that comes from trying to optimize improper scoring rules during model tuning and selection steps. $\endgroup$
    – Dave
    Commented Jul 24, 2021 at 12:28
  • $\begingroup$ Yes, in fact, my model outputs probabilities. This does not answer my question. $\endgroup$ Commented Jul 24, 2021 at 12:53
  • 1
    $\begingroup$ @FrancescoLadogana What do you mean that it does not answer your question? When you estimate the probability of class membership, you can decide what to do with that probability. In particular, if there is a strong class imbalance, the predicted probability should favor the dominant class unless the features give highly compelling evidence otherwise. $\endgroup$
    – Dave
    Commented Aug 9, 2021 at 20:36
0
$\begingroup$

To add a little to @Frank_Harrell's answer,

The class imbalance problem is often not because of imbalance per se, but because you have to few examples of the minority class to properly characterise it's distribution. Adding more data may resolve the imbalance problem if that is possible. Unless you have a very large dataset, undersampling is likely to make things worse by having too few examples to properly characterise the majority class as well. Given a choice between the two, I would choose differential weighting of examples of the two classes.

In most practical applications, where you do have to make a forced choice, especially things like medical screening tests, the false positive and false negative misclassification costs are differen. You may find that weighting the examples according to their misclassification costs is sufficient to deal with the problem (i.e. minimum risk classification). Accounting for misclassification costs is a good thing to do anyway, but if you have a probabilistic classifier that does not have an estimation issue in unbalanced settings, then as @Frank-Harrell suggests, it is better to apply them to well-calibrated probabilities from your classifier.

If you are going to weight the patterns, I suggest selecting the weights to exactly balance the positive and negative classes to avoid any bias in the estimation of the model parameters caused by the imbalance, but then scale the output probabilities from the model to reflect the operational class frequencies (because otherwise the model is likely to wildly over-predict the minority class in operational use).

I would also investigate the data with something simple, like regularised logistic regression before looking at deep learning (if you haven't already). There are far fewer pitfalls and DL may not necessarily work better (and could be a lot worse if you fall into one of the many pitfalls).

I would strongly recommend never use machine learning models with their default parameter settings.

$\endgroup$
4
  • $\begingroup$ Of course, the main problem is that there are too few examples for the minority class. But if my model has enough capacity (thanks to Transfer Learning), it can also generalize about an Undersampling of the Training Set. However this does not answer my question. I want to know if it is correct to determine the best strategy for dealing with unbalanced classes "by observing how the model behaves on the Test Set and not on the Validation Set. $\endgroup$ Commented Jul 24, 2021 at 12:58
  • $\begingroup$ My point was not to use undersampling anyway, it is very unlikely to be a good solution. Whether you need a validation set depends on whether you need an unbiased performance estimate. If you make any choice about your final model based on the test set, it will potentially give a biased performance estimate. Having lots of capacity and little data is a recipe for bad generalisation, which is why I would recommend against undersampling. High capacity is not necessarily a good thing. But if you do, use a proper scoring rule. $\endgroup$ Commented Jul 24, 2021 at 13:03
  • 1
    $\begingroup$ @dikran-marsupial don't weight samples according to utilities. Use a representative unbalanced sample to derive a well-calibrated probability. Then apply utilities to predicted probabilities to arrive at optimal decisions. $\endgroup$ Commented Jul 24, 2021 at 13:18
  • $\begingroup$ At the moment, I'd only recommend weighting samples if you have a model that has a problem with imbalanced problems (some of them do). If the model doesn't have an estimation problem (and outputs probabilities) then I agree that applying the utilities to the output of the classifier is better. I'll update my answer to be more careful about that in paragraph 2. $\endgroup$ Commented Jul 24, 2021 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.