6
$\begingroup$

I have a binary classification problem where my data is highly imbalanced (80:20). Given this imbalance is present in both training and test set, does it make sense to apply specific strategies during training to handle the imbalance problem?

$\endgroup$

4 Answers 4

6
$\begingroup$

A few general strategies:

  1. First and foremost, in imbalanced classification problems you want to do stratified cross-validation. This allows you to train your models with the same distribution in your samples.
  2. Second, you should probably use Cohen's Kappa metric when tuning your models. It is better in imbalanced scenarios because it takes into account random chance as well. A more detailed description was provided in the answer to this question
  3. If you are adventurous, you can look into cost-sensitive machine learning. In these methods you essentially tell the algorithm that it is better to positively identify certain classes. For example, it would be much worse to misidentify a person with cancer as opposed to accurately identifying them. There many methods including sampling (over, under, SMOTE, SMOTEBoost and EasyEnsemble as referenced in this prior question regarding imbalanced datasets and CSL), Weighting, Thresholding, and Ensemble Methods. These are mostly algorithm agnostic methods, there are also algorithms with CSL builtin but I think this is enough to get your started.
$\endgroup$
2
$\begingroup$

if you use SVM , you can assign param class_weight=balanced => weights will be taken into consideration by classifier when training (look sklearn docs)

if you want to make threshold different than 0.5. => you can move threshold - THE WAY: if you think false positives are worse than false negatives - you can take this into consideration in costs in the code OR on the stage of decision-making [NB use predict_proba for newly predicted sample & compare it with threshold you decided - e.g. estimator.predict_proba() < 0.3 or < 0.7 instead of estimator.predict().]

P.S. but also take a look here Are unbalanced datasets problematic AND here - Is threshold moving unnecessary in balanced classification problem

$\endgroup$
1
$\begingroup$

I have faced the same problem trying to predict a single emotion in the RAVDESS dataset. The thing that helped me is: to provide the model with the initial bias and weights; in this way, the model takes care of the class differences through data.

You can setup good initialization bias as follows $$ b_0 = \log_e\left(\text{# negative labels}\right)\\ b_1 = \log_e\left(\text{# positive labels}\right) $$

and good initialization weights for the output layer as follows $$ w_0 = \frac{1}{2} \cdot \frac{\text{# total samples}}{\text{# negative labels}}\\[15pt] w_1 = \frac{1}{2} \cdot \frac{\text{# total samples}}{\text{# positive labels}} $$ where $w_0$ is the weight for the negative class and $w_1$ for the positive one.

The meaning is that a better bias initialization helps the initial convergence, instead, a good weight initialization helps because you don't have very many of those positive (negative) samples to work with, so you would want to have the classifier heavily weight the few available examples.

You can plug the output_bias values inside the model as follows: b0 = np.log(neg) b1 = np.log(pos) output_bias = tf.keras.initializers.Constant([b0, b1]) ... model.add(tf.keras.layers.Flatten()) model.add(tf.keras.layers.Dense(2, activation="softmax", bias_initializer=output_bias))

and the initial class_weights in this way:

...
weight_for_negative = (1 / neg) * (total / 2.0)
weight_for_positive = (1 / pos) * (total / 2.0)
class_weight = {0: weight_for_negative, 1: weight_for_positive}
...
model_history = model.fit(x_traincnn,
                        y_train,
                        batch_size = 128,
                        epochs = 800,
                        validation_data = (x_validcnn, y_valid),
                        #callbacks = [mcp_save, lr_reduce, early_stopping, backup])
                        callbacks = [mcp_save, lr_reduce, early_stopping, tensorboard],
                        class_weight = class_weight)

Hope this will help.

Tips: I recommend check also fscore, precision, and recall metrics to better interpret the model.

Bibliography: TensorFlow Blog - Imbalanced data classification

$\endgroup$
0
$\begingroup$

I'd like to answer the actual question ask : "does it make sense to apply specific strategies ?"

80:20 can be interpreted as not so imbalanced data. It depends on the accuracy of your model.

  • If you final model had an accuracy of 99.9%, then please, do not do anything to balance it, your model already find everything.
  • If you have an accuracy of 80%, be careful. Maybe your model is classifying everything in the big group.

I would say, it really depends on the output. I always try a first really basic model first. Then I look at the results : what is good, what is not. If it's enough for you, keep it this way, if it's not, these results will be your benchmark you want to overtake.
Once you have this benchmark, you can try different algorithms which can help on the balance

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.