0
$\begingroup$

I am building a classification model using Random Forest technique using GridSearchCV. The target variable is binary where 1 is 7.5% of total population. I have used several values of GridSearch Parameters but results are almost always same

precision recall f1-score support
0 0.91 1 0.95 738
1 0 0 0 75
avg/total 0.82 0.91 0.86 813

As per confusion matrix, none of the 1’s are being correctly identified. Accuracy is equal to 1.

0 1
0 738 0
1 75 0

Are there any general ‘hacks’ there for high precision and high recall scores? Does 7.5% mean the model/classes are imbalanced? Any suggestions?

$\endgroup$
3
  • $\begingroup$ The first thing to try is threshold analysis: the default probability cutoff for the class is 50%, which often leads to this problem for imbalanced datasets. Your probabilistic predictions might be quite good, so just lowering the decision threshold may be enough. $\endgroup$ Oct 7, 2021 at 13:40
  • $\begingroup$ Oh, and since you're using sklearn's GridSearchCV, you should probably consider a different scoring method for comparing hyperparameters; accuracy (the default in sklearn for classification) is not a very good metric, especially for imbalanced classes. $\endgroup$ Oct 7, 2021 at 13:43
  • $\begingroup$ Why not evaluate the predicted probabilities directly with proper scoring-rules like cross-entropy loss? When you do this, class imbalance becomes minimally problematic! $\endgroup$
    – Dave
    Oct 7, 2021 at 14:53

1 Answer 1

0
$\begingroup$

Given that category 1 only accounts for 7.5% of your sample - then yes, your sample is highly imbalanced.

Look at the recall score for category 1 - it is a score of 0. This means that of the entries for category 1 in your sample, the model does not identify any of these correctly.

The high f-score accuracy of 86% is misleading in this case. It means that your model does very well at identifying the category 0 entries - and why wouldn't it? They comprise most of your sample. However, the model fails at identifying any of the category 1 entries.

There are some solutions to this - although the ones you should choose will depend on the data you are working with.

  • SMOTE (Synthetic Minority Oversampling Technique): This is a technique whereby artificial samples are generated for the minor class (in this case, category 1) - by replicating the properties of that data as much as possible. In this case, the number of samples for 0 and 1 will be equal - which is more likely to result in a more balanced accuracy metric.

  • You can specify a balanced class weight to the model. The class_weight='balanced' essentially works by penalising prediction errors on the minor class more severely than on the major class. However, if there exists too little data for the minor class, then the effectiveness might still be limited. For instance, given a RandomForest model with 15 estimators:

RandomForestClassifier(n_estimators=15, class_weight='balanced')
  • Using scale_pos_weight within XGBoost. If you wished to use XGBoost as a classification model, then you could use scale_pos_weight to specify the ratio of the negative class to the positive class. This could be set to 12 in this scenario (given that 92.5% major instances / 7.5% minor instances yields a ratio of 12.33).
import xgboost as xgb
xgb_model = xgb.XGBClassifier(learning_rate=0.001,
                            max_depth = 1, 
                            n_estimators = 100,
                              scale_pos_weight=12)

In your situation, I would recommend that you try multiple techniques to attempt to balance the accuracy results - and examine which one is most effective.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.