1
$\begingroup$

I am building a classification model using Random Forest technique using GridSearchCV. The target variable is binary where 1 is 7.5% of total population. I have used several values of GridSearch Parameters but results are almost always same

precision recall f1-score support
0 0.91 1 0.95 738
1 0 0 0 75
avg/total 0.82 0.91 0.86 813

As per confusion matrix, none of the 1’s are being correctly identified. Accuracy is equal to 1.

0 1
0 738 0
1 75 0

Are there any general ‘hacks’ there for high precision and high recall scores? Does 7.5% mean the model/classes are imbalanced? Any suggestions?

$\endgroup$
3
  • $\begingroup$ The first thing to try is threshold analysis: the default probability cutoff for the class is 50%, which often leads to this problem for imbalanced datasets. Your probabilistic predictions might be quite good, so just lowering the decision threshold may be enough. $\endgroup$ Commented Oct 7, 2021 at 13:40
  • 1
    $\begingroup$ Oh, and since you're using sklearn's GridSearchCV, you should probably consider a different scoring method for comparing hyperparameters; accuracy (the default in sklearn for classification) is not a very good metric, especially for imbalanced classes. $\endgroup$ Commented Oct 7, 2021 at 13:43
  • $\begingroup$ Why not evaluate the predicted probabilities directly with proper scoring-rules like cross-entropy loss? When you do this, class imbalance becomes minimally problematic! $\endgroup$
    – Dave
    Commented Oct 7, 2021 at 14:53

2 Answers 2

1
$\begingroup$

You go into the modeling by telling the model to expect category zero. Category zero is, after all, considerably more prevalent.

You then present evidence about class membership by giving the model features that are supposed to help the model discern between the two categories. Since category zero is so much more common than category one, you should require fairly compelling evidence of category one to predict category one. If you show the model features that are fairly routine, not so different than most of the feature values for category zero, why would the model want to predict category one?

And so the model does not predict category one.

Getting into the details of the random forest, the classification is determined by seeing how the various trees in the forest vote. If most of the trees vote for category zero, the software-default decision rule is to classify as category zero.

But you told those trees to expect category zero much more often than category one. In the absence of compelling evidence to the contrary, these trees are likely to predict category zero.

So your features are inadequate to distinguish between the two categories, right? Maybe. The situation is more nuanced than that, however. The final classifications used to create your confusion matrix and your precision/recall/F-scores came from a rule about how to use the predictions made by the individual trees, a rule where the majority vote wins.

Why does the classification decision have to be that the majority vote wins? You are free to vary the decision rule. Maybe a threshold like $30\%$ or $10\%$ in favor of category one is more appropriate. That's what Ben Reiniger means in his comment, The first thing to try is threshold analysis: the default probability cutoff for the class is 50%, which often leads to this problem for imbalanced datasets. Your probabilistic predictions might be quite good, so just lowering the decision threshold may be enough.

As it stands, however, none of your instances of category one have the majority of the trees voting in favor of category one, and that is why you get zero such classifications.

One way to assess if your model has much of any ability to discern between the categories is to analyze the receiver-operator characteristic curve, which calculates the sensitivity and specificity over all possible thresholds. The area under this curve relates to the ability of the model to discern between the two categories yet is not tied to any particular threshold. It could be that you get a perfect AUC of $1$ despite the inability, at the $50\%$ voting threshold, to classify any instances as category one.

Finally, give consideration to if what you really want to do is classification or to estimate the probability of class membership. Many people who come to machine learning from areas other than pure statistics do not even know this is a possibility. Vanderbilt University's Frank Harrell, a Cross Validated member, has two good blog posts on the advantages of estimating class-membership probability over performing classification. The first of these two articles makes the case for doing so, and the second makes the case that considering measures of performance like precision, recall, F1, accuracy, and confusion matrices is harmful to the analysis.

Classification vs. Prediction

Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

Watch out for when "classification" models do no classification on their own and rely on a two-stage process, original model predictions > decision rule, to make those classifications. If you find your classifications to be inadequate, it might be that the original model is fine but the decision rule is a poor one.

$\endgroup$
2
  • $\begingroup$ +1 for the reference to the ROC or even better the PR curves. It makes perfect sense for a more or less calibrated classifier (obtained through minimizing cross entropy) that you need to tune the decision thresholds to meet some discretized performance metrics. There is no need to hurt calibration by up or down sampling $\endgroup$
    – Ggjj11
    Commented May 17 at 20:18
  • $\begingroup$ Look into this video by the imbalance learn author: youtu.be/Gjrz4YCp6l8?si=iO4rJkaglbkrC5lQ guillaume lemaitre, there is no problem with imbalanced data ( I would add, for binary classification. For classification with multiple classes you essentially have a multi objective optimization problem: "be good in recognizing every class" and this is not as easily resolved through threshold tuning as in the binary case) $\endgroup$
    – Ggjj11
    Commented May 17 at 20:27
-1
$\begingroup$

Given that category 1 only accounts for 7.5% of your sample - then yes, your sample is highly imbalanced.

Look at the recall score for category 1 - it is a score of 0. This means that of the entries for category 1 in your sample, the model does not identify any of these correctly.

The high f-score accuracy of 86% is misleading in this case. It means that your model does very well at identifying the category 0 entries - and why wouldn't it? They comprise most of your sample. However, the model fails at identifying any of the category 1 entries.

There are some solutions to this - although the ones you should choose will depend on the data you are working with.

  • SMOTE (Synthetic Minority Oversampling Technique): This is a technique whereby artificial samples are generated for the minor class (in this case, category 1) - by replicating the properties of that data as much as possible. In this case, the number of samples for 0 and 1 will be equal - which is more likely to result in a more balanced accuracy metric.

  • You can specify a balanced class weight to the model. The class_weight='balanced' essentially works by penalising prediction errors on the minor class more severely than on the major class. However, if there exists too little data for the minor class, then the effectiveness might still be limited. For instance, given a RandomForest model with 15 estimators:

RandomForestClassifier(n_estimators=15, class_weight='balanced')
  • Using scale_pos_weight within XGBoost. If you wished to use XGBoost as a classification model, then you could use scale_pos_weight to specify the ratio of the negative class to the positive class. This could be set to 12 in this scenario (given that 92.5% major instances / 7.5% minor instances yields a ratio of 12.33).
import xgboost as xgb
xgb_model = xgb.XGBClassifier(learning_rate=0.001,
                            max_depth = 1, 
                            n_estimators = 100,
                              scale_pos_weight=12)

In your situation, I would recommend that you try multiple techniques to attempt to balance the accuracy results - and examine which one is most effective.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.