High Precision and High Recall issue- Random Forest Classification

Question

I am building a classification model using Random Forest technique using GridSearchCV. The target variable is binary where 1 is 7.5% of total population. I have used several values of GridSearch Parameters but results are almost always same

		precision	recall	f1-score	support
	0	0.91	1	0.95	738
	1	0	0	0	75
avg/total		0.82	0.91	0.86	813

As per confusion matrix, none of the 1’s are being correctly identified. Accuracy is equal to 1.

	0	1
0	738	0
1	75	0

Are there any general ‘hacks’ there for high precision and high recall scores? Does 7.5% mean the model/classes are imbalanced? Any suggestions?

Answer 1

Given that category 1 only accounts for 7.5% of your sample - then yes, your sample is highly imbalanced.

Look at the recall score for category 1 - it is a score of 0. This means that of the entries for category 1 in your sample, the model does not identify any of these correctly.

The high f-score accuracy of 86% is misleading in this case. It means that your model does very well at identifying the category 0 entries - and why wouldn't it? They comprise most of your sample. However, the model fails at identifying any of the category 1 entries.

There are some solutions to this - although the ones you should choose will depend on the data you are working with.

• SMOTE (Synthetic Minority Oversampling Technique): This is a technique whereby artificial samples are generated for the minor class (in this case, category 1) - by replicating the properties of that data as much as possible. In this case, the number of samples for 0 and 1 will be equal - which is more likely to result in a more balanced accuracy metric.

• You can specify a balanced class weight to the model. The class_weight='balanced' essentially works by penalising prediction errors on the minor class more severely than on the major class. However, if there exists too little data for the minor class, then the effectiveness might still be limited. For instance, given a RandomForest model with 15 estimators:

RandomForestClassifier(n_estimators=15, class_weight='balanced')

• Using scale_pos_weight within XGBoost. If you wished to use XGBoost as a classification model, then you could use scale_pos_weight to specify the ratio of the negative class to the positive class. This could be set to 12 in this scenario (given that 92.5% major instances / 7.5% minor instances yields a ratio of 12.33).

import xgboost as xgb
xgb_model = xgb.XGBClassifier(learning_rate=0.001,
                            max_depth = 1, 
                            n_estimators = 100,
                              scale_pos_weight=12)

In your situation, I would recommend that you try multiple techniques to attempt to balance the accuracy results - and examine which one is most effective.