cross validation method issues when evaluating biased data set

I think when cross validation calculate precision and recall (refer to my code above), it calculate average precision and recall of each class? Which means if prediction of one class has high precision/recall, the other class has relatively low precision/recall, the final results are still ok?

Suppose a two class classification problem. One class has more than 95% of labelled data, and the other class has 5% of labelled data. The two class are very biased.

I am doing class validation to evaluate different classifiers, I found if a classifier intentionally to predict to the class which has majority (95%) label, even if the prediction result on other class is not accurate, from precision/recall, it is hard to distinguish since the other class has only 5% labelled data.

Here are the methods/metrics (using precision/recall) I am using. I am wondering if any other better metrics or method to evaluate considering the minor 5% class? I am assign a weight to the minor 5% class, but I ask here for a more systematic method to measure biased data set.

If there are any solution in scikit-learn, it will be great.

BTW, do you think if average of precision/precision of two class is ok even if for biased data, since in model training, we treat two class as equal weight?

Using scikit learn + python 2.7.

scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='recall_weighted')
print("Recall: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))
scores = cross_validation.cross_val_score(bdt, X, Y, cv=10, scoring='precision_weighted')
print("Precision: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

precision    recall  f1-score   support
0       0.95      0.99      0.97       941
1       0.45      0.10      0.16        51
avg / total       0.93      0.95      0.93       992


Almost all practical two-class problems are unbalanced with predicting '1'or minority is always more important. e.g., fraud detection

To evaluate these models precision, recall) and F1 (https://en.wikipedia.org/wiki/F1_score) score are widely used. Recall & precision are not used in isolation but they are defined with one fixed. (please note that all of these are low in your output).

You can also use ROC & Kappa as well. ROC is useful in understanding the classifier as well as deciding the trade-off between accuracy & precision.

e.g., - For fraud detection you would want to tag all of the frauds correctly (minority class) even if it means few of the zero's are classified incorrectly. To achieve this you would basically change the probability threshold in the ROC curve. This will though reduce the accuracy

The bigger issue is to handle the imbalanced data set problem, which can be dealt in few different ways

• Thanks Nishad, vote up for your reply. (1) not sure why recall is helpful, since if almost all label are positive, and a naive predictor predict everything to be positive, recall is very high? (2) I think ROC is good why you say won't give too much for unbalanced case? – Lin Ma Sep 13 '16 at 7:41
• @LinMa - Added explanation for recall . Also edited for ROC, I absolutely agree that ROC is useful. The comment was in context that ROC will not give you more information than what you get from precision about a model performance (good/not useful). IMO It is more important for tuning. – Nishad Sep 14 '16 at 5:02
• Thanks Nishad, vote up for your reply and edit. For fraud detection case, suppose fraud is labeled as 0 (negative), and non-fraud is labeled as 1 (positive). Suppose a naive classifier classify everything into 1, I think the classifier will have both very high precision and recall, you mentioned we can combine precision and recall in fraud use case, but I am confused how do you combine them to make it more effective, as both precision and recall are very high number? – Lin Ma Sep 14 '16 at 6:15
• @LinMa - I think you are confused about what is positive class & negative class. Fraud detection as the name suggests positive means fraudulent, it is not dependent on how you code you variables (1/0 or yes/no) coding is only for programming purposes. Results need to be interpreted based on your function. Regarding precision & recall - Consider a crude example, you have 10 fraud cases out of 100. If the model marks 10 cases as fraud then recall is 100% but what if the marked one are all incorrect so precision is 0%. So combined together they will provide how the model is doing. – Nishad Sep 14 '16 at 6:57
• Micro & macro precision/recall is for multi-class problems. – Nishad Sep 14 '16 at 8:17