# Facing unbalanced data: AUC vs. Cohen's Kappa vs. Balanced Misclassification Rate

As the question title implies, I am dealing with unbalanced data (minority class 2%) classification. As a classification tool I chose Random Forest from R package "RandomForest".

So, I chose two ways to tackle the unbalance of my data. First, I tried oversampling minority class ("1"). Second, I tried to use the data as it is, but undersample the majority class ("0") in the sampsize argument, so it's something that I call "Pseudo-undersampling".

Found the answer: Cohen's kappa is dramatically affected by the prevalence and bias. So it's better to chose different metric.

"Pseudo-undersampling":

randomForest(x=train9[,-1],y=train9[,1],ntree=500,
mtry=mtry[i],replace=FALSE, strata= train9[,1],
sampsize = c(length(train9[,1][which(train9[,1]==1)]),
length(train9[,1][which(train9[,1]==1)])),nodesize=2,
importance = FALSE, norm.votes = TRUE, keep.forest = TRUE)


Oversampling (object NO9 is train9 cases with class "0"):

randomForest(x=train9[,-1],y=train9[,1],ntree=500,
mtry=mtry[i],replace=T, strata= train9[,1],
sampsize = c(length(NO9[,1]),length(NO9[,1])),nodesize=2,
importance = FALSE, norm.votes = TRUE, keep.forest = TRUE)


I did repeated cross-validation on both models (trying different mtry argument for each model). # NOTE: I did oversampling not prior to, but within cross-validation loop, so the models were tested on "new" data, therefore, we can rely on results.

So the results are: "Pseudo-undersampling" had higher AUC by 0.01 (0.93 < 0.94, though not significantly) and lower Balanced misclassification rate (1-balanced accuracy) by 0.1 (0.24> 0.14, p<0.01). However, its Cohen's Kappa was lower by 0.36 (0.56>0.2, p<0.01).

How should I interpret these results and which model is better and more acceptable?

Generalising the question: on which metric should one rely more, when dealing with unbalanced data?

There are so many problems with your approach that is difficult to know where to begin. As an aside the concordance index $c$ (AUROC) is not sensitive to the distribution of $Y$. But you have not used any proper accuracy scoring rules, and have cast the problem as a classification problem even though you may be more interested in tendencies than in forced choices. Any method that requires you to remove samples or over-sample is highly suspect and not based on good statistical principles. For more information see http://www.fharrell.com/2017/01/classification-vs-prediction.html and http://www.fharrell.com/2017/03/damage-caused-by-classification.html .

• That problem is unrelated to the current discussion. – Frank Harrell Sep 9 '17 at 17:27

Do you know the concrete costs of misclassification in both directions? I assume that missing the rare class must be much more costly than falsely adding a record to the rare class. (If not, then your problem is per definition trivial, ignore the rare class!) Only you can know the answer to that question based on your application scenario.

Put those misclassification costs, and perhaps also profits due to correct classification, in a loss function and compare performance on this function.

You also want to use the label confidences that a random forest classifier provides. You can adjust the classification cutoff according to your misclassification costs. This allows you to maximize expected profits/ minimize expected costs. In order to avoid manual overfitting by reverse engineering your classifier to the test-set and to do it in a principled way, you should set that cutoff based on classification costs alone, thus before training the classifier.

• That particular approach is too disconnected from the Bayes optimum decision rule, which is based on maximizing expected utility and needs to depend on accurate probability estimates. – Frank Harrell Aug 29 '17 at 13:34
• Thats actually not true @user7019377. Boosting, Random Forests, Neural Networks, in their modern and most successful forms, are based on solid statistical and probabilistic foundations. I believe Frank is right, that optimal decision making follows from estimating probabilities, and then applying decision rule based on understanding costs. This also has good engineering consequences: software engineers have long spoke in hallowed terms of "separation of concerns", and this is a great example of putting that principle into practice. – Matthew Drury Aug 29 '17 at 13:56
• This was just about your comment "Most machine learning techniques violate assumptions of the underlying statistical concepts". Everything else you say I agree with. – Matthew Drury Aug 29 '17 at 14:23
• Except that Matthew agrees with me that my aproach does this. It can always be argued if random forests provide the best probability estimates or if we should go with logistic regression or something else instead. This choice of the algorithm to provide those estimates is an orthogonal question to the issue whether or not there should be a forced choice classification and if yes whether the cutoff should be based on relative misclassification costs or determined by a lift curve and a fixed budget as in the article you linked (which obviously is still forced choice classification) or otherwise. – David Ernst Aug 29 '17 at 16:24
• I think you guys are talking past each other a bit. We all seem to agree that we should estimate probabilities and then base decisions on those probabilities along costs of correct and incorrect classification. Of course, some situations do not call for hard class assignment at all, i.e. insurance pricing. – Matthew Drury Aug 29 '17 at 17:33