Facing unbalanced data: AUC vs. Cohen's Kappa vs. Balanced Misclassification Rate

Question

As the question title implies, I am dealing with unbalanced data (minority class 2%) classification. As a classification tool I chose Random Forest from R package "RandomForest".

So, I chose two ways to tackle the unbalance of my data. First, I tried oversampling minority class ("1"). Second, I tried to use the data as it is, but undersample the majority class ("0") in the sampsize argument, so it's something that I call "Pseudo-undersampling".

Found the answer: Cohen's kappa is dramatically affected by the prevalence and bias. So it's better to chose different metric.

"Pseudo-undersampling":

randomForest(x=train9[,-1],y=train9[,1],ntree=500,
                          mtry=mtry[i],replace=FALSE, strata= train9[,1],
                          sampsize = c(length(train9[,1][which(train9[,1]==1)]),
                                       length(train9[,1][which(train9[,1]==1)])),nodesize=2,
                          importance = FALSE, norm.votes = TRUE, keep.forest = TRUE)

Oversampling (object NO9 is train9 cases with class "0"):

randomForest(x=train9[,-1],y=train9[,1],ntree=500,
                                mtry=mtry[i],replace=T, strata= train9[,1],
                                sampsize = c(length(NO9[,1]),length(NO9[,1])),nodesize=2,
                                importance = FALSE, norm.votes = TRUE, keep.forest = TRUE)

I did repeated cross-validation on both models (trying different mtry argument for each model). # NOTE: I did oversampling not prior to, but within cross-validation loop, so the models were tested on "new" data, therefore, we can rely on results.

So the results are: "Pseudo-undersampling" had higher AUC by 0.01 (0.93 < 0.94, though not significantly) and lower Balanced misclassification rate (1-balanced accuracy) by 0.1 (0.24> 0.14, p<0.01). However, its Cohen's Kappa was lower by 0.36 (0.56>0.2, p<0.01).

How should I interpret these results and which model is better and more acceptable?

Generalising the question: on which metric should one rely more, when dealing with unbalanced data?

Answer 1

There are so many problems with your approach that is difficult to know where to begin. As an aside the concordance index $c$ (AUROC) is not sensitive to the distribution of $Y$. But you have not used any proper accuracy scoring rules, and have cast the problem as a classification problem even though you may be more interested in tendencies than in forced choices. Any method that requires you to remove samples or over-sample is highly suspect and not based on good statistical principles. For more information see http://www.fharrell.com/2017/01/classification-vs-prediction.html and http://www.fharrell.com/2017/03/damage-caused-by-classification.html .

Answer 2

Do you know the concrete costs of misclassification in both directions? I assume that missing the rare class must be much more costly than falsely adding a record to the rare class. (If not, then your problem is per definition trivial, ignore the rare class!) Only you can know the answer to that question based on your application scenario.

Put those misclassification costs, and perhaps also profits due to correct classification, in a loss function and compare performance on this function.

You also want to use the label confidences that a random forest classifier provides. You can adjust the classification cutoff according to your misclassification costs. This allows you to maximize expected profits/ minimize expected costs. In order to avoid manual overfitting by reverse engineering your classifier to the test-set and to do it in a principled way, you should set that cutoff based on classification costs alone, thus before training the classifier.