Return to Question

Found the answer: Cohen's kappa is dramatically affected by the prevalence and bias. So it's better to choose different metric.

References: Agree or Disagree? A Demonstration of An Alternative Statistic to Cohen’s Kappa for Measuring the Extent and Reliability of Agreement between Observers

Kappa Statistic is not Satisfactory for Assessing the Extent of Agreement Between Raters

High agreement but low kappa: II. Resolving the paradoxes.

"Pseudo-undersampling":

Generalising the question: on which metric should one rely more, when dealing with unbalanced data?

Found the answer: Cohen's kappa is dramatically affected by the prevalence and bias. So it's better to chose different metric.

"Pseudo-undersampling":

Generalising the question: on which metric should one rely more, when dealing with unbalanced data?

Found the answer: Cohen's kappa is dramatically affected by the prevalence and bias. So it's better to choose different metric.

References: Agree or Disagree? A Demonstration of An Alternative Statistic to Cohen’s Kappa for Measuring the Extent and Reliability of Agreement between Observers

Kappa Statistic is not Satisfactory for Assessing the Extent of Agreement Between Raters

High agreement but low kappa: II. Resolving the paradoxes.

As the question title implies, I am dealing with unbalanced data (minority class 2%) classification. As a classification tool I chose Random Forest from R package "RandomForest".

So, I chose two ways to tackle the unbalance of my data. First, I tried oversampling minority class ("1"). Second, I tried to use the data as it is, but undersample the majority class ("0") in the sampsize argument, so it's something that I call "Pseudo-undersampling".

"Pseudo-undersampling":

randomForest(x=train9[,-1],y=train9[,1],ntree=500,
                          mtry=mtry[i],replace=FALSE, strata= train9[,1],
                          sampsize = c(length(train9[,1][which(train9[,1]==1)]),
                                       length(train9[,1][which(train9[,1]==1)])),nodesize=2,
                          importance = FALSE, norm.votes = TRUE, keep.forest = TRUE)

Oversampling (object NO is cases with class "0"):

randomForest(x=train9[,-1],y=train9[,1],ntree=500,
                                mtry=mtry[i],replace=T, strata= train9[,1],
                                sampsize = c(length(NO9[,1]),length(NO9[,1])),nodesize=2,
                                importance = FALSE, norm.votes = TRUE, keep.forest = TRUE)

I did repeated cross-validation on both models (trying different mtry argument for each model). # NOTE: I did oversampling not prior to, but within cross-validation loop, so the models were tested on "new" data, therefore, we can rely on the results.

So the results are: "Pseudo-undersampling" had higher AUC by 0.01 (0.93 < 0.94, though not significantly) and lower Balanced misclassification rate (1-balanced accuracy) by 0.1 (0.24> 0.14, p<0.01). However, its Cohen's Kappa was lower by 0.36 (0.56>0.2, p<0.01).

How should I interpret these results and which model is better and more acceptable?

Generalising the question: on which metric should one rely more, when dealing with unbalanced data?

As the question title implies, I am dealing with unbalanced data (minority class 2%) classification. As a classification tool I chose Random Forest from R package "RandomForest".

So, I chose two ways to tackle the unbalance of my data. First, I tried oversampling minority class ("1"). Second, I tried to use the data as it is, but undersample the majority class ("0") in the sampsize argument, so it's something that I call "Pseudo-undersampling".

"Pseudo-undersampling":

randomForest(x=train9[,-1],y=train9[,1],ntree=500,
                          mtry=mtry[i],replace=FALSE, strata= train9[,1],
                          sampsize = c(length(train9[,1][which(train9[,1]==1)]),
                                       length(train9[,1][which(train9[,1]==1)])),nodesize=2,
                          importance = FALSE, norm.votes = TRUE, keep.forest = TRUE)

Oversampling (object NO9 is train9 cases with class "0"):

randomForest(x=train9[,-1],y=train9[,1],ntree=500,
                                mtry=mtry[i],replace=T, strata= train9[,1],
                                sampsize = c(length(NO9[,1]),length(NO9[,1])),nodesize=2,
                                importance = FALSE, norm.votes = TRUE, keep.forest = TRUE)

I did repeated cross-validation on both models (trying different mtry argument for each model). # NOTE: I did oversampling not prior to, but within cross-validation loop, so the models were tested on "new" data, therefore, we can rely on results.

So the results are: "Pseudo-undersampling" had higher AUC by 0.01 (0.93 < 0.94, though not significantly) and lower Balanced misclassification rate (1-balanced accuracy) by 0.1 (0.24> 0.14, p<0.01). However, its Cohen's Kappa was lower by 0.36 (0.56>0.2, p<0.01).

How should I interpret these results and which model is better and more acceptable?

Generalising the question: on which metric should one rely more, when dealing with unbalanced data?