I used many different algorithms on a data set for binary classification. I used kNN, SVM radial, ANN, random forest, Gaussian process classification, etc. Every algorithm is well tuned in R using caret package.

One interesting thing is that pretty much all algorithms have a similar level of test accuracy (around 80-83%). Moreover, every algorithm is pretty good at predicting the negative class (90-93% accuracy) but they are all pretty poor in predicting positive class with an average accuracy of 66% and a maximum accuracy of 70%.

The dataset is not imbalanced and there is nothing particularly unusual about the dataset.

I have been thinking about the reasons behind this for quite a while since I would really like to improve the accuracy of the positive class but I have not been able to reach any conclusions. The only reason I could think of is that the negative class consists of various different classes which are not positive class and they were just assigned the label "negative". But I am not sure if that is the reason and how it is related to the poor accuracy of positive class.

I also did a cross-matching for the 4 best algorithms: kNN, ANN, SVM, RF. What I found was that there were 21 instances got misclassified simultaneously by all 4 algorithms (the test set has 287 instances) and 20 of them have true labels "positive" (they got predicted as negative). I really don't understand why is this happening.

I am wondering if anybody encountered any similar situations? Thanks a lot!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.