I used many different algorithms on a data set for binary classification. I used kNN, SVM radial, ANN, random forest, Gaussian process classification, etc. Every algorithm is well tuned in R using caret package.
One interesting thing is that pretty much all algorithms have a similar level of test accuracy (around 80-83%). Moreover, every algorithm is pretty good at predicting the negative class (90-93% accuracy) but they are all pretty poor in predicting positive class with an average accuracy of 66% and a maximum accuracy of 70%.
The dataset is not imbalanced and there is nothing particularly unusual about the dataset.
I have been thinking about the reasons behind this for quite a while since I would really like to improve the accuracy of the positive class but I have not been able to reach any conclusions. The only reason I could think of is that the negative class consists of various different classes which are not positive class and they were just assigned the label "negative". But I am not sure if that is the reason and how it is related to the poor accuracy of positive class.
I also did a cross-matching for the 4 best algorithms: kNN, ANN, SVM, RF. What I found was that there were 21 instances got misclassified simultaneously by all 4 algorithms (the test set has 287 instances) and 20 of them have true labels "positive" (they got predicted as negative). I really don't understand why is this happening.
I am wondering if anybody encountered any similar situations? Thanks a lot!