I'm having a classifier that tries to classify 5 different classes from a data set. It works pretty well in general, and when I'm plotting the confusion matrix almost all misses are 0, 1 or 2 max 3 except for the misses between 2 specific classes, which are >15. I'm a beginner with machine learning and I'd like to know if there's something I can do with this info. Of course I can tweak parameters, try other classifier etc, but I'm trying to remove chance or brute force from my method and would like to follow the logical steps.
So, what are some things to look after in a classifying situation in which the predictions are accurate between all classes, but 2 (the classifier mistakes one for the other often)