What is the best criterion for performance evaluation in classification algorithms?

Question

Here are some results for ANN and KNN on abalone data set using Weka:

Result for ANN 
Correctly Classified Instances  3183 76.203 % 
Incorrectly Classified Instances 994 23.797 % 
Mean absolute error      0.214 
Root mean squared error  0.3349 
Relative absolute error 58.6486 %

Result for KNN 
Correctly Classified Instances  3211 76.8734 % 
Incorrectly Classified Instances 966 23.1266 % 
Mean absolute error      0.2142 
Root mean squared error  0.3361 
Relative absolute error 58.7113 %

KNN has high accuracy but ANN has low errors. So which of the two algorithms should I say is better?

Which is the more preferable criterion, accuracy or error?

What I understood was that error should decrease with high accuracy, but the results here are opposite. Why is this so?

Answer 1

The methods appear to be very similar in performance and that's perhaps the main story.

But I don't think we can say much to help you decide.

Which is more important practically, making the right classification or reducing error? It can easily happen with mean error measures that even one odd observation pushes them up a fair bit.

I don't think that scrutinizing the figures of merit is going to be most informative here. Rather, focus on which observations were classified differently and think about what that implies. I don't know your software to advise how to do this. If it's not easy, you need better software.

Also, always use visualizations to see the classification in some space.

Answer 2

In this case, KNN is better than ANN. You should look at the accuracies, i.e., "Correctly Classified Instances".

Your logic about error and accuracy is correct. But here "relative absolute error" is probably different than what you imagined. See: http://list.waikato.ac.nz/pipermail/wekalist/2004-August/029217.html