Performance Evaluation for multiple classifiers in machine learning with Wilcoxon Rank Sum Test

Question

I use four feature descriptors for five independent runs with each run randomly selecting the training set and testing set from the total labeled samples. SVM is the classifier. After acquiring the classification accuracy of each run for each descriptor, I get a accuracy matrix whose size is $5 \times 4$. Then I use the ranksum function in MATLAB to do Wilcoxon run sum test from pair-wise. Then I get the p-value matrix shown as follows:

I select $\alpha = 0.05$. If the p-value is less than $0.05$, then the difference is significant, and if the p-value is larger than $0.05$, then the difference between the two features is not significant. Is it right? Is the total procedure of statistical tests right? Thank you.

Answer 1

You are right about significance and P.values. I am not sure Wilcoxon's Rank Sum test is the best way to compare accuracies, however, and hopefully someone will shed more light on this. I'd be considering Fisher's exact test for each pairwise comparison (Correct vs Incorrect calls).