1
$\begingroup$

I use four feature descriptors for five independent runs with each run randomly selecting the training set and testing set from the total labeled samples. SVM is the classifier. After acquiring the classification accuracy of each run for each descriptor, I get a accuracy matrix whose size is $5 \times 4$. Then I use the ranksum function in MATLAB to do Wilcoxon run sum test from pair-wise. Then I get the p-value matrix shown as follows:

enter image description here

I select $\alpha = 0.05$. If the p-value is less than $0.05$, then the difference is significant, and if the p-value is larger than $0.05$, then the difference between the two features is not significant. Is it right? Is the total procedure of statistical tests right? Thank you.

|cite|improve this question
$\endgroup$
2
$\begingroup$

You are right about significance and P.values. I am not sure Wilcoxon's Rank Sum test is the best way to compare accuracies, however, and hopefully someone will shed more light on this. I'd be considering Fisher's exact test for each pairwise comparison (Correct vs Incorrect calls).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thank you, sir. This paper web.mit.edu/rudin/www/BDNVWorking14.pdf is using the Wilcoxon's Rank Sum test for pair-wise performance evaluation. And I have read some papers on Wilcoxon's Ranks Sum test and Wilcoxon's Signed Rank test. It seems both are valid for paired tests, and the Wilcoxon's Rank Sum test seems to be the paired t-test but with arbitrary data distribution instead of normal distribution. Thanks again! $\endgroup$ – mining May 10 '14 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.