2
$\begingroup$

I am comparing four algorithms for ranking problem. My current experiment setup is as follows

1) I split the data into train, validation and test

2) I run the four algorithms and get the performance metric values (nDCG, MAP, Precision etc, a total of 5 metrics) for each algorithm on the test data and store it.

3) I carry out the above procedure (1 & 2) 10 times, and have the result for the four algorithms, for each algorithm a table of 50 values (1o iterations and 5 performance metrics).

Now, I would like to do the statistical significance of my algorithm against the other three algorithms. What is the best test for this, and how to do it ?, and if there is a readymade procedure in matlab, R, python or octave, can you please point to me ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Friedman test would be appropriate here. The null hypothesis of this test is that there is no significant difference between those algorithms. If the null hypothesis is rejected, you will need to perform a post-hoc test like Nemenyi test to check if the difference in performance of those algorithms are statistically significant or not.

In R, you can perform Friedman test using friedman.test. For post-hoc test, see PMCMR package.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for your suggestion. To be precise, in my case, i need to create a matrix for each of the performance metrics, with columns as the metric value for each algorithm and the rows correspond to the number of iterations. right ?. One more thing, can I run this test to compare classification algorithm also ? $\endgroup$
    – Shew
    Commented Aug 27, 2016 at 18:47
  • $\begingroup$ Yes, you are right. You can use this test to compare classification algorithm too. $\endgroup$
    – user1219801
    Commented Aug 27, 2016 at 19:40
  • 1
    $\begingroup$ You can refer to Statistical Comparisons of Classifiers over Multiple Data Sets by Janez Demsar for details (if you haven't already). $\endgroup$
    – user1219801
    Commented Aug 27, 2016 at 19:44
  • $\begingroup$ Sorry, I am a bit confused. According to the documentation, Friedman test on a matrix gives one p-value. How do I correlate it with corresponding algorithm ?. A single p-value (let us say is less 0.05) simply means that all the algorithms are significant. How do I say my algorithm is significantly better than other algorithms ? $\endgroup$
    – Shew
    Commented Aug 27, 2016 at 19:56
  • $\begingroup$ If p-value from Friedman test is less than alpha (let's say 0.05), then we reject the null hypothesis. That means there is a difference between those algorithms. But Friedman test cannot say which of your algorithms are better. So, we perform a post-hoc test, e.g. Nemenyi test. It performs pairwise comparison of average ranks of the algorithms. If the difference between average rank of two algorithms is greater than the critical value, then it means that the performance of these two algorithms are significantly different from each other. $\endgroup$
    – user1219801
    Commented Aug 27, 2016 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.