I am comparing four algorithms for ranking problem. My current experiment setup is as follows
1) I split the data into train, validation and test
2) I run the four algorithms and get the performance metric values (nDCG, MAP, Precision etc, a total of 5 metrics) for each algorithm on the test data and store it.
3) I carry out the above procedure (1 & 2) 10 times, and have the result for the four algorithms, for each algorithm a table of 50 values (1o iterations and 5 performance metrics).
Now, I would like to do the statistical significance of my algorithm against the other three algorithms. What is the best test for this, and how to do it ?, and if there is a readymade procedure in matlab, R, python or octave, can you please point to me ?