Significance tests for differences in performance of many clustering algorithms I have $n$ clustering algorithms which are trained and evaluated on the same dataset, and I want to test whether the differences in their performances are significant or not.
The dataset is PAN17-Clustering, in which there are multiple clustering problem sets (60 for training, 120 for testing) and the clustering is operated on each problem set, which means the different runs and sub-scores are independent from each other.
The final score of an algorithm is averaged over the 120 test problem sets. Inasmuch as the ground truth is given, the evaluation criteria are extrinsic, and are $B^3 F$ score and the adjusted rand index $ARI$. Here are the results:

As you see, the differences aren't that remarkable. I would like to test for the significance of the differences I see in the performances of the clustering algorithms, so would you please advise in that regard?
 A: There is little use in significance testing the performance at this level. Among the best performers is "single link", and the results are clearly marginally better than random. Clearly, none of the results was acceptable. Maybe due to preprocessing, maybe she to a badly chosen task, maybe because clustering text never works.
So what would a significance test tell you? Essentially just that one method such as spherical k-means is "significantly more similar to random" than the other.
Make sure to also include some trivial baselines in your plots! Such as making every object it's own cluster, putting all objects into the same cluster, and choosing k random objects and assigning each remaining document to the nearest of that set. Methods that cannot beat these trivial baselines clearly did not work...
Furthermore, include the standard error of the mean. This gives you how certain your estimate of the mean is. Two methods where these estimates overlap cannot be significantly better than the other.
Before even bothering to think of a more complex significance test (Friedman-Nemenyi would be a candidate worth looking at), you'd need to first get some acceptable results at all... Otherwise you are just testing what was worse...
