# Significance tests for differences in performance of many clustering algorithms

I have $$n$$ clustering algorithms which are trained and evaluated on the same dataset, and I want to test whether the differences in their performances are significant or not.

The dataset is PAN17-Clustering, in which there are multiple clustering problem sets (60 for training, 120 for testing) and the clustering is operated on each problem set, which means the different runs and sub-scores are independent from each other.

The final score of an algorithm is averaged over the 120 test problem sets. Inasmuch as the ground truth is given, the evaluation criteria are extrinsic, and are $$B^3 F$$ score and the adjusted rand index $$ARI$$. Here are the results:

As you see, the differences aren't that remarkable. I would like to test for the significance of the differences I see in the performances of the clustering algorithms, so would you please advise in that regard?

• What do these plots show? Are these ratios? Why are there intervals? Sep 3 '19 at 13:09
• They're the final average scores of the algorithms on the dataset, segregated by language -it's text clustering.The black intervals are the error bars, serving as some indication of the uncertainty around that estimate over 120 runs. Sep 3 '19 at 13:21
• So If I understand correctly, you want to test whether the clustering algorithms perform similarly within countries? For each country and algorithm you performed many analyses, obtained some scores, and now want to test whether these scores are all equal? Sep 3 '19 at 13:25
• I want to test for the significance of differences we see in their performances, are these differences significant or not? can we significantly say one is better than the other? that can be cascaded to the genre or language level, or even overall. I have previously tested for the significance of two classifiers' difference in performance using McNemar's test, but in my case it's unsupervised clustering with no response variable.. hence the query. Sep 3 '19 at 13:43

• Thank you for the response, I have few doubts though. Why none of the results are acceptable? the task is authorial clustering, and carefully crafted approaches barely exceed 0.57 in terms of $B^ 3F$. I agree that the task is challenging. $ARI$ was used only for the sake of the constant baseline property, but it doesn't fare well maybe due to the strong assumptions it makes. I was hoping to statistically tell whether one algorithm outperforms the others (similar to McNemar's test for instance). As for the standard error of the estimated mean, it is depicted with the black error bars. Sep 3 '19 at 23:05