1
$\begingroup$

I have $n$ clustering algorithms which are trained and evaluated on the same dataset, and I want to test whether the differences in their performances are significant or not.

The dataset is PAN17-Clustering, in which there are multiple clustering problem sets (60 for training, 120 for testing) and the clustering is operated on each problem set, which means the different runs and sub-scores are independent from each other.

The final score of an algorithm is averaged over the 120 test problem sets. Inasmuch as the ground truth is given, the evaluation criteria are extrinsic, and are $B^3 F$ score and the adjusted rand index $ARI$. Here are the results:

The results of the competing clustering algorithms

As you see, the differences aren't that remarkable. I would like to test for the significance of the differences I see in the performances of the clustering algorithms, so would you please advise in that regard?

Improve this question
$\endgroup$
4
  • $\begingroup$ What do these plots show? Are these ratios? Why are there intervals? $\endgroup$
    – user2974951
    Sep 3 '19 at 13:09
  • $\begingroup$ They're the final average scores of the algorithms on the dataset, segregated by language -it's text clustering.The black intervals are the error bars, serving as some indication of the uncertainty around that estimate over 120 runs. $\endgroup$
    – Jabro
    Sep 3 '19 at 13:21
  • $\begingroup$ So If I understand correctly, you want to test whether the clustering algorithms perform similarly within countries? For each country and algorithm you performed many analyses, obtained some scores, and now want to test whether these scores are all equal? $\endgroup$
    – user2974951
    Sep 3 '19 at 13:25
  • $\begingroup$ I want to test for the significance of differences we see in their performances, are these differences significant or not? can we significantly say one is better than the other? that can be cascaded to the genre or language level, or even overall. I have previously tested for the significance of two classifiers' difference in performance using McNemar's test, but in my case it's unsupervised clustering with no response variable.. hence the query. $\endgroup$
    – Jabro
    Sep 3 '19 at 13:43
1
$\begingroup$

There is little use in significance testing the performance at this level. Among the best performers is "single link", and the results are clearly marginally better than random. Clearly, none of the results was acceptable. Maybe due to preprocessing, maybe she to a badly chosen task, maybe because clustering text never works.

So what would a significance test tell you? Essentially just that one method such as spherical k-means is "significantly more similar to random" than the other.

Make sure to also include some trivial baselines in your plots! Such as making every object it's own cluster, putting all objects into the same cluster, and choosing k random objects and assigning each remaining document to the nearest of that set. Methods that cannot beat these trivial baselines clearly did not work...

Furthermore, include the standard error of the mean. This gives you how certain your estimate of the mean is. Two methods where these estimates overlap cannot be significantly better than the other.

Before even bothering to think of a more complex significance test (Friedman-Nemenyi would be a candidate worth looking at), you'd need to first get some acceptable results at all... Otherwise you are just testing what was worse...

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for the response, I have few doubts though. Why none of the results are acceptable? the task is authorial clustering, and carefully crafted approaches barely exceed 0.57 in terms of $B^ 3F$. I agree that the task is challenging. $ARI$ was used only for the sake of the constant baseline property, but it doesn't fare well maybe due to the strong assumptions it makes. I was hoping to statistically tell whether one algorithm outperforms the others (similar to McNemar's test for instance). As for the standard error of the estimated mean, it is depicted with the black error bars. $\endgroup$
    – Jabro
    Sep 3 '19 at 23:05
  • 1
    $\begingroup$ Well, what is the baseline value of a trivial method for B³ F1? 0.56 maybe? That's why I said you need to include trivial approaches. These should have an ARI close to 0, but you'll still see the baseline for B³. Plus, the adjustment in ARI doesn't cover everything - it's based on random permutation of labels. People have suggested other trivial baselines to use, too. But ARI<<0.5 means it pretty much failed... $\endgroup$
    – Has QUIT--Anony-Mousse
    Sep 4 '19 at 6:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.