Some context:
I currently have two groups of 50 text documents. Each group is about a different subject and curated by people. As such, I assume the two groups to be my ground truth.
I mix the two groups of narratives and use WarpLDA, specifying I want two groups (k = 2).
Goal
I want to assess if WarpLDA performs (and in the future other clustering algorithms), better than a random baseline model. The random baseline model just randomly assigns any of the 100 text documents to either cluster.
Evaluation
Because I have a ground truth, I first started by measuring the model accuracy using the Adjusted Random Index (ARI).
As such, I can obtain one measure of performance from WarpLDA by comparing it against the ground truth. Likewise, I can obtain one measure of performance by comparing the random model against the ground truth.
Question
My question comes at this stage. Considering clustering algorithms random initialization, my understanding is that a poor or good performance could be due to chance when compared to the random model. I could then run the algorithm 3000 times, for example, and likewise the random model, calculating the adjusted random index 3000 times for WarpLDA, and 3000 times for the random model.
Visually, I now have two histograms I wish to compare. My understanding is that assumptions willing, a 1-tailed 2-sample t-test would allow me to assess the WarpLDA performance, as measured by the adjusted random index, which is superior to a random model under a p<0.05, thus addressing my concern with the random initialization.
1) I do not have statistical training and am more familiar with the "train/test/validation" approach. Is this a fair test to use, or commonly used in the literature for evaluation of clustering algorithms? I am particularly uncertain if the assumption of independent observations holds in this case, as some material I found argues against it when used in classification, and favors McNemar's Test or a corrected t-test by Claude Nadeau and Yoshua Bengio which Weka implements it.
2) If not, would there be a more appropriate test?
Related Questions
A similar question also propose the uses of t-test, but the set-up proposes cross-validation. My interest in maintaining the dataset the same, just to observe the effect of the random initialization.
