I have a set of 300 observations of known class and there are 20 classes in total.
I want to compare the performance of a random classifier with a weighted-random classifier at classifying the observations. The random classifier obviously allocates each observation to a random class whereas the weighted-random classifier uses information about the expected probability distribution of the class occurrence to weight the allocation of each observation to a class (this probability distribution is potentially different for every observation).
I would expect the weighted-random classifier to be more accurate than the completely random classifier, and this is borne out by a comparison of the proportion-correctly-classified: about 30% for the weighted random classifier; about 20% for the random classifier.
I understand that I can assess whether the difference in accuracy is statistically significant by performing McNemar's test.
Obviously, though, if I repeat the classifications I get different results because both classifiers are random to a greater or lesser degree. Thus it is possible that the McNemar's test will report different results from run to run. As such I am not sure how to analyse this situation. Does it make sense to run the classifiers, say, 100 times and report the proportion of times McNemar's test returns a significant difference? Also, as an aside, if the accuracy of the weighted-random classifier is greater than the random classifier and McNemar's test reports that the difference in accuracy is statistically significant, can I say that the weighted-random classifier is better than the random classifier?