I have a set of 300 observations of known class and there are 20 classes in total.

I want to compare the performance of a random classifier with a weighted-random classifier at classifying the observations. The random classifier obviously allocates each observation to a random class whereas the weighted-random classifier uses information about the expected probability distribution of the class occurrence to weight the allocation of each observation to a class (this probability distribution is potentially different for every observation).

I would expect the weighted-random classifier to be more accurate than the completely random classifier, and this is borne out by a comparison of the proportion-correctly-classified: about 30% for the weighted random classifier; about 20% for the random classifier.

I understand that I can assess whether the difference in accuracy is statistically significant by performing McNemar's test.

Obviously, though, if I repeat the classifications I get different results because both classifiers are random to a greater or lesser degree. Thus it is possible that the McNemar's test will report different results from run to run. As such I am not sure how to analyse this situation. Does it make sense to run the classifiers, say, 100 times and report the proportion of times McNemar's test returns a significant difference? Also, as an aside, if the accuracy of the weighted-random classifier is greater than the random classifier and McNemar's test reports that the difference in accuracy is statistically significant, can I say that the weighted-random classifier is better than the random classifier?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.