How to get random classification to assess the performance of classifier with McNemar test?

Question

I'm trying to replicate a study where the author used the McNemar test to assess the performance of classification compared to random classification. I have the original classifier and I'm using R to do the McNemar test, but I don't know how I'm supposed to obtain the random classification, can someone help me?

Answer 1

You might try to contact the author to learn more about what they did. Your description doesn't make sense (which doesn't necessarily mean it's inaccurate or you misunderstood—people do a lot of things that don't make sense). As @ttnphns notes in the comments, McNemar's test is for a paired comparison of yes/no data; you'd use that when comparing two different classifiers on the same dataset (cf., Compare classification performance of two heuristics).

For your situation, to determine if your accuracy is better than would occur by chance alone, you want to do a one-sample test of your classifier's percent correct vs. the proportion correct you would get if you just always guessed the more commonly occurring class.

On the other hand, you could use McNemar's test to determine if your model is biased (in the sense that it guesses one of the classes more often than it 'should'; see: How to calculate information included in R's confusion matrix).

Answer 2

Does the original publication not elaborate on that?

In my understanding, you just "randomly" assign a class to an observation. For starters, you could go with something like:

sample(vector_of_classes, number_of_observations, replace = T)