I am working on a medical data set of 60 test samples and I have the decisions of 45 medical experts for them over 5 diagnostic classes. How should I do hypothesis testing to show that the classifier I trained is not statistically very different than the group of experts with some significance level?

I basically have :

  • Diagnosis by medical experts : 60 x 45
  • Diagnosis by my classifier : 60 x 1
  • Ground truth : 60 x 1

Solutions that I can think of:

  • Calculating the accuracies of both mine (1) and the experts (45). Then doing ztest with the mean and standard deviation of accuracies of experts to show that my score is coming from the same distribution.

  • Doing McNemar's test between mine and each of the experts and then saying 'we cannot reject the hypothesis of my algorithm is not different from ,i.e., 20 expert.

I am not really sure how to do this analysis scientifically. Basically I want to show that my algorithm is as successful as one of those experts in this task. Any other recommendation is also welcomed.


I'll assume your goal is to show that your classifier is as good as, or beats, the mean accuracy of an expert.

I think the easiest thing you could do would be a bootstrap. Basically, you randomly generate a sample of 45 expert accuracies by sampling from the original distribution of expert accuracies with replacement. You then calculate the mean accuracy from this distribution. Do this, say, 1000 times, and you'll generate a distribution of mean expert accuracies. Then calculate what fraction of those mean expert accuracies your classifier accuracy is better than. If this number is not very small, then you can claim that your classifier is consistent with being as good as a group of experts. If it's very close to one you can claim that your classifier beats a group of experts. (In which case the p-value will be one minus that fraction.)

Now, if you just want to claim that your classifier is as good as some expert, your task is easier. As long as your classifier beats any expert in the sample your job is done.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.