I've had discussions with several professors, as well as scouring the internet (and chatgpt), trying to improve the description of the methods I've used, or to look for better ones. I'm hoping a wider audience might have some tips.

I am trying to compare the performance of two methods of classification for a specific medical situation. My claim is that when the 95% CIs of the median performance metric overlap, the methods are equivalent in performance.

There are a few challenges, including ones that make standard methods like Mann–Whitney U tests impossible:

  • There is no ground truth, so the classification metric is Cohen's kappa.
  • For the current technique (sample A), there have only been a few (small) studies that have calculated kappa among recordings, and, even worse, access to that data is prohibited. This means that I only have summary statistics for the current technique. This technique is the gold standard (i.e. humans classifying the data using the agreed upon rules). But, two humans will disagree sometimes on the same data.
  • For my technique, I do have a kappa value for each recording, so I can do any number of tests on this sample (sample B). But for each recording I can only calculate one kappa value (because each was only classified by one human).

Given the limitations, I have done the following:

  1. For sample A, I have the median and 1st and 3rd quartile kappas, and I calculated the 95% confidence intervals by assuming normality and using the known IQR and sample size.
  2. For sample B, I have the kappa value for each recording, so I performed bootstrapping to get the median and 95% CIs.
  3. Using these, I found that the 95% CIs of sample A and sample B overlap, and thus I cannot reject H0.

The professors I have talked to feel that the technique, while creative, is acceptable. My concern is that because of the limitations, there doesn't seem to be a simple name for the technique I've used.

  • 3
    $\begingroup$ There are several lapses of logic and statistical reasoning here. One is the use of CI overlap: that is by no means a test conducted at a nominal 95% confidence level! The other is the misinterpretation of overlap: failure to reject a null does not imply equivalent performance. Because your focus is on performance, then that's what you should be measuring. You need a relevant, valid concept of effect size to compare. A useful search term here is "TOST," or two one-sided tests. $\endgroup$
    – whuber
    Commented Jan 21 at 18:26
  • $\begingroup$ Thank you. I believe to use TOST I'll need actual sample values from Sample A. But, as I stated, I don't have them. So, would you suggest simulations? The same issue exists with trying to compare the difference in the samples to see if the 95% CIs of the differences include 0--I actually need the points from Sample A. $\endgroup$
    – Adam Jones
    Commented Jan 21 at 19:36

1 Answer 1


You might backup a little and look at how classifier performance is determined or compared. Firstly, during the final stages in classification analysis to determine if one classifier is better than another, statistical testing is not the default approach that's used. But it is easy to understand why an analyst would want to go there. Typically, there is a lot of computation used to determine the final e.g. class prediction accuracy for a classifier. But unfortunately, statistical hypothesis testing using kappa (interrator agreement) or chi-squared multiway tables for confusion matrices is not commonly used to determine which classifier is better. Rather, accuracy, F1 recall, and ROC-AUC are used. In other words, statistical tests are not commonly used to compare classifiers based on their result. Instead, plots of accuracy, recall, AUC are compared.

No Free Lunch (NFL) Theorem (Wolpert & MacReady) - "In the universe of all classifiers, there is no best set of classifiers or single classifier."

Back to your professors and what you did. First, there's no problem with what you did and it's very appropriate to apply what you did to use tests (kappa) to essentially prove beyond chance occurrence (p<0.05) that one classifier is better than another. However, if you were in a CS dept or presented results at an IEEE-related pattern recognition, ML, or neural network type conference, you would be asked why you ran statistical hypothesis testing on your classifier results. The rationale is that if you change the data, all of your results won't apply. Second, there's no guarantee on robustness of your results - that is, your conclusions on classifier performance will not apply to a variety of different operating environments using different datasets and applications. Third, if you changed the features you used, everything changes - so this further diminishes the robustness of your results. That is, your results can break down with different data/features.

Ugly Duckling Theorem (Watanabe): "In the universe of all feature sets, there is no best set of features."

In summary, if your professors were CS faculty they would say, "okay, what you did is fine for the unique dataset/problem you looked at, but when you get out in the wild (real world) the particular classifiers you employed are not that good and really break down and fail under the following conditions.... Thus, in this CS dept, the concern is robustness of classifier performance - i.e., comparison of classifier performance in many different environments, not tacking on a lot of contrived statistical hypothesis testing to compare a couple classifiers for a limited experiment involving a single dataset."

Last, classifier comparisons typically involve use of 15-20 standard classification datasets which have a good mixture of number of features, number of objects, and number of classes. The tests you're doing for a small singleton dataset won't make a strong contribution to the literature, since you can't guarantee robustness with two classifiers and one dataset.

What you did is not wrong or erroneous; there are just many more limitations to the generalizability of your results -- that is, they only apply to the very small set of classifiers you used and the single dataset you employed.

  • $\begingroup$ Although it may sound strange to CS (and several other fields) not to use accuracy (or F1, recall, etc), medicine is replete with classification problems where kappa is the only reported or acceptable measure; ground truth—and therefore a sense of accuracy—sometimes just doesn't exist. I agree, in general, regarding the caveats of running hypthosis testing for classifier comparisons. Although, I'm a bit perplexed as to why you highlighted this several times, as your examples (e.g. changing features) apply to any classifier—even those that can use accuracy. $\endgroup$
    – Adam Jones
    Commented Jan 21 at 4:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.