Can supervised classification models be used as hypothesis testing to compare alternative groping of labels?

Question

Since I didn't find any resource online, I'm asking here.

In this paper of Pereira et al.,(2007; https://doi.org/10.1600/036364407780360201), they use cross validated Canonical Discriminant analysis to test whether, from a morphological point of view, the species studied are worth to be separated or must be merged.

Is this, from the Machine Learning point of view legit? Specifically:

1. Can supervised classification models be used in hypothesis testing?
2. What are the effects of reducing the number of classes on the possibility of getting an higher accuracy just by chance?
3. Does che change of labels influence the structure of the data?
4. If it is wrong, is there any more sounded methods to do so?

Thank you in advance

Answer 1

Here is an open-source Python library I helped develop that contains functionality specifically for helping you decide on this question. The dataset.health_summary() method will report pairs of classes that are most often confused by your data labelers that you may consider merging.

Of course merging classes will mean you see higher accuracy values when training ML models. Whether merging classes is a good idea or not really depends on the application, not only the data itself, so you should decide this using your domain knowledge as well.

As an example, the famous ImageNet image classification dataset contains very confusing pairs of classes that should probably be merged: "keyboard" vs "space bar", "missile" vs "projectile", and "maillot" vs "maillot" (yes there are 2 classes both called "maillot" in ImageNet...)