0
$\begingroup$

I have a dataset that each feature in a data could have two separate labels depending on separate definitions. According to definition 1, each feature could have one of two labels (A, B). According to definition 2, each feature could have one of eight labels (I, II, III, IV, V, VI, VII, VIII). My goal is to compare the strengths of the two types of classifications within each data. This is to answer a scientific question with the hypothesis that, in some data, the classification strength is stronger for Type I labels than for Type II labels, while in other data, the strength might be stronger for Type II labels than for Type I labels; in other cases, the strengths could be equally strong or equally weak for both classifications. Then I plan to divide my data into four groups based on the comparisons (Strong Type I, weak Type II; Strong Type II, weak Type I; Strong Type I, strong Type II; Weak Type I, weak Type II) and relate them to my other measurements to further answer some questions.

Previously, I have already built one algorithm for the Type I single-label classification to discriminate between 2 different labels (A, B). I used Gaussian kernelized SVM with nested cross-validation, and it worked out pretty well. Now, I can either just use the same algorithm to build a separate classifier for Type II labels (I, II, III, IV, V, VI, VII, VIII). But I'm not confident about the reliability of drawing a conclusion of the classification strengths between the two classifiers directly because they are two classifiers with different chance levels, etc. (i.e., close to 50% for type I label classifier, and close to 12.5% for type II label classifier). I know there are other metrics that might be more reasonable for direct strengths comparison than accuracy (e.g.,f1 score, AUC), but I'm still not sure if training two seperate classifiers is the best way for the comparison.

Alternatively, I learned about multi-label classification. According to my understanding, it trains one classifier for multiple types of labels. I wonder if multi-label classification could be a better way for me to get a reliable comparison score between classification strengths for the two different types of labels than a direct comparison between two separate single-label classifications? If yes, is there a recommended algorithm/python package/metrics for my case? Also, the algorithm I used for single-label classification (i.e., Gaussian kernelized SVM with nested cross-validation) worked out well, is it possible to keep this part of the algorithm if I scale up to multi-label classifications?

I am personally lining towards multi-label classification because I might have more than 2 types of labels for each feature in the future according to additional definitions. In that case, I might also need to compare Type II strengths with Type IV strengths, and Type I strengths with Type III strengths. It might be easier to implement one multi-label classification model than implement 4 or 5 different single-label classification models at that point, but I'd like to get advice from people who have implemented this before.

Thanks. Any input is appreciated.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

This boils down to scientific value and interpretation of the different labels. If both label definitions carry scientific information that can not be decoupled then multi-output classification is recommended such as sklearn.multioutput.MultiOutputClassifier, avoiding the assessment of the relative performance between two single classifiers.

If both label definitions are de-coupled completely from scientific perspective then two classifier, binary and multi-class should be build and compared.

See also so called Stacking

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your comment. I think I will go for two single classifiers then. My only follow-up question is, how could I compare the strengths of the two classifiers then? One is classifying among 2 categories, the other is among 8 categories. $\endgroup$
    – LCheng
    Jun 14, 2021 at 19:57
  • $\begingroup$ @Lily Use median Top-1 score coming from two classifiers for comparison. see recent post on this: top-1 evaluation . PS: If you find answer helpful, can you up-vote and accept. $\endgroup$
    – patagonicus
    Jun 15, 2021 at 4:39
  • $\begingroup$ Happy to up-vote, but what is the median top-1 score? Can you give me an example? Also, are there any references for using the median top-1 score as the comparison metrics (I'm asking because I need to persuade other scientists that my method is valid)? $\endgroup$
    – LCheng
    Jun 15, 2021 at 21:34
  • $\begingroup$ Also, by saying top-1 score, are you just referring to recall instead of precision? Why is that? $\endgroup$
    – LCheng
    Jun 15, 2021 at 21:49
  • $\begingroup$ @Lily 'median Top-1 score' is average of recall and fall-out in binary classifier, sometime called balanced accuracy. But for multi class confusion matrix, the terminology of recall and fall-out comes short, so actually they both are top-1 scores. So we compute Top-1 score per class and take the median for multi-classifier evaluation. This is advantageous because (1) Resistant to class imbalance (2) One can compare two classifier having different number of classes. Though naming terminology may be different depending on the field. $\endgroup$
    – patagonicus
    Jun 16, 2021 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.