I have a dataset that each feature in a data could have two separate labels depending on separate definitions. According to definition 1, each feature could have one of two labels (A, B). According to definition 2, each feature could have one of eight labels (I, II, III, IV, V, VI, VII, VIII). My goal is to compare the strengths of the two types of classifications within each data. This is to answer a scientific question with the hypothesis that, in some data, the classification strength is stronger for Type I labels than for Type II labels, while in other data, the strength might be stronger for Type II labels than for Type I labels; in other cases, the strengths could be equally strong or equally weak for both classifications. Then I plan to divide my data into four groups based on the comparisons (Strong Type I, weak Type II; Strong Type II, weak Type I; Strong Type I, strong Type II; Weak Type I, weak Type II) and relate them to my other measurements to further answer some questions.
Previously, I have already built one algorithm for the Type I single-label classification to discriminate between 2 different labels (A, B). I used Gaussian kernelized SVM with nested cross-validation, and it worked out pretty well. Now, I can either just use the same algorithm to build a separate classifier for Type II labels (I, II, III, IV, V, VI, VII, VIII). But I'm not confident about the reliability of drawing a conclusion of the classification strengths between the two classifiers directly because they are two classifiers with different chance levels, etc. (i.e., close to 50% for type I label classifier, and close to 12.5% for type II label classifier). I know there are other metrics that might be more reasonable for direct strengths comparison than accuracy (e.g.,f1 score, AUC), but I'm still not sure if training two seperate classifiers is the best way for the comparison.
Alternatively, I learned about multi-label classification. According to my understanding, it trains one classifier for multiple types of labels. I wonder if multi-label classification could be a better way for me to get a reliable comparison score between classification strengths for the two different types of labels than a direct comparison between two separate single-label classifications? If yes, is there a recommended algorithm/python package/metrics for my case? Also, the algorithm I used for single-label classification (i.e., Gaussian kernelized SVM with nested cross-validation) worked out well, is it possible to keep this part of the algorithm if I scale up to multi-label classifications?
I am personally lining towards multi-label classification because I might have more than 2 types of labels for each feature in the future according to additional definitions. In that case, I might also need to compare Type II strengths with Type IV strengths, and Type I strengths with Type III strengths. It might be easier to implement one multi-label classification model than implement 4 or 5 different single-label classification models at that point, but I'd like to get advice from people who have implemented this before.
Thanks. Any input is appreciated.