I have a problem statement where I have two dataset one labeled where a data point can belong to only one class say class1 or class2 and there's another unlabeled dataset. Now for unlabeled dataset I want to predict the class label where a particular data point can belong to class1 or class2 or both or none of the classes. How to approach this problem?
3
-
$\begingroup$ This is a classical supervised classification problem. You build a 'best' classifier (RF, SVM, etc.) on the 'labeled' dataset and then use it to predict the labels on the unlabeled dataset. $\endgroup$– utobiCommented May 24, 2023 at 10:32
-
1$\begingroup$ @utobi Because of the possibility of the unlabeled set having members belong to both or neither class, I see a wrinkle beyond the classical supervised classification setting. $\endgroup$– DaveCommented May 24, 2023 at 10:35
-
$\begingroup$ Oh I see, I missed the 'none of the classes' part! $\endgroup$– utobiCommented May 24, 2023 at 10:39
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
A classical supervised classification problem is similar to yours. In such a setting, you use the data with the known outcomes to train a model. Once you have confidence in such a model, you use it to predict the category (or probability of category membership) in data where the outcome is not known. The standard machine learning classification models like logistic regressions and neural networks do exactly this and provide you the probability of membership in each of two mutually exclusive categories (or $3+$ mutually exclusive categories, depending on what exactly you do).
However, you want to allow for membership in both categories or neither category. That makes this a so-called multi-label problem. The underlying theory is not much more complicated. You just predict the probability of each category without the two probabilities having to add up to one; that is, the categories are not mutually exclusive. One of the simple models for such a problem is a bivariate probit model, which can be implemented in R through the VGAM::binom2.rho function, documented here with references to textbooks and the primary literature for further reading to dive down the rabbit hold. It is typical to use a threshold of $0.5$ to classify the probabilistic predictions as belonging to the category (above the threshold) or not (below the threshold), though this might not be the ideal threshold for you, and the raw probabilities are useful without you having to use any threshold at all.
Beyond generalized linear models like bivariate probit, other standard machine learning models can be adjusted for multi-label problems, too. For instance, the sklearn documentation discusses multi-label implementations for $k$-nearest neighbors, random forest, and neural network models.
I have some skepticism about sticking your data into a multi-label model, however.
By training on data where the categories are mutually exclusive, you are telling the model not to expect membership in both categories. This makes it unlikely to predict a high probability of membership in both categories. Maybe this is correct behavior, but maybe it is not.
If the unlabeled data allow for membership in both categories (or neither category) yet the training data do not, I wonder what else has changed. The unlabeled data might not play by the same rules or even similar rules, meaning that the model you train on the labeled data might not be a good one for the unlabeled data, despite strong (validated) performance on the labeled data. Worse, you have no way to check this, since you literally lack labels on the unlabeled data set, rather than having them but holding them out from the training specifically to check performance on unseen data.
