The problem I have a dataset where ca. 90 % of the dataset is unlabeled and the rest is positively and negatively labeled (unbalanced). I want to describe the performance of a classifier on this dataset. Are there any metrics which can describe recall and precision of the classifier while taking into account the unlabeled data as well? This should be different to PU learning, since I have unlabeled, positive, and negative examples.
Example My classifier classifies 10 samples as positive. Out of these 10, 1 has a positive label, 2 negative labels, and 8 are unlabeled. My precision - ignoring the unlabeled - would be 1/(1+1) = 0.5 More extreme example with 10 selected as positive, out of these 1 with a positive label and 9 unlabeled would yield a precision of 1/(1+0) = 1.0
I feel that ignoring the unlabeled samples in the performance metrics might introduce a bias to the evaluation.