By having a data-set 1000 (900 unlabeled, 100 labeled) record data-set for binary classification, I want to apply a semi supervised algorithm. The problem is that I don't know how to get values for the ROC plot?
Actually the algorithm tries to find a threshold on a probability value (base on the 100 labeled records) in order to achieve the best F1 score for binary classification.
What I'm thinking is to set different values to the threshold but I don't know that after selecting a threshold should I try it on the 100 records in training to get output point for ROC (which means evaluating the performance of the threshold on training set) or I have to test it on 900 records (test) to get the output point for ROC (which is the evaluation of the select threshold's performance on the test set)?
Please help me!