How to generate ROC Plot for semi-supervised algorithm? By having a data-set 1000 (900 unlabeled, 100 labeled) record data-set for binary classification, I want to apply a semi supervised algorithm.
The problem is that I don't know how to get values for the ROC plot?
Actually the algorithm tries to find a threshold on a probability value (base on the 100 labeled records) in order to achieve the best F1 score for binary classification.
What I'm thinking is to set different values to the threshold but I don't know that after selecting a threshold should I try it on the 100 records in training to get output point for ROC (which means evaluating the performance of the threshold on training set) or I have to test it on 900 records (test) to get the output point for ROC (which is the evaluation of the select threshold's performance on the test set)?  
 A: To obtain an ROC curve using a statistics program, for each case/observation/record you need a predicted probability in the range from 0 to 1 and an observed value of 0 or 1.  There's no need for you to set different values of a cut point for assigning predicted values of 0 or 1 (if cut point is what you mean by threshold), because that's taken care of by an ROC procedure.  If you're trying to create your own ROC algorithm, sorry  but I can't help you there.
A: *

*In order to calculate the ROC, you need cases labelled by a reference, and a continuous score assigned to the cases by the classifier (so no hard 0/1 output after your algorithm decided on an "optimal" threshold). So your unlabelled cases cannot be used for the ROC.

*What kind of samples (training or independent test data) to use for the ROC plot doesn't matter for the calculation that produces the plot. It does matter for the interpretation: if the data is already known to the model, you test goodness-of-fit, if the cases are completely unknown (as e.g. in properly done out-of-bootstrap or cross validation), you test generalization performance. For many multivariate classification problems, goodness-of-fit is nearly useless, as you very easily get into overfitting situations. 

*If you optimize a model (hyper)parameter like threshold, that is a data-driven modeling step. I.e. the choosing of the parameter is part of the modeling, and needs to be tested with independent data.  

*From what you write, your algorithm chooses a threshold. Applying this,you will not get a ROC, but a single point. But what you can do is plot the ROC and mark the working point chosen by your algorithm. 
