1
$\begingroup$

By having a data-set 1000 (900 unlabeled, 100 labeled) record data-set for binary classification, I want to apply a semi supervised algorithm. The problem is that I don't know how to get values for the ROC plot?

Actually the algorithm tries to find a threshold on a probability value (base on the 100 labeled records) in order to achieve the best F1 score for binary classification.

What I'm thinking is to set different values to the threshold but I don't know that after selecting a threshold should I try it on the 100 records in training to get output point for ROC (which means evaluating the performance of the threshold on training set) or I have to test it on 900 records (test) to get the output point for ROC (which is the evaluation of the select threshold's performance on the test set)?

Please help me!

$\endgroup$
  • $\begingroup$ This is a nice example of making your own ROC plot using R. rrandomness.com/r/simple-roc-plots-with-ggplot2-part-1 $\endgroup$ – D L Dahly Dec 30 '12 at 22:46
  • $\begingroup$ Thank you very much for the example, it helped to understand the logic behind the ROC, but now I'm wondering that should I draw it base on the training or the test? Is it used to help for the cut point value selection (in this case I will draw that for the training set) or it is used to show the performance on the algorithm no matter what is the cut point (on the test set)? $\endgroup$ – NULL Jan 1 '13 at 3:55
1
$\begingroup$
  • In order to calculate the ROC, you need cases labelled by a reference, and a continuous score assigned to the cases by the classifier (so no hard 0/1 output after your algorithm decided on an "optimal" threshold). So your unlabelled cases cannot be used for the ROC.

  • What kind of samples (training or independent test data) to use for the ROC plot doesn't matter for the calculation that produces the plot. It does matter for the interpretation: if the data is already known to the model, you test goodness-of-fit, if the cases are completely unknown (as e.g. in properly done out-of-bootstrap or cross validation), you test generalization performance. For many multivariate classification problems, goodness-of-fit is nearly useless, as you very easily get into overfitting situations.

  • If you optimize a model (hyper)parameter like threshold, that is a data-driven modeling step. I.e. the choosing of the parameter is part of the modeling, and needs to be tested with independent data.

  • From what you write, your algorithm chooses a threshold. Applying this,you will not get a ROC, but a single point. But what you can do is plot the ROC and mark the working point chosen by your algorithm.

$\endgroup$
  • $\begingroup$ Regarding your first bullet (which is valid): you may be interested in a technique we developed to compute strict bounds on ROC curves based on only positive and unlabeled data that is based on assigning surrogate labels. We show that unlabeled data can greatly contribute to performance estimation, given some reasonable assumptions. You can find a (somewhat dated) version of the manuscript at arxiv.org/abs/1504.06837. $\endgroup$ – Marc Claesen Dec 8 '15 at 15:20
  • $\begingroup$ @MarcClaesen: thanks, I'll have a look at it. Unfortunately, for the problems I usually deal with, typical assumptions for semi-supervised learing like the cluster assumption or assuming that labelled data and unlabelled data have the same distribution typically do not hold (we have many unlabelled samples at the class boundaries). $\endgroup$ – cbeleites Dec 9 '15 at 11:28
1
$\begingroup$

To obtain an ROC curve using a statistics program, for each case/observation/record you need a predicted probability in the range from 0 to 1 and an observed value of 0 or 1. There's no need for you to set different values of a cut point for assigning predicted values of 0 or 1 (if cut point is what you mean by threshold), because that's taken care of by an ROC procedure. If you're trying to create your own ROC algorithm, sorry but I can't help you there.

$\endgroup$
  • $\begingroup$ Thanks for the answer Roland, yes I mean the cut point by threshold. You said for each record I have to do do that, but I have a 100 record training set (with label) and 900 test set, base on the nature of the semi-supervised, is it a good idea to draw a ROC curve for test set (900 record)? Because first I try to find the best cut point for the algorithm on the training set (100 records), so I thought that for this purpose I have to use ROC?! $\endgroup$ – NULL Jan 1 '13 at 2:50
  • $\begingroup$ I mean if for the semi supervised we assume that we have no idea about the label (observed value) for the majority (900 records). So if I try to find the best cut point by testing the algorithm on those records it is actually won't be a semi-supervised algorithm, am I right? $\endgroup$ – NULL Jan 1 '13 at 2:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.