0
$\begingroup$

Im working on assessment of algorithm sensitivity and specificity. I've developed a simulation in order to detect true and false positives and negatives. My intersest is to know if my algorithm is capable of detect a event, or not. So, in the end, I got several yes or no, capable or not capable of detection. Is that possible to run ROC analysis/curves with binary input? Made some tests with ROCR in R without sucess.

I.e.

1400 events true positives 600 events false negatives

1900 events true negatives 100 events false positives

Thanks in advance

$\endgroup$

1 Answer 1

1
$\begingroup$

If you only have binary inputs this will only give you one point on the ROC curve. Usually you need something like a probability estimate. Many algorithms (random forests) that aren't meant to produce probability estimates have been modified to provide some estimate.

Another alternative method is to use cost sensitive learning to produce k models which will give k points on the ROC curve. By changing your cost of FN/FP for each model you will produce different number of FP/FN.

$\endgroup$
5
  • $\begingroup$ Thank you for the answer and alternatives. Im curious if there's another possibility instead of ROC analysis to do this job? $\endgroup$
    – AndreiR
    Aug 17, 2014 at 18:14
  • $\begingroup$ ROC curves are very popular, but not always necessary. They are an attempt to graphically represent the trade-off between FP and FN. If this isn't of interest, no need to use them. Other than it might be expected. Many people report (sens + specificity and F1 score) or (precision+recall+F1) or (accuracy)/(percentage correctly classified (PCC)). Depends of your audience. All of these assume the cost of FP= cost of FN. $\endgroup$
    – charles
    Aug 18, 2014 at 3:29
  • $\begingroup$ I'll add that most of want I read of ML assumes FP cost = FN cost. This issue is not covered very well, so if you're coming for a ML perspective can be hard to find good review. WEKA has a few cost sensitive learners, R much fewer. $\endgroup$
    – charles
    Aug 18, 2014 at 3:38
  • $\begingroup$ Charles, as you may noticed, I have no math background. Now, things are far more clear. Thank you for the informations and patience. Beyond accept, would up vote. Think I'll need to wait until I get 15 :) $\endgroup$
    – AndreiR
    Aug 18, 2014 at 5:06
  • $\begingroup$ No problem. Hope useful. Good luck! $\endgroup$
    – charles
    Aug 18, 2014 at 5:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.