0
$\begingroup$

I have a ROC curve like the figure below: enter image description here

What can be inferred given this kind of curve in general?

What are the differences of states 1, 2 and 3 in the figure?

What causes the diagram to jump straight from state 1 to state 3? Most of the ROC curves on the internet look like the figure below:

enter image description here

$\endgroup$
  • $\begingroup$ I gave a general answer. If you could add details about your data (number of samples above all) I could probably detail it some more. $\endgroup$ – Firebug Aug 17 '16 at 14:26
  • 2
    $\begingroup$ Probably your model predictions only have 2 values. $\endgroup$ – Sycorax Aug 17 '16 at 14:31
  • $\begingroup$ @GeneralAbrial Yes.It is actually a binary classifier. or what do you mean by 2 values? $\endgroup$ – user126608 Aug 17 '16 at 14:35
  • 3
    $\begingroup$ I mean that when you present some data to the classifier to make predictions, it gives either A or B as an output, rather than a continuous outcome in some range. For example, logistic regression gives output in (0,1), so any real number between 0 and 1 (such as 0.2, 0.123123, or 0.996) is a legitimate prediction. But your model might only give 0.0 or 1.0. $\endgroup$ – Sycorax Aug 17 '16 at 14:37
  • 1
    $\begingroup$ Could you post a small samples of the predicted probabilities? Or perhaps a histogram. $\endgroup$ – Firebug Aug 17 '16 at 14:43
2
$\begingroup$

I gather from the comments that you only have 2 unique predicted values. When you present some data to the classifier to make predictions, it gives either A or B as an output, rather than a continuous outcome in some range. For example, logistic regression gives output in (0,1), so any real number between 0 and 1 (such as 0.2, 0.123123, or 0.996) is a legitimate prediction. But your model might only give 0.0 or 1.0.

Because your ROC curve must go from (0,0) to (1,1) and be non-decreasing, and you have 2 unique prediction values, you can only have 2 points in between. Thus, the plot you have.

Most ROC curves are produced by models that give continuous outcomes. Because there are more decision points, there are more opportunities to measure error rates rates. That produces the kind of ROC curve in the second figure. By way of contrast, for example, logistic regression gives output in (0,1), so any real number between 0 and 1 (such as 0.2, 0.123123, or 0.996) is a legitimate prediction. But your model might only give 0.0 or 1.0.

$\endgroup$
  • $\begingroup$ @ General Abrial Is my case considered to be a problem?if so how can I fix it? $\endgroup$ – user126608 Aug 17 '16 at 14:57
  • 1
    $\begingroup$ If you're using an SVM, you need the signed distances from the hyper plane. Read the documentation. $\endgroup$ – Sycorax Aug 17 '16 at 15:01
  • $\begingroup$ If the output was only {0,1} the ROC would have three points (there are three possible thresholds with only two output values), so that point is invalid. $\endgroup$ – Firebug Aug 17 '16 at 15:17
2
$\begingroup$

You have four point estimates of sensitivity and specificity. This could be due to really grouped estimates or because you have too few samples.

$\endgroup$
0
$\begingroup$

When calculating the AUC, you need to provide a range of values (as @General Abrial mentioned), not the class labels (0/1).. so you need to find how to get the non thresholded output from your SVM. For instance in scikit learn you would use decision_function, rather than the predict function

http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy