6
$\begingroup$

When constructing the ROC curve for various classifiers I've noticed that their actual shapes tend to be very different for models such as logistic regression or SVM compared to k-NN. For instance, in the image below we see a ROC curve that corresponds to a k-NN classifier, and as it can be seen there are hardly any 'steps' or jumps, if any at all, hence being quite smooth. On the other hand, if we observe the ROC curve that I obtain for the SVM model the steps are definitely more abundant. I've tried this for various datasets, and steps or jumps always seem to be absent in the case of the ROC curve for k-NN in scikit-learn; I don't know if R does this as well. Does anybody know why the ROC curve for k-NN might adopt this specific shape, whereas the ones for SVM or LR involve noticeably more steps? Thanks a lot in advance.

enter image description here

enter image description here

$\endgroup$
4
  • $\begingroup$ How are you creating these ROC curves? Really every ROC curve should look like your second example: it must make discrete jumps as new data points become classified as being members of the positive class. It really can't happen that an ROC curve is piecewise linear, as in the first example. $\endgroup$ Aug 28 '17 at 3:06
  • 1
    $\begingroup$ @MatthewDrury Piecewise linearity can happen, depending on how you deal with ties. $\endgroup$
    – Sycorax
    Aug 28 '17 at 3:52
  • $\begingroup$ ....Thinking.... $\endgroup$ Aug 28 '17 at 3:58
  • 1
    $\begingroup$ @MatthewDrury These ROC curves are being created on scikit-learn, and for the k-NN one $k=9$ and $p=1$. Conversely, for SVM $\gamma=0.1$ and $C=2^{-5}$. And it definitely can happen, as I said, I've verified this with multiple datasets. Also, if you have a look at this link that I found on the internet, they seem to obtain a similar ROC curve shape for k-NN as me: stat.washington.edu/courses/stat391/spring13/_images/…, $\endgroup$
    – Jayjay95
    Aug 28 '17 at 11:37
9
$\begingroup$

Diagonal lines occur in ROC curves when you have ties, that is when you have one or more observation with the exact same test value in both the positive and negative groups.

SVM classifiers output continuous probabilities. The chance to get a tie is normally very low. This is why you get a "steppy" curve.

On the other hand, due to the limited number of observations that make a k-NN decision, the output probabilities are typically not continuous. For instance if you considered a k-NN classifier with k = 3, probabilities can only be 1.0, 0.67, 0.33, 0. It is very likely that you observe these values in both groups, hence the tie and the diagonal line.

$\endgroup$
4
  • $\begingroup$ That's an extremely helpful response, and it makes complete sense! My k-NN currently uses $k=9$ and $p=1$, that is, Manhattan distance, but I guess the principle that you have laid out applies to my model too. Furthermore, for my test set I only have 13 1s out of 88 outcomes, which is a very small fraction compared to the 0s. Therefore, this might be influence this as well. Do you know of any book or reference which seem to mention your explanation? Also, whenever a ROC curve has a lot of steps, this implies that a lot of thresholds can be fixed, right? Again, amazing explanation. $\endgroup$
    – Jayjay95
    Aug 28 '17 at 11:57
  • $\begingroup$ I usually like to point to Fawcett's really good "Introduction to ROC analysis" cvrr.ucsd.edu/ece285/papers/Fawcett_IntroductionToROC.pdf . See figure 6 for this specific case. $\endgroup$
    – Calimo
    Aug 28 '17 at 12:02
  • $\begingroup$ I would just add to the explanation that some variants of roc conventionally wrap a hull around the steps and compare the area under that curve instead of the step curve. So if all your curves are piecewise linear in some software, that might be the reason. $\endgroup$ Aug 28 '17 at 14:29
  • 1
    $\begingroup$ Minor note: SVM does not naturally yield a probability for predictions but a signed distance from the separating hyperplane. Subsequent methods like Platt scaling can enforce a probability, but is secondary to the SVM itself. $\endgroup$
    – Sycorax
    Aug 28 '17 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.