ROC curve crossing the diagonal I am running a binary classifier at the moment. When I plot the ROC curve I get a good lift at the beginning then it changes direction and crosses the diagonal then of course back up, making the curve a tilted S like shape.
What can be an interpretation/explanation to this effect? 
Thanks
 A: You get a nice symmetric ROC plot only when standard deviations for both outcomes are the same. If they are rather different then you may get exactly the result you describe.
The following Mathematica code demonstrates this. We assume that a target yields a normal distribution in response space, and that noise also yields a normal distribution, but a displaced one. The ROC parameters are determined by the area below the Gaussian curves to the left or right of a decision criterion. Varying this criterion describes the ROC curve.
Manipulate[
 ParametricPlot[{CDF[NormalDistribution[4, \[Sigma]], c], 
                 CDF[NormalDistribution[0, 3], c]
                }, {c, -10, 10}, 
                Frame -> True, 
                Axes -> None, PlotRange -> {{0, 1}, {0, 1}}, 
                Epilog -> Line[{{0, 0}, {1, 1}}]], 
 {{\[Sigma], 3}, 0.1, 10, Appearance -> "Labeled"}]

This is with equal standard deviations:

This is with rather distinct ones:

or with a few more parameters to play with:
Manipulate[
 ParametricPlot[{CDF[NormalDistribution[\[Mu]1, \[Sigma]1], c], 
   CDF[NormalDistribution[\[Mu]2, \[Sigma]2], c]}, {c, -100, 100}, 
  Frame -> True, Axes -> None, PlotRange -> {{0, 1}, {0, 1}}, 
  Epilog -> Line[{{0, 0}, {1, 1}}]], {{\[Mu]1, 0}, 0, 10, 
  Appearance -> "Labeled"},
 {{\[Sigma]1, 4}, 0.1, 20, Appearance -> "Labeled"},
 {{\[Mu]2, 5}, 0, 10, Appearance -> "Labeled"},
 {{\[Sigma]2, 4}, 0.1, 20, Appearance -> "Labeled"}]


A: Having a string of negative instances in the part of the curve with high FPR can create this kind of a curve. This is ok as long as you are using the right algorithm for generating the ROC curve. 
The condition where you have a set of 2m points half of which are positive and half are negative-all having exactly the same score for your model is tricky. If while sorting the points based on the score (standard procedure in plotting ROC) all the negative examples are encountered first, this will cause your ROC curve to stay flat and move to the right.This paper talks about how to take care of such issues:
Fawcett| Plotting ROC curves
