0
$\begingroup$

I am trying to implement a multi-criteria classifier algorithm. It uses 6 criteria that output 1 or 0 if a specific signal is detected in the input data or not.

Then it computes a weighted average of these criteria and compares it with a threshold value. If is greater than the threshold, it outputs 1; otherwise, it outputs 0.

The weights are:

[0.04761905, 0.0952381 , 0.14285714, 0.19047619, 0.28571429, 0.23809524]

The problem is that the ROC curve for the algorithm contains a loop, which, as my understanding goes, should not appear in a ROC plot. As far as I know, a ROC curve increases monotonically.

enter image description here

The image shows the ROC curves for each criterion (1 to 6) and for the algorithm described (7).

For generating the plots, I calculate the results from all criteria (confusion matrix => TPR and FPR) for each ROC point corresponding to a threshold value, using all data sets. After changing the threshold value, I repeat the calculations, until the desired number of ROC points is reached.

If it has any significance, I mention that the threshold ranges of the criteria are different (e.g., 0-100, 0-1 or 0-20), but they are iterated in proportionally equal steps (e.g., if 10 ROC points are calculated, then the iteration steps are 100/10, 1/10 or 20/10). Also, I use an unequal number of data sets for the two cases (signal present, signal absent).

Why is the loop appearing in the plot? Am I misunderstanding how to properly implement this multi-criteria algorithm? Am I calculating the ROC curve incorrectly?

$\endgroup$

closed as off-topic by whuber Mar 19 at 14:16

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – whuber
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ You're right that a ROC curve is monotonic, but a typical ROC curve is constructed using a much simpler procedure that doesn't seem closely related to what you present here. Where did you find this procedure? Does it have a name? Can you edit your post to include an article explaining how this procedure works? $\endgroup$ – Sycorax Mar 16 at 22:02
  • 1
    $\begingroup$ If the "loop" you refer to is the non-monotonicity of the dashed line, that looks to me like a bug in the graphing code. That suggests you should offer more details of how you computed and plotted the points in your plot. $\endgroup$ – whuber Mar 16 at 22:12
  • $\begingroup$ @Sycorax I found and corrected the problem and the plot is OK now. Basically, I have a classifier algorithm (based on 6 criteria) and for calculating the ROC curve I determine the TPR and FPR (using all data sets available) corresponding to each decision threshold value (ROC point). So for, e.g. 10 ROC points, I run the algorithm 10 times on all data sets to get the TPR and FPR. This is my understanding on how the ROC curve is generated. Please tell me if you know a simpler/more efficient procedure. $\endgroup$ – Cristian M Mar 18 at 12:33
  • 1
    $\begingroup$ If this question was based on an idiosyncratic error, then it sounds like deletion may be in order. $\endgroup$ – beta1_equals_beta2 Mar 18 at 12:35
0
$\begingroup$

I figured it out. By mistake, when calculating the ROC values for the algorithms I forgot to change the threshold. Instead, I was calculating using a fixed threshold and taking into account the variable thresholds of the 6 criteria.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.