I am trying to implement a multi-criteria classifier algorithm. It uses 6 criteria that output 1 or 0 if a specific signal is detected in the input data or not.
Then it computes a weighted average of these criteria and compares it with a threshold value. If is greater than the threshold, it outputs 1; otherwise, it outputs 0.
The weights are:
[0.04761905, 0.0952381 , 0.14285714, 0.19047619, 0.28571429, 0.23809524]
The problem is that the ROC curve for the algorithm contains a loop, which, as my understanding goes, should not appear in a ROC plot. As far as I know, a ROC curve increases monotonically.
The image shows the ROC curves for each criterion (1 to 6) and for the algorithm described (7).
For generating the plots, I calculate the results from all criteria (confusion matrix => TPR and FPR) for each ROC point corresponding to a threshold value, using all data sets. After changing the threshold value, I repeat the calculations, until the desired number of ROC points is reached.
If it has any significance, I mention that the threshold ranges of the criteria are different (e.g., 0-100, 0-1 or 0-20), but they are iterated in proportionally equal steps (e.g., if 10 ROC points are calculated, then the iteration steps are 100/10, 1/10 or 20/10). Also, I use an unequal number of data sets for the two cases (signal present, signal absent).
Why is the loop appearing in the plot? Am I misunderstanding how to properly implement this multi-criteria algorithm? Am I calculating the ROC curve incorrectly?