I am testing two different predictive classifiers to determine which one is more accurate. Each classifier gives a predictive value on a continuous scale from -10 (not present) to 10 (present). After applying them to the classifiers to the data I take the top K elements from each set and get a human expert to order the set of 2K elements. Now I can apply the Mann-Whitney U test to determine if the order shows a significant difference between the classifiers.
The problem is that because I am applying the classifiers to the same data sets I sometimes have the same element in the top K of each set. This makes sense as they are trying to classify the same thing. It doesn't happen often, but it does happen. The example below shows a top 5 list with one element common in both lists. If I ignore Object D (because it is common to both lists) I could add a new element to each list to make up for the one I lose.
I have considered treating Object D as a tie between the two data sets. With that in mind, I could use the M-W U test, but I'm not sure if this is a good solution. In most cases I have 1 or 2 collisions, but the ties reduce my the margin for a statistically significant result.
Here are the big questions 1. Does ignoring the common element invalidate the test? 2. Should I include the common element and just increase the sample size to make up the difference?