Mann-Whitney U on overlapping data sets I am testing two different predictive classifiers to determine which one is more accurate. Each classifier gives a predictive value on a continuous scale from -10 (not present) to 10 (present). After applying them to the classifiers to the data I take the top K elements from each set and get a human expert to order the set of 2K elements. Now I can apply the Mann-Whitney U test to determine if the order shows a significant difference between the classifiers.
The problem is that because I am applying the classifiers to the same data sets I sometimes have the same element in the top K of each set. This makes sense as they are trying to classify the same thing. It doesn't happen often, but it does happen. The example below shows a top 5 list with one element common in both lists. If I ignore Object D (because it is common to both lists) I could add a new element to each list to make up for the one I lose.

I have considered treating Object D as a tie between the two data sets. With that in mind, I could use the M-W U test, but I'm not sure if this is a good solution. In most cases I have 1 or 2 collisions, but the ties reduce my the margin for a statistically significant result.
Here are the big questions
1. Does ignoring the common element invalidate the test?
2. Should I include the common element and just increase the sample size to make up the difference?
 A: I've never been good at theoretical statistics, so I did a small simulation under the Null hypothesis, i.e. both classifiers equally good, and checked the type I error. I've tweaked the parameters a bit so that we would see results in reasonable time, i.e. higher likelihood of ties.
N = 2000 #Total of objects scored
n = 80 # The total of objects ranked by the human
Nsim = 10000 #Number of Monte-Carlo draws
set.seed(4568)
commonPop = seq_len(N) #A hypothetical population
weights = commonPop^3/sum(commonPop^3) # The weights, increasing with     outcome
groups = factor(rep(c("Group1","Group2"), each = n))
resList = sapply(seq_len(Nsim), function(x){
  Sample = c(sapply(1:2, function(y){sample(commonPop, size = n, replace =FALSE, prob = weights)})) # Smple with the same weights: simulation under the null
  keepTies = wilcox.test(Sample ~ groups)$p.value #WMW with the ties
  tiesID = Sample %in% as.numeric(names(table(Sample)[table(Sample)!=1]))
  dropTies = if(!any(tiesID)) keepTies else wilcox.test(Sample[!tiesID] ~ groups[!tiesID])$p.value #WMW discarding the ties
  c("keepTies" = keepTies, "dropTies" = dropTies)
})
unifExp = (seq_len(Nsim)-0.5)/Nsim
qqplot(resList["keepTies",], unifExp)
mean(resList["keepTies",]<0.05)

0.0403
qqplot(resList["dropTies",], unifExp)
mean(resList["dropTies",]<0.05)          

0.418
You can tweak the code further if you like, also looking at the power, but I couldn't find an immediate problem with discarding the ties in terms of type I error.
