1
$\begingroup$

I am testing two different predictive classifiers to determine which one is more accurate. Each classifier gives a predictive value on a continuous scale from -10 (not present) to 10 (present). After applying them to the classifiers to the data I take the top K elements from each set and get a human expert to order the set of 2K elements. Now I can apply the Mann-Whitney U test to determine if the order shows a significant difference between the classifiers.

The problem is that because I am applying the classifiers to the same data sets I sometimes have the same element in the top K of each set. This makes sense as they are trying to classify the same thing. It doesn't happen often, but it does happen. The example below shows a top 5 list with one element common in both lists. If I ignore Object D (because it is common to both lists) I could add a new element to each list to make up for the one I lose.

enter image description here

I have considered treating Object D as a tie between the two data sets. With that in mind, I could use the M-W U test, but I'm not sure if this is a good solution. In most cases I have 1 or 2 collisions, but the ties reduce my the margin for a statistically significant result.

Here are the big questions 1. Does ignoring the common element invalidate the test? 2. Should I include the common element and just increase the sample size to make up the difference?

Improve this question
$\endgroup$
  • $\begingroup$ Please clarify: Are you comparing the orderings of the two classifiers with each other or with the human expert? How can you compare orderings if they do not contain the same elements. I'm a bit puzzled here. $\endgroup$ – Knarpie Sep 8 '17 at 8:45
  • $\begingroup$ As I understand it, the point of the M-W U test is that if you combine data from two sets and then order or rank them you can compare the two rankings with U. If one set is consistently higher-ranked than the other the difference in the rankings can be shown to be statistically significant. My human expert is only providing the ranking. $\endgroup$ – BSD Sep 8 '17 at 15:06
  • $\begingroup$ I see, so the order in which the two classifiers report the objects is of no consequence? This is a bit of a waste of information, I think you need something more complicated than the WMW, from the area of rater's agreement. Btw, why do you use the term classifiers if the predictions are continuous? You could indeed discard these types of objects, but if your methods get better I think it will occur more frequently. $\endgroup$ – Knarpie Sep 8 '17 at 15:15
  • $\begingroup$ The classifiers each predict the probability of the object having or not having that characteristic. You are right that if they were both reasonably good there would be more colisions. Right now I'm seeing very few because the lists are not very similar in ranking. I conducted a spearman rho test and found that in most cases the ordering is significantly different. For right now M-W U works. As for the rater's agreement, the problem is the sample size. I can ask a human to rank 10 or 20, but not 2000. The M-W U gives me a short cut to compare the rankings. $\endgroup$ – BSD Sep 8 '17 at 15:20
  • $\begingroup$ But you said they return values between -10 and 10? How do they relate to probabilities? If you are satisfied, so am I, so I will leave it here then. $\endgroup$ – Knarpie Sep 8 '17 at 15:43
1
$\begingroup$

I've never been good at theoretical statistics, so I did a small simulation under the Null hypothesis, i.e. both classifiers equally good, and checked the type I error. I've tweaked the parameters a bit so that we would see results in reasonable time, i.e. higher likelihood of ties.

N = 2000 #Total of objects scored
n = 80 # The total of objects ranked by the human
Nsim = 10000 #Number of Monte-Carlo draws
set.seed(4568)
commonPop = seq_len(N) #A hypothetical population
weights = commonPop^3/sum(commonPop^3) # The weights, increasing with     outcome
groups = factor(rep(c("Group1","Group2"), each = n))
resList = sapply(seq_len(Nsim), function(x){
  Sample = c(sapply(1:2, function(y){sample(commonPop, size = n, replace =FALSE, prob = weights)})) # Smple with the same weights: simulation under the null
  keepTies = wilcox.test(Sample ~ groups)$p.value #WMW with the ties
  tiesID = Sample %in% as.numeric(names(table(Sample)[table(Sample)!=1]))
  dropTies = if(!any(tiesID)) keepTies else wilcox.test(Sample[!tiesID] ~ groups[!tiesID])$p.value #WMW discarding the ties
  c("keepTies" = keepTies, "dropTies" = dropTies)
})
unifExp = (seq_len(Nsim)-0.5)/Nsim
qqplot(resList["keepTies",], unifExp)
mean(resList["keepTies",]<0.05)

0.0403

qqplot(resList["dropTies",], unifExp)
mean(resList["dropTies",]<0.05)

0.418

You can tweak the code further if you like, also looking at the power, but I couldn't find an immediate problem with discarding the ties in terms of type I error.

Improve this answer
$\endgroup$
  • $\begingroup$ Knarpie, that is great! I was thinking that the next step for me would be a monte carlo simulation to see what kind of differences came up and you helped enormously by doing it for me. Thanks, $\endgroup$ – BSD Sep 11 '17 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.