I have three rankings, each of length 10, where in each ranking people have ranked 10/20 possible patients $$ [P_{i, 1}...P_{i, 10}],\ i=1, 2, 3 $$

by the probability they will get a particular disease. Furthermore, it is known whether each patient does in fact get this disease. Importantly, the overlap between the patients ranked by each method is not null, nor is it complete.

If the sets that these predictions were being performed on were either completely overlapping or not overlapping I would know how to compare if the AUC between them in a statistically rigorous way: pROC, for example, using methods from DeLong and others. But I do not know how to test if the AUC between these rankings, which are partially overlapping and therefore partially correlated (?) are statistically different.

Any help would be great!

Edit: I am completely fine with a non-parametric/simulation based method here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.