I have a dataset where samples are stratified in groups. That is, there are N samples per examination, and M examinations per subject.
I would like to account for this when estimating the correlation of a random variable X to an outcome Y, both defined per sample.
For the AUC, I 1) randomly draw one sample per subject, 2) compute the AUC on the drawn samples, 3) repeat the process 100 times, 4) compute mean and confidence intervals of the AUCs.
For the p-value, I 1) randomly draw one sample per subject, 2) compute the pvalue using Mann-Whitney-U test, 3) repeat the process 100 times, 4) use the Fisher's method to combine the p-values.
Now, the pvalues returned by Fisher's method are ridiculously low. On the order of 1.^(-100). I've read that Fisher methods assumes the p-values to come from independent distributions. I'm not sure if that's true in my case though, they do come from the same data distribution. How can I estimate the statistical significance of the AUC then?