I'm trying out the boot() function for internal validation of a logistic glm model using the AUC (aka c-statistic) as my performance measure. My problem is that depending on the dataset I use, sometimes the function gets "50 or more warnings," all of which are of this type:

Warning messages:
1: In wilcox.test.default(pred[obs == 1], pred[obs == 0],  ... :
  cannot compute exact p-value with ties

Even though the function produces these warnings, it does also produce the relevant bootstrap statistics.

I'm not sure what to make of this situation since I don't even know in what context boot() is calling wilcox.test. The documentation ?boot doesn't mention the use of the Wilcoxon test.

Here is an example that generates these warnings:


AUC = function(data, i) {
    d = data[i,]
    rs1 = glm(ne ~ cost, family=binomial(link="logit"), data=d)
    return(roc.area(d$ne, predict(rs1))$A)

boot(data=nuclear, statistic=AUC, R=600)

And an example that doesn't:


AUC = function(data, i) {
    d = data[i,]
    rs1 = glm(ulcer ~ thickness, family=binomial(link="logit"), data=d)
    return(roc.area(d$ulcer, predict(rs1))$A)

boot(data=melanoma, statistic=AUC, R=600)

(I apologize if this question belongs in Stack Overflow instead of here -- not sure how statistical vs. programmatic the issue is.)


This is not boot that is calling the Wilcoxon test, but verification::roc.area which can be checked by looking at the on-line help:

P-value produced is related to the Mann-Whitney U statistics. The p-value is calculated using the wilcox.test function which automatically handles ties and makes approximations for large values

(or directly by looking at the source code, e.g. verification:::roc.area.)

As you are using bootstrap with replacement, we know that about one third of the sample will not be used in each run, and so you naturally introduce ties and ranks will not be unique anymore, which is what wilcox.test complains about. This function will return an approximate $p$-value (using asymptotic normal distribution).

As as sidenote, you may want to take a look at the rms package which features everything you need to estimate, calibrate and validate GLMs (with bootstrap techniques, among others).

  • $\begingroup$ Ah, makes sense. Thank you very much also for the pointer to the rms package! Wish I'd known about it a long time ago... $\endgroup$ – half-pass Jul 11 '12 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.