3
$\begingroup$

I am trying to use the R pROC package to estimate the confidence interval for the area under the curve (AUC). I am running into a memory limitation. My R script looks like the following:

library(pROC)
data <- read.csv("data.csv",header=TRUE)
rocobj <- roc(data$observed, data$predicted)
ci(rocobj)

I have modified my 64-bit R v3.0.2 shortcut with the following

"C:\Program Files\R\R-3.0.2\bin\x64\Rgui.exe" --max-mem-size=8192M

still, when I run the ci(rocobj) command, I get the following message:

Error: cannot allocate vector of size 256 Kb
In addition: There were 11 warnings (use warnings() to see them)
warnings()
Warning messages:
1: In m * n : NAs produced by integer overflow
2: In lapply(X = X, FUN = FUN, ...) :
  Reached total allocation of 8192Mb: see help(memory.size)
3: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
4: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
5: In lapply(X = X, FUN = FUN, ...) :
  Reached total allocation of 8192Mb: see help(memory.size)
6: In lapply(X = X, FUN = FUN, ...) :
  Reached total allocation of 8192Mb: see help(memory.size)
7: In unlist(x, recursive = FALSE) :
  Reached total allocation of 8192Mb: see help(memory.size)
8: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
9: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
10: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
11: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
Warning messages:
1: Reached total allocation of 8192Mb: see help(memory.size) 
2: Reached total allocation of 8192Mb: see help(memory.size) 

my laptop is windows 7 64-bit with 16 GB of RAM. But I am running other stuff, and so I can only really use 8 GB (which is already pushing it). I tried to limit the data size and it works.

n <- 10000
o <- head(data$observed,n=n)
    p <- head(data$predicted,n=n)
rocobj <- roc(o, p)
ci(rocobj)

But obviously, I want more than 10,000 data points. Any help on alternative packages or how to resolve this memory issue is appreciated. By the way, I have 165,000 predicted points.

> head(data)
  observed   predicted
1        0 0.005101049
2        0 0.042003527
3        0 0.045466933
4        0 0.017828838
5        0 0.092385962
6        0 0.053715844
> summary(data)
    observed        predicted        
 Min.   :0.0000   Min.   :        0  
 1st Qu.:0.0000   1st Qu.:        0  
 Median :0.0000   Median :        0  
 Mean   :0.1714   Mean   :    11071  
 3rd Qu.:0.0000   3rd Qu.:        3  
 Max.   :1.0000   Max.   :277494082  
> 

using the caTools package, the script to compute the confidence interval of AUC is as follows.

r <- 1000
s <- 120000
bAuroc <- replicate(r, {
 m <- sample(s, replace=TRUE)
 colAUC(data$predicted[m], data$observed[m])
});
quantile(bAuroc, c(0.05, 0.95))

note that changing the value of r (e.g. r = 1000, 2000, 3000, ...) does not really produce any problems. however, changing the value of s (e.g. s = 100000, 110000, 120000, 130000) does produce problems. it seems that s = 120000 is the upper limit, and going anything above that produces the integer overflow problem.

the solution for my problem is as follows. note that there is no more limit on the number of samples, s < nrow(data) but s = nrow(data), and also, i had to specify alg="ROC".

r <- 1000
s <- nrow(data)
bAuroc <- replicate(r, {
 m <- sample(s, replace=TRUE)
 colAUC(data$predicted[m], data$observed[m], alg="ROC")
});
quantile(bAuroc, c(0.05, 0.95))
$\endgroup$
3
$\begingroup$

I believe the colAUC from caTools package is the most efficient AUROC implementation for R, and it should manage to handle your problem. Still, it does not generate CI, so you will have to implement it on your own, like

bAuroc<-replicate(1000,{
 sample(nrow(data),replace=TRUE)->m;
 colAUC(data$predicted[m],data$observed[m])
 });
quantile(bAuroc,c(0.05,0.95))
$\endgroup$
  • $\begingroup$ could you clarify what you mean by "CA"? what does CA stand for? $\endgroup$ – Jane Wayne Nov 20 '13 at 21:25
  • $\begingroup$ Confidence Interval, that was a typo. $\endgroup$ – user88 Nov 20 '13 at 22:40
  • $\begingroup$ when i run the replicate command, i get this message: There were 50 or more warnings (use warnings() to see the first 50). the warning messages all look like this: 1: In n1 * n2 : NAs produced by integer overflow. any ideas? $\endgroup$ – Jane Wayne Nov 20 '13 at 22:57
  • $\begingroup$ Strange; can you edit your question and post head(data) and summary(data)? $\endgroup$ – user88 Nov 21 '13 at 7:08
  • $\begingroup$ could you comment on the revision of your script above? what is the difference, if any, between, sample(nrow(data),replace=TRUE) and sample(120000,replace=TRUE)? $\endgroup$ – Jane Wayne Nov 21 '13 at 12:51
2
$\begingroup$

Two solutions:

  1. Use a bootstrap test with ci(rocobj, method="bootstrap"). The notation you used performed a DeLong's test, which was (see next point) extremely poorly optimized in pROC and tried to allocate a NxM matrix (with N = number of controls, and M = number of cases in your ROC curve).
  2. Update pROC to the latest version (1.7). The algorithm behind DeLong's test has been significantly improved and doesn't allocate this matrix any more. Type install.packages("pROC") to get the latest version (check that you have it with packageDescription("pROC")).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.