Memory problems using pROC library in R to estimate confidence interval for area under the curve

Question

I am trying to use the R pROC package to estimate the confidence interval for the area under the curve (AUC). I am running into a memory limitation. My R script looks like the following:

library(pROC)
data <- read.csv("data.csv",header=TRUE)
rocobj <- roc(data$observed, data$predicted)
ci(rocobj)

I have modified my 64-bit R v3.0.2 shortcut with the following

"C:\Program Files\R\R-3.0.2\bin\x64\Rgui.exe" --max-mem-size=8192M

still, when I run the ci(rocobj) command, I get the following message:

Error: cannot allocate vector of size 256 Kb
In addition: There were 11 warnings (use warnings() to see them)
warnings()
Warning messages:
1: In m * n : NAs produced by integer overflow
2: In lapply(X = X, FUN = FUN, ...) :
  Reached total allocation of 8192Mb: see help(memory.size)
3: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
4: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
5: In lapply(X = X, FUN = FUN, ...) :
  Reached total allocation of 8192Mb: see help(memory.size)
6: In lapply(X = X, FUN = FUN, ...) :
  Reached total allocation of 8192Mb: see help(memory.size)
7: In unlist(x, recursive = FALSE) :
  Reached total allocation of 8192Mb: see help(memory.size)
8: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
9: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
10: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
11: In unique.default(unlist(lapply(x, length))) :
  Reached total allocation of 8192Mb: see help(memory.size)
Warning messages:
1: Reached total allocation of 8192Mb: see help(memory.size) 
2: Reached total allocation of 8192Mb: see help(memory.size) 

my laptop is windows 7 64-bit with 16 GB of RAM. But I am running other stuff, and so I can only really use 8 GB (which is already pushing it). I tried to limit the data size and it works.

n <- 10000
o <- head(data$observed,n=n)
    p <- head(data$predicted,n=n)
rocobj <- roc(o, p)
ci(rocobj)

But obviously, I want more than 10,000 data points. Any help on alternative packages or how to resolve this memory issue is appreciated. By the way, I have 165,000 predicted points.

> head(data)
  observed   predicted
1        0 0.005101049
2        0 0.042003527
3        0 0.045466933
4        0 0.017828838
5        0 0.092385962
6        0 0.053715844
> summary(data)
    observed        predicted        
 Min.   :0.0000   Min.   :        0  
 1st Qu.:0.0000   1st Qu.:        0  
 Median :0.0000   Median :        0  
 Mean   :0.1714   Mean   :    11071  
 3rd Qu.:0.0000   3rd Qu.:        3  
 Max.   :1.0000   Max.   :277494082  
> 

using the caTools package, the script to compute the confidence interval of AUC is as follows.

r <- 1000
s <- 120000
bAuroc <- replicate(r, {
 m <- sample(s, replace=TRUE)
 colAUC(data$predicted[m], data$observed[m])
});
quantile(bAuroc, c(0.05, 0.95))

note that changing the value of r (e.g. r = 1000, 2000, 3000, ...) does not really produce any problems. however, changing the value of s (e.g. s = 100000, 110000, 120000, 130000) does produce problems. it seems that s = 120000 is the upper limit, and going anything above that produces the integer overflow problem.

the solution for my problem is as follows. note that there is no more limit on the number of samples, s < nrow(data) but s = nrow(data), and also, i had to specify alg="ROC".

r <- 1000
s <- nrow(data)
bAuroc <- replicate(r, {
 m <- sample(s, replace=TRUE)
 colAUC(data$predicted[m], data$observed[m], alg="ROC")
});
quantile(bAuroc, c(0.05, 0.95))

Answer 1

I believe the colAUC from caTools package is the most efficient AUROC implementation for R, and it should manage to handle your problem. Still, it does not generate CI, so you will have to implement it on your own, like

bAuroc<-replicate(1000,{
 sample(nrow(data),replace=TRUE)->m;
 colAUC(data$predicted[m],data$observed[m])
 });
quantile(bAuroc,c(0.05,0.95))

Answer 2

Two solutions:

1. Use a bootstrap test with ci(rocobj, method="bootstrap"). The notation you used performed a DeLong's test, which was (see next point) extremely poorly optimized in pROC and tried to allocate a NxM matrix (with N = number of controls, and M = number of cases in your ROC curve).
2. Update pROC to the latest version (1.7). The algorithm behind DeLong's test has been significantly improved and doesn't allocate this matrix any more. Type install.packages("pROC") to get the latest version (check that you have it with packageDescription("pROC")).