Cforest Runs out of RAM when running 'predict' function

Question

I am trying to run the cforest function from the party package in R (or caret, but both have yielded me the same issue). I started with a dataset of 50000+ observations, with 1 binary response variable and 4 independent variables (2 characters with 6 and 8 categories respectively, and 2 continuous). I converted the characters to binary variables (1 hot) and now have 16 predictors (with 14 being binary) and 2 continuous.

Next I ran through a slew of predictive methods including logit, rpart, svm, nnet, etc. My best prediction error came from the function randomForest with ntree=2000, mtry=16 from the randomForest package. I though it best to test ctree (which outperformed rpart) and finally cforest as I've read it is often slightly more accurate than randomForest.

Up to this point I had no trouble with the predict function for any of my tests.

When I ran:

mcf<-cforest(y~x1+x2+x3+x4+x5+x6+x7+x8+x9+x9+x10+x11+x12+x13+x14+x15+x16, data=train1)

(I left all defaults the same, i.e. mtry=6, ntree=500)

R took about 30 minutes to compute(I'm well aware the task is very computationally expensive; even more so than randomForest), but came out with a model smaller in size than `randomForest' and RAM usage never exceeded ~40%

However when I ran:

pmcf<-predict(mcf), pmcf<-predict(mcf, newdata=train1), pmcf<-predict(mcf, newdata=train1, type='response'), and pmcf<-predict(mcf, type='response')

each time R took over an over an hour and then returned an error message saying:

error: Cannot allocate vector of size 127kb

(those predictions were all separate attempts by the way. I ran it all those different ways just to try and make sure I wasn't making a silly error in the arguments)

Upon further inspection I watched my memory usage as the function ran, and it kept climbing from 20% to about 90% until it finally returned the error.

It seems only the predict function is giving me fits when I call my model, and only for predict.cforest.

About my machine: I'm running windows 10 Home, 64-bit, on a Lenovo ThinkPad p50 (about 1.5 years old) with Intel Quad Core i7 Processor, 4gb NVIDIA Quadro M1000M GPU, 16GB of DDR4 Memory (with 15.8GB usable). I also have a 512gb SSD but I thought I recall reading that R keeps everything in memory anyway. (additionally I had no other program opens while running predict).

A few things I've looked into: I am running rtudio 64-bit, so that is not the limiting factor. I've checked memory.limit() and it is maxed out at just over 16000MB, so that also isn't it. I tried adjusting the hyperparamters in cforest to less ntrees and a low mtry but predict still didn't work. (Also, lowering these parameters too low pretty much defeats the purpose of me running cforest as a way to beat randomForest). I've given the 'package:party' PDF a thorough read but still can't find what maybe wrong (although admittedly I am new to ML). Finally, I know cforest(form~.) formula argument isn't preferred, as it slows down computation and uses more memory, but cforest doesn't have a cforest(x,y) argument. I tried running it that way (cforest(x,y)) in caret but got the same issues.

So I'm really just wondering if this predict.cforest was too computationally expensive for my computer? I was under the impression people have done a lot more with a lot less as far as computing power goes (my machine has a lot). If this is the case is there a remedy? Maybe attempt it with a smaller dataset from the training set?

Could it be the dimensionality? Again, I feel I've seen lesser machines handle 20 and 30 variables no problem. Perhaps I should dump the 1 hot encoding?

And finally, I know coding questions aren't allowed, but could there be an obvious mistake in what I've shown that is yielding me a useless cforest model, which in turn, is failing to predict when I call it? I've used cforest with success before so I'm not sure why it won't predict now unless maybe there is something wrong with the actual model that I produced when creating the cforest model initially.

I've included a photo of the data below. 50,000+ observations that look just like that, I've checked that they're all coded correctly as binary.

I tried to be thorough, and not include coding questions, but if you need anymore information just let me know. Sorry the post is so long, I just wanted to try to be clear.

Additionally, if you feel the question is of topic, I have no problem removing or revising it, just let me know in the comments because I would prefer not to get banned from asking questions. Obviously, I felt this was a legitimate question about memory usage in R and model building, not a general code question that wastes space and time; otherwise I wouldn't have asked.

With 1 binary response variable read as.factor

Answer 1

predict() is based on nearest neighbor weights. The weight matrix is NROW(data) x NROW(newdata) and this is quite big in your case.

You can simply loop over chunks of newdata in predict.

partykit::cforest can be used with binning (ie, looking at only a small number of possible split points instead of NROW(data) in the worst case), see the nmax argument to partykit::ctree_control

Torsten

Answer 2

Having run into the same issue, I wrote a script that works across a large test set in a piecewise manner, but returns the aggregate prediction object just like predict(). Just like Torsten replied above, it loops over chunks over newdata, where the split argument determines the number of chunks.

Below;

pwpredict = function(model, test, split) {

require(data.table)

total_len = nrow(test)
chunk_size = floor(nrow(test) / split)
vals = c(0, sapply(1:split, function(x) x*chunk_size))

pred = data.table()

for (i in 1:split){

   rows = c(1+vals[i], vals[i+1])
   new_preds = predict(model, type = 'prob', test[rows[1]:rows[2],])
   pred = rbindlist(list(pred, new_preds))

}

if (max(vals) < total_len) {

   diff = total_len - max(vals)
   index = tail(1:total_len, diff)
   pred = rbindlist(list(pred, predict(model, type = 'prob', test[index,])))

}

gc()
return(data.table(pred))

}