I have 4.4 million observations, 160 binary features, and a binary response. Using rpart on Windows (64-bit, with the 64-bit R v2.13.0 build), I run out of memory on a machine with 64GB RAM. My memory limit seems reasonable:
> memory.limit()  61440
But on running rpart, I still receive this error after watching memory usage spike:
> mytree = rpart(formula=fmla,data=mydata,method="class",xval=0) Error in rpart(formula = fmla, data = mydata, method = "class", : cannot allocate memory block of size 5.0 Gb
I'm hoping to avoid sampling from my population. I've tried using logical and factor representations of my data. I've also tried
tree and looked at
party packages, none of which appear to offer more efficient multi-class CART implementations. Can anyone suggest an alternate package or different run parameters?
Barring that, I'm looking for a package to streamline running it through rpart in pieces. I could run rpart on subsets of features and choose the best split from all my runs at each node, but reassembling the results into an rpart object for plotting and pruning sounds like a chore. Can anyone suggest a cleaner solution or a pre-written package for this?