8
$\begingroup$

I have a dataset with 5,818,446 lines and 51 columns, where 50 of them are predictors. My response is quantitative, so I am interested in a regression model. I am trying to fit a random forest to my data using the caret package. However, I do not have enough RAM to do it.

I've been looking for solutions to my problem. Besides having a more powerful computer, it seems I can make use of bagging to solve my problem. Therefore, my idea is as follows:

  1. Create both train and test partitions from my original dataset

  2. Sample with replacement a small part of my train dataset into R (let's say 1% of it, i.e., 58,185 lines)

  3. Fit a random forest to this small part of data

  4. Save the model result

  5. Repeat steps 2-4 1,000 times

  6. Combine these 1,000 models obtained from steps 2-5

However, random forest itself uses bagging to fit the model to the data and thus I am not sure if my approach is correct. Therefore, I have some questions for you:

i) Is my approach correct? I mean, since I do not have enough RAM in my system, is it correct to fit many different random forest models to different chunks of data and combine them after?

ii) Assuming my approach is correct, 1% of data is a good rule of thumb for my sample size? Even with 1% of data, I still have $n \gg p$.

iii) Assuming my approach is correct, is there a number of replications for models I should be using? I thought of 1,000 because of reasons.

Improve this question
$\endgroup$
6
  • $\begingroup$ Since bagging is a method of generating each decision tree in a random forest, how about you just do a single DT per 1% and then combine the trees? Voila, a home baked random forest. Then you can build 1000 or more trees and do random combinations of trees to see what works best on your test data; although then you are fitting to your test data, so maybe just take them all and run with it. $\endgroup$
    – Engineero
    May 26 '18 at 17:05
  • $\begingroup$ Ooo! Or combine them all with a fully connected output layer and then train that layer with l1 regularization, which typically drops insignificant components' weights to very nearly zero, so you can see by inspection which trees you should keep. $\endgroup$
    – Engineero
    May 26 '18 at 17:09
  • $\begingroup$ @Engineero The method you outline is generally known as "random kitchen sinks": fit a large number of weak learners for form a basis representation of the original features (even decision stumps) and then weight terminal nodes, perhaps with some regularization of node weights. people.eecs.berkeley.edu/~brecht/kitchensinks.html $\endgroup$
    – Sycorax ♦
    May 26 '18 at 17:11
  • $\begingroup$ 1/ I don't have any definitive knowledge, so I'm not posting an answer. But this method ought to be fine. Note that the number of samples in each "batch" implies an upper bound on the depth of each tree, and hence the ability of each tree to fit the data, so you might consider tuning over 0.5%, 1%, 2%, 4% (notionally) ... of the data when generating each tree. The number of trees to use is purely a question of at what point the model's loss "plateaus;" this choice does not need to be tuned as the expected loss is a monotonic decreasing function of $T$ for continuous proper scoring rules. $\endgroup$
    – Sycorax ♦
    May 26 '18 at 17:20
  • $\begingroup$ 2/ For more information about the number of trees, see my answer here: stats.stackexchange.com/questions/348245/… $\endgroup$
    – Sycorax ♦
    May 26 '18 at 17:21
3
$\begingroup$

This does not address your specific questions, but the motivation behind them. The bigRF package may solve your problem:

This is an R implementation of Leo Breiman's and Adele Cutler's Random Forest algorithms for classification and regression, with optimizations for performance and for handling of data sets that are too large to be processed in memory.

Also:

For large data sets, the training data, intermediate computations and some outputs (e.g. proximity matrices) may be cached on disk using "big.matrix" objects. This enables random forests to be built on fairly large data sets without hitting RAM limits, which will cause excessive virtual memory swapping by the operating system.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.