# Random Forest in a Big Data setting

I have a dataset with 5,818,446 lines and 51 columns, where 50 of them are predictors. My response is quantitative, so I am interested in a regression model. I am trying to fit a random forest to my data using the caret package. However, I do not have enough RAM to do it.

I've been looking for solutions to my problem. Besides having a more powerful computer, it seems I can make use of bagging to solve my problem. Therefore, my idea is as follows:

1. Create both train and test partitions from my original dataset

2. Sample with replacement a small part of my train dataset into R (let's say 1% of it, i.e., 58,185 lines)

3. Fit a random forest to this small part of data

4. Save the model result

5. Repeat steps 2-4 1,000 times

6. Combine these 1,000 models obtained from steps 2-5

However, random forest itself uses bagging to fit the model to the data and thus I am not sure if my approach is correct. Therefore, I have some questions for you:

i) Is my approach correct? I mean, since I do not have enough RAM in my system, is it correct to fit many different random forest models to different chunks of data and combine them after?

ii) Assuming my approach is correct, 1% of data is a good rule of thumb for my sample size? Even with 1% of data, I still have $n \gg p$.

iii) Assuming my approach is correct, is there a number of replications for models I should be using? I thought of 1,000 because of reasons.

• Since bagging is a method of generating each decision tree in a random forest, how about you just do a single DT per 1% and then combine the trees? Voila, a home baked random forest. Then you can build 1000 or more trees and do random combinations of trees to see what works best on your test data; although then you are fitting to your test data, so maybe just take them all and run with it. May 26 '18 at 17:05
• Ooo! Or combine them all with a fully connected output layer and then train that layer with l1 regularization, which typically drops insignificant components' weights to very nearly zero, so you can see by inspection which trees you should keep. May 26 '18 at 17:09
• @Engineero The method you outline is generally known as "random kitchen sinks": fit a large number of weak learners for form a basis representation of the original features (even decision stumps) and then weight terminal nodes, perhaps with some regularization of node weights. people.eecs.berkeley.edu/~brecht/kitchensinks.html
– Sycorax
May 26 '18 at 17:11
• 1/ I don't have any definitive knowledge, so I'm not posting an answer. But this method ought to be fine. Note that the number of samples in each "batch" implies an upper bound on the depth of each tree, and hence the ability of each tree to fit the data, so you might consider tuning over 0.5%, 1%, 2%, 4% (notionally) ... of the data when generating each tree. The number of trees to use is purely a question of at what point the model's loss "plateaus;" this choice does not need to be tuned as the expected loss is a monotonic decreasing function of $T$ for continuous proper scoring rules.
– Sycorax
May 26 '18 at 17:20
This does not address your specific questions, but the motivation behind them. The bigRF package may solve your problem: