I am planning to run random forests to predict a binary outcome. I have a relatively (from my point of view) large dataset, composed of 500,000 units and around 100 features (a mix of continuous, binary and categorical variables). I am planning to use the rf package from the caret library in R.

I used to run random forests on smaller datasets on my personal laptop or on a small AWS-EC2. Any advice on how to run it efficiently in terms of computational power? For instance,

  • if I opt for an AWS-EC2 server, which one should I use?
  • Should I consider using SparkR (a frontend for Spark on R)?
  • Should I consider parallel computing?
  • How much time should I expect the algorithm to obtain a solution?

Thank you so much! :)


Some hints:

  • 500k rows with 100 columns do not impose problems to load and prepare, even on a normal laptop. No need for big data tools like spark. Spark is good in situations with hundreds of millions of rows.
  • Good random forest implementations like ranger (available in caret) are fully parallelized. The more cores, the better.
  • Random forests do not scale too well to large data. Why? Their basic idea is to pool a lot of very deep trees. But growing deep trees eats a lot of resources. Playing with parameters like max.depth and num.trees help to reduce computational time. Still, they are not ideal. In your situation, maybe 20 minutes with ranger on a normal laptop would be sufficient. (A rough guess).
    n <- 500000
    p <- 100
    df <- data.frame(matrix(rnorm(n * p), ncol = p))
    df$y <- factor(sample(0:1, n, TRUE))
    object.size(df) # 400 MB


    fit <- ranger(y ~ ., 
                  data = df, 
                  num.trees = 500,
                  max.depth = 8,
                  probability = TRUE)

enter image description here

With higher max.depth, quite a lot of additional time will be required.

| cite | improve this answer | |
  • $\begingroup$ Thanks so much Michael! More generally what classifier would you suggest in case of large data? $\endgroup$ – Michael De Santa Apr 2 at 20:15
  • $\begingroup$ It depends very much on the situation and the data structure. Furthermore, an appropriate train/validation/test strategy is of major importance. All plays together very closely, so the best algorithm might fail if data is ill-prepared or the train/valid/test strategy is bad. $\endgroup$ – Michael M Apr 4 at 12:42

The answer is already given in the other answer (+1), the dataset you describe is not that big and should not need any specialized software or hardware to handle it. The only thing that I'd add, is that you rather should not use Spark. You can check those benchmarks, Spark "is slower and has a larger memory footprint" and for some versions of Spark "random forests having low prediction accuracy vs the other methods", so basically, the Spark implementation of random forest is poor.

| cite | improve this answer | |
  • 1
    $\begingroup$ Just to add insult to injury here: Spark's MLlib random forests are constrained to max depth of 30. $\endgroup$ – usεr11852 Apr 2 at 20:00
  • 1
    $\begingroup$ well, it's very unusual to have a RF depth of more than 30 (especially with a dataset that small). If you needed depth >30 (image classification/segmentation maybe?) it's usually a sign that RF is a poor method and something else would work better. $\endgroup$ – JDL Apr 3 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.