0
$\begingroup$

I have a big data set (with more than 9,000,000 rows) with 7 features and 1 label. The label is ordinal data. I would like to run a random forest regression. I'm fairly new to random forests so I have a few questions on parameter settings in my case:

  1. How many trees should I run based on my sample size?

  2. Shall I set a limit on the depth of my tree?

  3. How could I tell if a feature is important or not? I know there is a feature ranking. In my test run, none of the feature importance values is 0, but I wonder if there could be an importance value threshold to filter out the feature?

  4. I saw people visualize trees as part of their analysis output. However, if I end up training many trees (e.g., 1000 trees), shall I just randomly visualize 1 tree? I don't really get the point of visualizing one of the many trees as I doubt if that could really help people understand my result. Could anyone help rationalize the importance of visualizing trees in the random forests?

  5. I also wonder if XGBoost will be a better option than random forest in my case. Because my label is ordinal data and I will have to treat it as a continuous variable and use regression in random forest. It seems XGBoost can specifically analyze ranking data and it's faster than random forest. But I could be wrong. Any input would be appreciated.

Other relevant info: I'm using Python sklearn for modeling on Google Colab (with 13 GB RAM limit).

Please feel free to answer all or one of the questions. Thank you so much!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I see you have any questions. Q&A sites like this are usually not best suited for such cases, I'd encourage you to ask one question at a thread next time. Also given the number of questions, it'd be probably a good idea to start with one of many great handbooks on machine learning. Nonetheless, I feel like the questions are fairly easy to answer, so let me try.

  1. How many trees should I run based on my sample size?

Sample size and the number of trees are unrelated. With random forest, the general rule is that you use as many trees as you can. With a large sample size, the number would be likely limited by the memory available and the training time that is acceptable for you.

  1. Shall I set a limit on the depth of my tree?

It is a hyperparameter, changing it would affect the results. Usual advice would be: do it and check what happens. Additionally, with shallower trees, computation time and memory use would be smaller.

  1. How could I tell if a feature is important or not? [...]

What for? With only seven features and so much data, you don't really need to do the feature selection.

  1. I saw people visualize trees as part of their analysis output. However, if I end up training many trees (e.g., 1000 trees), shall I just randomly visualize 1 tree? [...]

Visualizing a single tree is pointless. The point of random forest is that one would expect each of the trees to be (randomly) different. Visualizing a single tree tells you nothing about the overall model. There is no single, simple way to visualize random forest.

  1. I also wonder if XGBoost will be a better option than random forest in my case. [...]

It depends. Again, the usual answer would be to try it and which model works better.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Hi, thank you for the quick reply. I have a few follow-up questions. $\endgroup$
    – LCheng
    Commented Sep 17, 2021 at 18:14
  • $\begingroup$ For the number of trees and the depth of trees, I was expecting some “rule of thumb” protocols. I’ve read many other Q&As before I raised my questions and I knew that for the number of trees, the more the better; and for the depth of trees, the shallower the more computationally inexpensive. However, that’s not really getting me anywhere in implementing my algorithms to the data. What I would like to learn more is something like what value would be good to start trying out if you were me, and when would you feel it’s good enough and you would stop trying more values. $\endgroup$
    – LCheng
    Commented Sep 17, 2021 at 18:14
  • $\begingroup$ For the feature importance ranking, my goal is not doing feature selection. I want to understand them from a scientific point of view, so it’s important for me to know if a feature really contributes to explaining the label or if it just performing similarly as an arbitrary variable. Does that make sense? Any suggestion will be appreciated. $\endgroup$
    – LCheng
    Commented Sep 17, 2021 at 18:14
  • $\begingroup$ @Lily if you want to use a rule of thumb, just stick to the defaults the software uses, but this won't work in many cases. As for interpretability, there are whole books on this subject and we have several threads on this subject. $\endgroup$
    – Tim
    Commented Sep 17, 2021 at 18:30
  • $\begingroup$ Hi, I don't think the book talks about random forest interpretation though. There is only one section on decision tree. $\endgroup$
    – LCheng
    Commented Sep 19, 2021 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.