1
$\begingroup$

I know that the random forest algorithm works by generating a set of decision trees with a subset of features.

Say I was using random forest as a classification algorithm looking at someone's data usage out of a data allowance.

One decision within one of the trees may be if data_usage > 500 branch right else branch left.

My question is: does a decision tree ever make a decision based on two variables at one split?

For example if data_usage > data_allowance branch right else branch left. Or would it be better to encode a new feature data_utilisation as a % usage of their data allowance? So if data_usage > 100 branch right else branch left?

Hope this makes sense,

Thanks.

$\endgroup$
  • $\begingroup$ My understanding is that what you're describing does not happen in decision trees (at least the standard implementations) - it would take place as successive splits instead of one split. But would this differences in one vs two splits matter very much? $\endgroup$ – mkt Apr 5 at 13:33
  • $\begingroup$ If it took place in successive splits then it would need a branch for each data allowance and then I guess the classifier would work see a large gain in information when the next feature was data usage with the value being the same as the allowance value. So successive splits makes sense it this case. However this would also require both features in the same subset of features which might not always happen. $\endgroup$ – Colossal Harry Apr 5 at 13:46
0
$\begingroup$

does a decision tree ever make a decision based on two variables at one split?

No, not in standard decision tree implementations. However, you are correct that you could "featurize" the inputs first.

If you do that, you might want to take care to mitigate feature "redundancy", however, I don't have theoretical justification for this claim. In this case, the three features (data_usage / data_allowance, data_usage, data_allowance) are redundant, because the former is a function of the latter two (however, they're not "redundant" when data_usage=0, as all nonzero values of data_allowance yield equivalent values of data_usage / data_allowance).

$\endgroup$
  • $\begingroup$ Thank you for your response. The other issue is what to do if someone has 0 data_allowance as that would imply infinite data_utilisation for all nonzero data_usage values. Do you have any recommendations for what to do in this scenario? What would be the best way to featurize usage and allowance into a model? $\endgroup$ – Colossal Harry Apr 5 at 14:54
  • $\begingroup$ In that case, you should just set it to something that indicates it's invalid. Since all are nonnegative numbers, this is easy. Setting it to -1 would work. In python pseudocode: data_utilization = data_usage/data_allowance if data_allowance > 0 else -1. $\endgroup$ – eqzx Apr 5 at 14:57
0
$\begingroup$

First, when you say random forests grow trees of limited depth you are incorrect. Random forests typically grow trees of full depth since overfitting is mitigated by the subset of features used and the bootstrapping of data. Second, random forest does not consider x1>x2 as a split. As you suggest, you would have to construct this feature.

$\endgroup$
  • $\begingroup$ Sorry I should have been more clear, random forests can be set to grow to a limited depth but this does not have to be the case. $\endgroup$ – Colossal Harry Apr 5 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.