2
$\begingroup$

I am constructing a random forest model to predict a dependent variable Y. Two features are X_1 and X_2. However, from domain knowledge, I suspect that X_1 and X_2 don't strongly predict Y on their own, and the feature that really matters is the ratio X_1/X_2. (There are many other features X_i as well.)

For example, suppose that Y is the fraction of people in a given city that drive to work, while X_1 represents the number of cars in the city and X_2 represents the total population of the city. Clearly X_1 and X_2 would independently have at best a weak relationship with Y, while the ratio X_1/X_2 would strongly predict Y.

Can I expect a random forest model to automatically detect that Y scales primarily with X_1/X_2 rather than with either variable independently? Or is it important to use domain knowledge to explicitly create the more relevant feature X_3 = X_1/X_2 before training?

(Maybe another way to say it is that the features X_1 and X_2 are highly correlated, while neither feature is highly correlated with X_1/X_2. I am not a statistical sophisticate.)

I am interested in the answer for both random forest regressors (as in this setup) and classifiers, if they differ.

$\endgroup$
  • 1
    $\begingroup$ Just remember that each split happens on single variable. So by providing the relationship between them, you may decrease noise and increase accuracy with a smaller tree. $\endgroup$ – Fernando Mar 2 '17 at 18:48
  • $\begingroup$ Sounds like an answer? $\endgroup$ – abeboparebop Mar 3 '17 at 9:20
2
$\begingroup$

Just remember that each split happens on single variable. So by providing the relationship between them, you may decrease noise and increase accuracy with a smaller tree.

To see this, try to model a straight line both ways. The image below is taken from the book Pattern Classification, by Richard O. Duda et al:

Keep in mind that this is a very specific example, you need to understand your data. Most of the time it means: test all your ideas.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.