2
$\begingroup$

This is a question about feature engineering for decision trees/random forests. Given two continuous variables X1 and X2, is it informative to construct a third feature variable $\frac{X1}{X2}$, i.e. the ratio between these variables? What about $\frac{X2}{X1}$?

I think it is equivalent to whether a tree/random forest can solve the following problem:

Consider the following dataset

X1  X2  Y
1   3   0.333333333
3   4   0.75
2   3   0.666666667
5   6   0.833333333
6   15  0.4
8   12  0.666666667
...

Here, Y is the ratio of the first two variables. Can a tree/random forest predict Y given X1 and X2?

$\endgroup$
3
$\begingroup$

The CART model is fundamentally piece-wise constant. The ensemble variations (rf and gbm) can have more smoothness but are still piece-wise constant in element. They can approximate and, in a cousin-manner to neural networks, are universal approximators but are not the best non-local generalizers.

The CART basis, like Navier-Stokes for the continuity assumption[1], is only valid when the piece-wise constant model is valid. Like NS when the big-box average is no longer valid over the infinitesimal element, interpolation or extrapolation using CART when the constant mean model is not the appropriate approximation, breaks down.

(It must be late, I'm think of NS in terms of the big-box model. I blame Dr. J, formerly known as 'jaws', one of the most excellent teachers.)

If you train them over a restricted domain, one with a region of the domain missing, for $$ f \left( x_1, x_2 \right) = \frac {x_1} {x_2} $$ and then try to interpolate, the interpolation is going to have issues.

In regions where:
$$ \frac{\partial f\left(x_{1},x_{2} \right)}{\partial x_{1}} = 0 $$ and
$$\frac{\partial f\left(x_{1},x_{2} \right)}{\partial x_{2}} = 0 $$

either by taking very small changes in independent variables, or where $x_{2}$ is really big compared to $x_{1}$ and $x_{1}$ doesn't change sign, then the approximation is going to have less interpolation error.

$\endgroup$
0
$\begingroup$

I would say decision tree may not learn ratios, but random forest (taking the average from many trees) can definitely learn ratios.

In the case you have only two variables, it should be quite easy for random forest to catch Y=X1/X2.

In the case where you have many variables, it will be hard to easy which ratio to use, so if you think X1/X2 is useful and meaningful, you should manually add this feature. Remember that tree models (random forest or XGBoost, they have parameters to random pick features, so if that ratio is really useful, it may not be picked by the algorithm simply due to randomness.)

$\endgroup$
-1
$\begingroup$

Suppose each row corresponds to a person, with $X1$ representing the number of posts they've made to stats.stackexchange.com with the "data-transformation" tag, and $X2$ represents the total number of posts that person made. If you were trying to answer the question "which person loves data transformation the most?" you couldn't just use raw counts of $X1$ because some people don't post as often. So as a compromise, $X1/X2$ would represent the proportion of posts a person makes with "data-transformation" tag and conceivably, a higher ratio indicates stronger preference (you'd have to discard people with low number of posts $X2$).

So yes, ratios are nice when you want a unitless relationship between two features. This doesn't really have much to do with random forests and the answer of whether a random forest can predict a ratio strongly depends on how you setup the cost and link functions it uses. Given enough trees, it can predict anything so in theory it will predict your ratios after binning $X1,X2$ in enough ways and combinations to give you a prediction of the ratio.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.