4
$\begingroup$

This is a question about feature engineering for decision trees/random forests. Given two continuous variables X1 and X2, is it informative to construct a third feature variable $\frac{X1}{X2}$, i.e. the ratio between these variables? What about $\frac{X2}{X1}$?

I think it is equivalent to whether a tree/random forest can solve the following problem:

Consider the following dataset

X1  X2  Y
1   3   0.333333333
3   4   0.75
2   3   0.666666667
5   6   0.833333333
6   15  0.4
8   12  0.666666667
...

Here, Y is the ratio of the first two variables. Can a tree/random forest predict Y given X1 and X2?

$\endgroup$

3 Answers 3

4
$\begingroup$

The CART model is fundamentally piece-wise constant. The ensemble variations (rf and gbm) can have more smoothness but are still piece-wise constant in element. They can approximate and, in a cousin-manner to neural networks, are universal approximators but are not the best non-local generalizers.

The CART basis, like Navier-Stokes for the continuity assumption[1], is only valid when the piece-wise constant model is valid. Like NS when the big-box average is no longer valid over the infinitesimal element, interpolation or extrapolation using CART when the constant mean model is not the appropriate approximation, breaks down.

(It must be late, I'm think of NS in terms of the big-box model. I blame Dr. J, formerly known as 'jaws', one of the most excellent teachers.)

If you train them over a restricted domain, one with a region of the domain missing, for $$ f \left( x_1, x_2 \right) = \frac {x_1} {x_2} $$ and then try to interpolate, the interpolation is going to have issues.

In regions where:
$$ \frac{\partial f\left(x_{1},x_{2} \right)}{\partial x_{1}} = 0 $$ and
$$\frac{\partial f\left(x_{1},x_{2} \right)}{\partial x_{2}} = 0 $$

either by taking very small changes in independent variables, or where $x_{2}$ is really big compared to $x_{1}$ and $x_{1}$ doesn't change sign, then the approximation is going to have less interpolation error.

$\endgroup$
5
  • $\begingroup$ Thanks for the answer @EngrStudent. However, what is your last opinion on the question: is it good or not to model a ratio (in my case [0,1]). Should I just learn different models (rf, brt, XGBoost...) and assess which one has the best performance? $\endgroup$
    – FrsLry
    Jul 28, 2022 at 11:43
  • 1
    $\begingroup$ Begin with the end in mind. The model is the means, not the goal. What do you want to get from the model? There are ways to transform [0,1] that make it more tractable for different use cases. $\endgroup$ Jul 28, 2022 at 15:32
  • $\begingroup$ I want to predict the probability of something. I am now looking for ways to transform my probability in order to increase the performance (the first learning on the raw proba [0,1] had bad performances). $\endgroup$
    – FrsLry
    Jul 29, 2022 at 10:00
  • 1
    $\begingroup$ Converting probability to logons makes it a more linear problem. This is how logistic regression works in its guts. It maps the domain between zero and one to the domain allegedly between negative infinity and positive infinity, but in practice between a large negative number and a large positive number. Machine learners are much happier with quasi linear styles of regression. $\endgroup$ Jul 29, 2022 at 13:20
  • 1
    $\begingroup$ Very interesting answer! Actually I made the comparisons between GLM (as you mentioned), GAM and tree based approaches and it seems like XGBoost performs the best, even without transforming the target variable $\endgroup$
    – FrsLry
    Aug 1, 2022 at 13:22
0
$\begingroup$

I would say decision tree may not learn ratios, but random forest (taking the average from many trees) can definitely learn ratios.

In the case you have only two variables, it should be quite easy for random forest to catch Y=X1/X2.

In the case where you have many variables, it will be hard to easy which ratio to use, so if you think X1/X2 is useful and meaningful, you should manually add this feature. Remember that tree models (random forest or XGBoost, they have parameters to random pick features, so if that ratio is really useful, it may not be picked by the algorithm simply due to randomness.)

$\endgroup$
-1
$\begingroup$

Suppose each row corresponds to a person, with $X1$ representing the number of posts they've made to stats.stackexchange.com with the "data-transformation" tag, and $X2$ represents the total number of posts that person made. If you were trying to answer the question "which person loves data transformation the most?" you couldn't just use raw counts of $X1$ because some people don't post as often. So as a compromise, $X1/X2$ would represent the proportion of posts a person makes with "data-transformation" tag and conceivably, a higher ratio indicates stronger preference (you'd have to discard people with low number of posts $X2$).

So yes, ratios are nice when you want a unitless relationship between two features. This doesn't really have much to do with random forests and the answer of whether a random forest can predict a ratio strongly depends on how you setup the cost and link functions it uses. Given enough trees, it can predict anything so in theory it will predict your ratios after binning $X1,X2$ in enough ways and combinations to give you a prediction of the ratio.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.