2
$\begingroup$

Consider tree ensemble methods such as XGBoost, Lightgbm and/or Catboost.

Is the adj. $R^2$ a valid metric for tree ensembles?

I'm curious because these methods handle factor variables differently. E.g. XGBoost needs some kind of one-hot encoding, Lightgbm tries to unite one-hot encoded variables and Catboost uses an unique handling called ordered target encoding. Without going into details, the aforementioned handling does not extend the feature space since the factors are encoded inside of the factor variable. At least for XGBoost and Catboost this always leads to a different number of variables which favors Catboost in terms of the adj. $R^2$ metric.

$\endgroup$

1 Answer 1

2
$\begingroup$

No, adjusted $R^2$ is not a valid metric of tree ensembles.

In order to have an "adjusted" $R^2$ we need some agreed concept of degrees of freedom. Tree ensembles don't have one, so the idea of "adjusting for model complexity" in a universal manner is ill-defined. To that extent, different GBM implementations even use different tree-growing strategies thus making even simple "tree-to-tree" comparison's somewhat moot. We should just use a proper cross-validation schema with $R^2$ (if it is a metric relevant to the application at hand).

$\endgroup$
2
  • $\begingroup$ Can you please explain why tree ensembles do not have degrees of freedom? $\endgroup$
    – bigdata2
    Oct 31, 2022 at 20:27
  • 1
    $\begingroup$ This is too big of a subject for comment section but in short we do not have universally accepted way as with linear models. The two obvious issues are that: 1. As each tree is a non-parametric learner that can accommodate multiple definitions of DoF (e.g. based on an approximate hat matrices, the residuals, the number of leafs, etc.) that can be "good" and 2. each base learner is regularised by the learning rate so it's ensemble contribution is unequal. Feel free to open a new question on this! :) Defining an appropriate validation schema circumvents this issue. $\endgroup$
    – usεr11852
    Oct 31, 2022 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.