3
$\begingroup$

I am currently trying to compare the complexity of models. Among the models I have are some trees. The trees are not parametric models, hence they don't have the notion of 'trainable parameters' that are used in information criterion (AIC / BIC). I was wondering if there is a way to compare the complexity of trees to models with trainable parameters. Intuitively I would use the number of values to encode the model. So the complexity of the model would be the number of splits (split level) + the number of leafs (terminal value).

Is this approach correct ? Is there better way to compare trees complexity to other models ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

As outlined here Ye developed a method for estimating the effective number of parameters used by recursive partitioning, when Y is continuous. The results are scary, i.e., trees effectively estimate a huge number of parameters, which gives insights into why trees are so brittle and independently validate so seldom.

The simplest way to go is to use the bootstrap to estimate the likely future performance of a tree in comparison with a model. You'll quickly see why trees are not competitive. In order to be accurate they have to be pruned which removes some predictive ability.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks Pr. Harrell. I was looking at kaggle competitions and was wondering if it would make sense to set up a competition with an information criterion metric. I started with this question to know if we could compare NN and gbdt... From your answer I understand its best to evaluate performance out of sample. Right ? $\endgroup$
    – Lucas Morin
    Commented Aug 9, 2022 at 13:04
  • 2
    $\begingroup$ Unless roughly $N > 20000$ it's usually better to use strong internal validation with resampling. Split sample validation is too unstable/noisy. In terms of performance metrics the deviance (-2 log likelihood) is the gold standard but also see fharrell.com/post/addvalue and calibration curves. $\endgroup$
    – Frank Harrell
    Commented Aug 9, 2022 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.