My main objective is to construct a regression (decision) tree. It is a part of a boosting algorithm using additive regression trees.

The first question is what other functions (other than least square error, which I am using now) do you suggest to use for the best split choice?

In fact, I am not really using standard regression trees but I am using oblivious trees - i.e. trees where you use the same split (same feature, same threshold) for all the decision nodes in a particular depth of the tree (which is different to a standard regression tree where you can use a different split for each node). What would you suggest to use as a splitting function in this case?

As I mentioned before, I am currently using LSE, which is not really suitable for this purpose, e.g. because it doesn't "penalize" trees with very unbalanced branches (i.e. it is totally ok with splits like "1 samples vs. 10000 samples").

There are construction algorithms as Information Fuzzy Network or EODG. Those are using functions as Adjusted mutual information which is good for classification problems but not for regression.

Thank you for any ideas on splitting function choice


1 Answer 1


For regression trees what I have seen in literature have almost the same results: minimize the sum of left and right variance, minimize the sum of left and right standard deviations. To handle cases like you mentioned I saw only simple constructs like minimum number of instances on left and right, or minimum percentage of instances on left and right.

However in order to incorporate other loss functions in a boosting tree I used for gradient boosting trees (and I bet it works also for others) the loss function at the GBT fitting level. Thus, considering that the tree is a greedy approach, anyway, it would not provide many different results if you would change the split function. A change in behavior is achieved only if incorporated at boosting procedure level.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.