My main objective is to construct a regression (decision) tree. It is a part of a boosting algorithm using additive regression trees.
The first question is what other functions (other than least square error, which I am using now) do you suggest to use for the best split choice?
In fact, I am not really using standard regression trees but I am using oblivious trees - i.e. trees where you use the same split (same feature, same threshold) for all the decision nodes in a particular depth of the tree (which is different to a standard regression tree where you can use a different split for each node). What would you suggest to use as a splitting function in this case?
As I mentioned before, I am currently using LSE, which is not really suitable for this purpose, e.g. because it doesn't "penalize" trees with very unbalanced branches (i.e. it is totally ok with splits like "1 samples vs. 10000 samples").
There are construction algorithms as Information Fuzzy Network or EODG. Those are using functions as Adjusted mutual information which is good for classification problems but not for regression.
Thank you for any ideas on splitting function choice