15
$\begingroup$

I'm actually writing an implementation of Random Forests but I believe the question is specific to decision trees (independent of RFs).

So the context is that I'm creating a node in a decision tree and both the prediction and target variables are continuous. The node has a split threshold to partition data into two sets, and I create a new prediction for each subset based on the average target value in each set. Is this the correct approach?

The reason I ask is that when predicting binary variables I believe the typical (correct?) approach is to divide the data into 0 and 1 subsets without taking an average over the data rows in each subset. Subsequent splits will divide into finer grained subsets and taking an average at each split results subsequent splits (lower down the decision tree) operating on what are now continuous variables rather than binary variables (because we are operating on the residual error values instead of the original targets).

Side question: Is the distinction between the two approaches (binary vs continuous) significant - or will they actually give identical results for a complete decision tree?

$\endgroup$
  • 1
    $\begingroup$ Splitting on a continuous variable will be sure to make the resulting "model" not fit the data properly. If you have one continuous X and one continuous Y consider using the loess nonparametric smoother. $\endgroup$ – Frank Harrell Dec 31 '11 at 15:56
  • $\begingroup$ The problem I'm working on right now has many predictor variables (a mix of continuous and binary) and a single target variable. Hence I believe RF is a reasonable approach to take. $\endgroup$ – redcalx Dec 31 '11 at 16:25
  • 2
    $\begingroup$ Very likely so. But a random forest is a mixture of trees (it is not a decision tree), so it approximates continuous relationships by making multiple splits, and in effect, using shrinkage. So I don't think your original question applies, if I understand it. $\endgroup$ – Frank Harrell Jan 1 '12 at 5:21
  • $\begingroup$ I'm tempted to say that your description of the continuous case is correct (i.e. the standard way of doing things), but then your description of the binary variable case does not match up at all with my understanding of how random forests (or decision trees) work, so I'm worried that one of us is confused. $\endgroup$ – joran Jan 3 '12 at 5:11
  • $\begingroup$ @joran. Yah, by forcing predictions to be 0 or 1 you lose the ability to make subtle adjustments to predictions (between 0 and 1) that can lower error (e.g. mean squared prediction error). As such I suspect that approach to be inferior. I tried it and most attempts at building a decision tree fail to find even a single split that improves error. $\endgroup$ – redcalx Jan 8 '12 at 23:46
10
$\begingroup$

One potential issue with trees is that they tend to fit poorly in the tails. Think of a terminal node that captures the low range of the training set. It will predict using the mean of those training set points, which will always under-predict the outcome (since it is the mean).

You might try model trees [1]. These will fit linear models in the terminal nodes and (I think) do a better job than regression trees. Better yet, use a more evolved version called Cubist that combines different approaches ([1] and [2] below).

These models also handle continuous and discrete predictors differently. They can do multi-way splits for categorical variables. The splitting criterion is very similar to CART trees.

Model trees can be found in R in the RWeka package (called 'M5P') and Cubist is in the Cubist package. Of course, you can use Weka too and Cubist has a C version available at the RuleQuest website.

[1] Quinlan, J. (1992). Learning with continuous classes. Proceedings of the 5th Australian Joint Conference On Artificial Intelligence, 343–348.

[2] Quinlan, J. (1993). Combining instance-based and model-based learning. Proceedings of the Tenth International Conference on Machine Learning, 236–243.

$\endgroup$
  • 1
    $\begingroup$ Couldn't you just have deeper trees to minimise the poor fit to the tails? $\endgroup$ – Jase Feb 9 '14 at 7:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.