Could you explain how to perform decision tree regression? How is it different from decision tree classification?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Classification tends to use a "right or wrong" loss function. Regression allows a "how wrong"-based loss function. $\endgroup$– HenryCommented Jun 29, 2021 at 9:40
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
6
In both cases, the algorithm works pretty much the same with some small differences. One of the differences is the loss function. Another difference is how they make predictions. For classification, a decision tree may do a majority vote, by predicting for a node of the tree the class of most of the training samples in this node. Using the example from the scikit-learn documentation (below), the leftmost final node had [0, 47, 0] training samples in the node, so the majority of the samples came from the second class. Another thing a classification decision tree can do is to calculate the probability, i.e. [0, 1, 0] for the example since all the samples ended in the middle class.
The regression tree would instead calculate something like the mean of the target variable for the samples in the node. If you used the regression tree for classification data, this would mean predicting the probabilities, the same as the classification tree would do (but not exactly the same as would be predicted by a classification tree). This is how the classification and regression algorithms are nearly the same.
If you plot the results predicted by a regression tree, you would see jumps, the predictions would be a step function (see another example from the scikit-learn documentation below). You can see from the plot that the decision tree had a node for the data values (x-axis) between 0.5 and 1.5, for which the mean of the target variable (y-axis) was around 0.9 and this is the prediction that it returned for the node.
You can notice the obvious problem with using a regression tree, that it cannot extrapolate outside the data. In the example below, if you asked it to predict for data values higher than 5, it would return the last seen value of around -1.
-
$\begingroup$ what is the impurity metric used in each node in the case of decision tree regression? $\endgroup$ Commented Jul 4, 2021 at 7:33
-
$\begingroup$ @AI_Revolt they use mean squared error or mean absolute error, as mentioned in the first link, not impurity score. $\endgroup$– TimCommented Jul 4, 2021 at 8:09
-
$\begingroup$ Could you provide links to references where we use MSE or MAE as impurity metric like we use gini index or entropy in the case of classification? $\endgroup$ Commented Jul 4, 2021 at 8:43
-
$\begingroup$ @AI_Revolt not as a impurity metric but as a loss. This is covered in any machine learning handbook that mentions regression trees, e.g. this one web.stanford.edu/~hastie/ElemStatLearn $\endgroup$– TimCommented Jul 4, 2021 at 9:16
-
$\begingroup$ Could you brief how feature importance scores are computed using MSE? $\endgroup$ Commented Jul 8, 2021 at 11:19
$\begingroup$
$\endgroup$
2
The main difference between classification and regression trees is that the target attribute (i.e. the variable you want to predict) of the classification tree is a continuous variable, while the target attribute of the decision tree is a categorical variable.
The main idea behind both is the same though.
For classifications, the metric used in the splitting process is an impurity index (e.g. Gini index) whilst for the regression tree, it is the Mean Squared Error.
-
1$\begingroup$ Could you brief how feature importance scores are computed using MSE? $\endgroup$ Commented Jul 8, 2021 at 11:18
-