Regression using decision tree Could you explain how to perform decision tree regression? How is it different from decision tree classification?
 A: The main difference between classification and regression trees is that the target attribute (i.e. the variable you want to predict) of the classification tree is a continuous variable, while the target attribute of the decision tree is a categorical variable.
The main idea behind both is the same though.
For classifications, the metric used in the splitting process is an impurity index (e.g. Gini index) whilst for the regression tree, it is the Mean Squared Error.
A: In both cases, the algorithm works pretty much the same with some small differences. One of the differences is the loss function. Another difference is how they make predictions. For classification, a decision tree may do a majority vote, by predicting for a node of the tree the class of most of the training samples in this node. Using the example from the scikit-learn documentation (below), the leftmost final node had [0, 47, 0] training samples in the node, so the majority of the samples came from the second class. Another thing a classification decision tree can do is to calculate the probability, i.e. [0, 1, 0] for the example since all the samples ended in the middle class.

The regression tree would instead calculate something like the mean of the target variable for the samples in the node. If you used the regression tree for classification data, this would mean predicting the probabilities, the same as the classification tree would do (but not exactly the same as would be predicted by a classification tree). This is how the classification and regression algorithms are nearly the same.
If you plot the results predicted by a regression tree, you would see jumps, the predictions would be a step function (see another example from the scikit-learn documentation below). You can see from the plot that the decision tree had a node for the data values (x-axis) between 0.5 and 1.5, for which the mean of the target variable (y-axis) was around 0.9 and this is the prediction that it returned for the node.

You can notice the obvious problem with using a regression tree, that it cannot extrapolate outside the data. In the example below, if you asked it to predict for data values higher than 5, it would return the last seen value of around -1.
