Difference between regression and classification for random forest, gradient boosting and neural networks

Question

I am having a hard time completely understanding the difference between classification and regression for the three methods: Random forest, Gradient boosting and Neural networks (specifically Multilayer perception networks). As far as I understand the general concepts are the same for the three methods whether it is classification or regression.

But for Random Forest, as I understand, for classification it is the mode of the predicted classes which is the output and for regression, it is the mean of the individual decision trees. But I am uncertain about the two other methods.

EDIT:
With general concepts, I mean the following:

The Random forest method is an ensemble method that consists of multiple decision trees and is used for both regression and classification. A decision tree is a very simple technique and resembles a flowchart-like structure where each node represents a question that splits the data. The disadvantage of decision trees is however that they have a high risk of overfitting and often finds local optima instead of global. Random forest helps prevent this by constructing a collection of decision trees, thereby the name forest, and uses the results of all trees to compute the final outcome. The randomness in the name comes as bagging is used to produce a set of datasets from the original data set (by sampling with replacement). Furthermore, each tree is only trained on a random sample of the training data. Each tree is thus different from each other which again helps the algorithm prevent overfitting.

The gradient boosting algorithm is, like the random forest algorithm, an ensemble technique which uses multiple weak learners, in this case also decision trees, to make a strong model for either classification or regression. Where random forest runs the trees in the collection in parallel gradient boosting uses a sequential approach. Where the random forest method samples with replacement, with equal probability of choosing each observation, the boosting method assigns higher weights to observations which are harder to predict, this information is passed on to the next tree in the sequence. This procedure is repeated for a number of iterations, thus constructing the sequential approach and minimizing the prediction error.

The specific type of neural network used is the multilayer perceptron (MLP) which is a fully connected feedforward neural network that is used for both classification and regression. It consists of an input layer, one or several hidden layers and an output layer. All layers are connected such that one layer feeds into the next and all neurons, besides the ones in the input layer, uses a nonlinear activation function to map the weight of the previous layer to the next layer. When training the network backpropagation is used which is an algorithm used to compare the true and predicted value to compute the error using gradient descent. The algorithm goes backwards through the network to detect the weights which influence the errors the most and changes the weights to lower the errors

I might be wrong in my explanations and that there in fact are big differences?

Answer 1

I might understand your question and I'll keep it very hand-wavey.

You are correct for how random forests predict but for gradient boosting (although they have similarities) it is an iterative ensemble which means that we do have several models, however, each model is essentially just updating the previous model's predictions so it is nothing like the random forest in that respect. A MLP is not like the others in that the nodes are working together concurrently to combine your inputs for the prediction. So:

• Random Forest: Ensemble where each tree is a separate model predicting for the same thing. The bootstrapping and variable subset can be applied to basically any other model.
• Gradient Boosted Tree: Ensemble where each tree is a separate model which is dependent on the last tree and is trying to adjust for the last tree's error. The boosting algorithm which takes each round's residuals and trains the next model on these 'psuedo' residuals can be applied to basically any other model.
• MLP: A single model where the nodes are working together to create the prediction. Unlike the other two this is not an ensemble method, i.e. we can't typically can't take a single node of a trained mlp and do much with it.

To your other point, the boosting and mlp methods follow the same routine for classification or regression we just do transformations to allow for classification. Unlike random forests which do voting for classification and then use the mean for prediction.

Answer 2

I'll assume binary classification throughout. Multiclass or multilabel, and multioutput regression, may change things a bit more.

Gradient boosted trees don't have significant differences between regression and classification: everything is the same except the loss function whose derivatives are used as targets for the individual trees

Neural networks have more of a structural difference: everything is the same except for the activation of the final layer and the loss function applied to that layer and then backpropagated.

Random forests show perhaps the greatest difference. The individual trees are constructed slightly differently, using a different splitting criterion. Then the predictions are aggregated differently: for regression, it's just the mean; for classification, it can be the mode of the trees' hard classifications (to obtain a hard classifier), or the mean of the trees' hard classifications, or the mean of the trees' soft classifications.