I am having a hard time completely understanding the difference between classification and regression for the three methods: Random forest, Gradient boosting and Neural networks (specifically Multilayer perception networks). As far as I understand the general concepts are the same for the three methods whether it is classification or regression.
But for Random Forest, as I understand, for classification it is the mode of the predicted classes which is the output and for regression, it is the mean of the individual decision trees. But I am uncertain about the two other methods.
With general concepts, I mean the following:
The Random forest method is an ensemble method that consists of multiple decision trees and is used for both regression and classification. A decision tree is a very simple technique and resembles a flowchart-like structure where each node represents a question that splits the data. The disadvantage of decision trees is however that they have a high risk of overfitting and often finds local optima instead of global. Random forest helps prevent this by constructing a collection of decision trees, thereby the name forest, and uses the results of all trees to compute the final outcome. The randomness in the name comes as bagging is used to produce a set of datasets from the original data set (by sampling with replacement). Furthermore, each tree is only trained on a random sample of the training data. Each tree is thus different from each other which again helps the algorithm prevent overfitting.
The gradient boosting algorithm is, like the random forest algorithm, an ensemble technique which uses multiple weak learners, in this case also decision trees, to make a strong model for either classification or regression. Where random forest runs the trees in the collection in parallel gradient boosting uses a sequential approach. Where the random forest method samples with replacement, with equal probability of choosing each observation, the boosting method assigns higher weights to observations which are harder to predict, this information is passed on to the next tree in the sequence. This procedure is repeated for a number of iterations, thus constructing the sequential approach and minimizing the prediction error.
The specific type of neural network used is the multilayer perceptron (MLP) which is a fully connected feedforward neural network that is used for both classification and regression. It consists of an input layer, one or several hidden layers and an output layer. All layers are connected such that one layer feeds into the next and all neurons, besides the ones in the input layer, uses a nonlinear activation function to map the weight of the previous layer to the next layer. When training the network backpropagation is used which is an algorithm used to compare the true and predicted value to compute the error using gradient descent. The algorithm goes backwards through the network to detect the weights which influence the errors the most and changes the weights to lower the errors
I might be wrong in my explanations and that there in fact are big differences?