Using Random Forest Regression vs Classification for Confidence Metrics

Question

I have a fairly large dataset with each sample corresponding to one of two target values. I'm using a random forest to assign confidences to new samples for what class they may belong to.

My question is: Does the output of a random forest regressor (mapping each new sample to [0.0, 1.0], where 0 and 1 are the labels for the training samples) differ as a metric for "confidence" from the probabilities given by a random forest classifier on the same data? In other words, would it be valid to train a random forest regressor on the data and interpret the model's outputs on various new samples as the probability that it belongs to class 1?

Any explanation/intuition behind answers would be much appreciated!

Answer 1

Random forest algorithms for regression in classification are nearly the same. The important difference is the loss function used for the training. For classification, they typically use entropy, or Gini impurity, for regression squared or absolute error. You can use squared error for classification tasks, it even has a special name: Brier score. Recently there was an interesting paper by paper by Hui and Belkin (2020) showing that using squared error as a loss function for a neural network may give as good if not better results as compared to the "default" log loss. However, keep in mind that using different loss functions may lead to different results.

But why would you do that? Random forest classifier does predict the probabilities, scikit-learn has the predict_proba function for that purpose.

Moreover, if you care more about the probabilities than the hard classification, you should validate and calibrate the probabilities. For more details check questions tagged as calibration.

Answer 2

yes, that should be the objective. train on one dataset and test on other datasets. Confidence and accuracy should be same. if not your model might be over fit or under fit.