# Why are my random forest regression predicts valid probability distributions?

I have tabular input data where the labels correspond a probability distribution on five actions, E.g. a row might look like: x_0, x_1, ...., x_n, .1, .1, .3, .0, .5

I am using sklearn's Random forest regressor for this. It handles it well enough, in the sense that the cross-validated cross entropy loss is significantly lower than for a randomly generated prob. distribution. It also always outputs a pdf, i.e. the sum of the 5 predictions always = 1. Frankly, I don't understand why that is happening, since it's just doing linear regression, right?

I don't think this is the right way to do it, still. Surely the splitting is not being done to minimize cross entropy loss, which is the right metric. Also the fact that the output sums to 1 is disconcerting. I've considered doing regression on the 4 softmax logits of the target prob. distro instead, but I haven't tried it since I wrote this post (also I don't see why that would work better, other than clearly guaranteeing a prob distro as output). Is there a way to create a custom splitting criterion that will minimize CEL instead of MSE or whatever the regression trees are using currently?

Frankly, I don't understand why that is happening, since it's just doing linear regression, right?

Random forests are not linear regression. A linear regression estimates coefficients of a linear model, while random forest simply does not. A useful resource for learning about random forests is Leo Breiman's 2001 paper "Random Forests."

Also the fact that the output sums to 1 is disconcerting.

The "targets" of the regression forest you describe are numbers between 0 and 1 that always sum to 1. The prediction of each tree is an average of the terminal leaf targets. The prediction of the ensemble is the average of the predictions of the trees. So we predictions that are a convex combination of probability distribution, so the result is also a probability distribution. All discrete probability distributions sum to 1.

• Thanks. I actually realized this shortly after I posted the question. Someone told me in the meantime that RandomForestClassifier would be better suited, but the problem is that sklearn's implementation does not take floats as targets. Any advice? Commented Mar 22, 2022 at 0:42
• I don't see anything obviously wrong with what you're doing right now. Perhaps you could post a question explaining what problem you're trying to solve, why you're worried that random forest regression is not the correct solution, and why you think RandomForestClassifier is a better choice.
– Sycorax
Commented Mar 22, 2022 at 0:54
• That's what I tried to do here. It's alpha zero, but using trees and only worrying about the move selection policy. Forest regression is working right now but its probably using the wrong error. It would be like training the policy head of alpha zero by trying to minimize the euclidian distance of the target and predicted policy, instead of cross entropy. I know trees aren't using linear regression now. Commented Mar 22, 2022 at 1:01
• It's fine to ask a second question. Your first question, in the body of the post, was answered, and now you have a second one, but I don't understand why you're worried, and I'm not sure that I know the answer to the second one. No one else is likely to see these comments, but they're more likely to see & understand a second question that lays out what you know, what problem you're solving, and where you're stuck. The constraints of the comment box and software make it hard to ask/answer a question in comments, which is why Questions exist.
– Sycorax
Commented Mar 22, 2022 at 1:05
• Implement a tree ensemble that accepts floats as targets. But you still haven't explained why using a regression tree is insufficient, which is why you should ask a new question.
– Sycorax
Commented Mar 22, 2022 at 1:07