Predicting limits of bounded dependent variable in Random Forest

Question

I am new to machine learning and trying to use Random Forest to predict a bounded dependent variables (percentage from 0 - 100). The majority of the training data points (~80%) are at the limits of these bounds, so either 0 or 100.

What I am finding is that my predictions never reach these limits. For example, instead of predicting 100 it will predict say 80% (see actual vs predicted plot below). For other dependent variables I have, the majority of the data points are at 100 exactly (but don't go down as low as 0), however as with the other example this never is able to predict the limit, instead giving values of around 96% max. This small difference unfortunately has major implications for my work and cannot be used.

Can anyone help explain why this might be the case and suggest some tips to solve?

I am using the R package "MultivariateRandomForest" and have so far tried to systematically vary the number of trees, features and leaves used in the model, but without seeing any real improvement. I have also tried splitting the dependent variable into two halves (0-50 & 50-100). Whilst this improved predictions for the top half, the same issue persisted and the limits not reached.

I have read this paper https://arxiv.org/pdf/1901.06211.pdf which talks about using a beta distribution but as I am new to this, I am not sure how to implement this or if it is actually a viable solution.

Many Thanks, Josh

Answer 1

A Decision Tree essentially works by splitting the train set into partitions. A prediction is obtained by selecting the partition which your unseen data point belongs to, then the output value is calculated as the mean of all the dependent variable values in that partition.

My guess is that the partitions you get never contain all 0 (or all 100) values, so when you calculate the mean you get slightly higher (or lower) values than expected. Moreover, Random Forest trains a lot of Decision Trees and then takes the mean of the predictions of each tree, so here you have even more chances that the output deviates from extreme values.

You could try to train trees with only one (or a few) sample per leaf so that you are more likely to reach the extremes, but I would suggest not to do it because the model would overfit a lot.

I have never heard of that Random Forest with Beta distribution approach you linked so I don't know whether it would work well. However, you could try to use a more simple linear model instead of a tree-based one.

There is something called Beta Regression (try to look at this since you use R), but I don't know anything about it so I'm not sure whether it would help.

Otherwise you can try to look at Generalized Linear Models. They are a general class of linear models which can fit different distributions of a target variable. Unfortunately I have never used them in practice, so I will just leave it as a suggestion. Maybe someone more expert than me can give you better advice on this topic.