Why won't my xgBoost regressor predict above a certain value, even though it sees higher in training?

Question

I have created my first xgboost regressor. I input some self collected data, which I scale using sklearn's standardScaler. The model is trained on approximately 25,000 samples, and then tested on approximately 7000 examples, which are entirely separate from my training set.

In both the training and test set, the maximum y-label seen is 12. However, my xgboost regressor won't predict a value above approximately 6. See this example of my results:

My question is: What might be causing this? I have tried changing the parameters of the xgboost regressor, changing which factor I input, as well as trying sklearn's gradientBoostingRegressor. The result is broadly the same whatever I try, which leads me to believe that there is something I don't understand about the process I am using. I would really appreciate any advice on why this might be occurring, and what I might be able to do to rectify my problem. Thanks in advance.

Answer 1

I'm about to do something ... possibly dumb.

Here is your image with some "annotations":

In the top-slice the predicted is about the same as the mean of the measured. There is lots of hand-waving to be had: it is a skewed slice so the right side is sparse, what is a bin-size, etcetera.

The point here is that the CART, the "atom" of both the RandomForest and GradientBoostedMachine performs "averaging" over the leaf for the predictor output. This means the prediction is going to try very hard to be inside of the training data. How "inside" it is depends on the data and the model parameters.

See also: some crazy dude making sketches about Random Forests.