I have created my first xgboost regressor. I input some self collected data, which I scale using sklearn's standardScaler. The model is trained on approximately 25,000 samples, and then tested on approximately 7000 examples, which are entirely separate from my training set.

In both the training and test set, the maximum y-label seen is 12. However, my xgboost regressor won't predict a value above approximately 6. See this example of my results: example results

My question is: What might be causing this? I have tried changing the parameters of the xgboost regressor, changing which factor I input, as well as trying sklearn's gradientBoostingRegressor. The result is broadly the same whatever I try, which leads me to believe that there is something I don't understand about the process I am using. I would really appreciate any advice on why this might be occurring, and what I might be able to do to rectify my problem. Thanks in advance.


I'm about to do something ... possibly dumb.

Here is your image with some "annotations": enter image description here

In the top-slice the predicted is about the same as the mean of the measured. There is lots of hand-waving to be had: it is a skewed slice so the right side is sparse, what is a bin-size, etcetera.

The point here is that the CART, the "atom" of both the RandomForest and GradientBoostedMachine performs "averaging" over the leaf for the predictor output. This means the prediction is going to try very hard to be inside of the training data. How "inside" it is depends on the data and the model parameters.

See also: some crazy dude making sketches about Random Forests.

  • 1
    $\begingroup$ I can somewhat understand what you want to say and I think you are right (I thought the same) but this exposition is quite unclear... (+1 cause you are probably right but yeah... too raw.) $\endgroup$
    – usεr11852
    Jul 14 at 23:04
  • $\begingroup$ @usεr11852 - I will try to improve. $\endgroup$ Jul 15 at 13:47
  • $\begingroup$ (sorry forgot to upvote originally) $\endgroup$
    – usεr11852
    Jul 15 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.