So I am using HistGradientBoostingRegressor (scikit learn) to predict temperature values. After training and testing, the model seems to provide predictions that stagnates after certain values even if the actual values go beyond those values (see following Figure).

Is this a limitation of the model or am I missing something?

Scatter plot of predictions and actual temperature values


1 Answer 1


This is a limitation that's inherent to the particular version you are using (which is the most common one). What you do there, is that you have multiple trees (the overall prediction is a weighted average of the predictions of all the trees) and each tree has branches. Each observation "goes along" branches and at each branching point either goes right or left. The decision is based on some split of feature values that occur in the training data (i.e. there will not be any splits once you are beyond the feature range of the training data, and there will not be splits at multiple values between two adjacent feature values). After several splits, you eventually reach a leaf node and the prediction is a single value at that leaf node. See e.g. this notebook for some illustrations in a simple setting.

So, inherently, once you go beyond the values seen in training for features, predictions will stay constant (such tree based models basically don't extrapolate beyond the training data, or interpolate between training data points). That's why it's really good, if there is a lot of varied training data and if the training data covers the whole range of the feature space of interest for practical predictions.

Alternative versions might e.g. build some simple model (e.g. something like a (penalized?) linear regression with only certain features) at each leaf node, such models would extrapolate to an extent.

  • $\begingroup$ Great, thank you for your detailed response! $\endgroup$ Mar 2 at 20:45
  • $\begingroup$ Clarification: gradient boosters can give predictions outside the original range of the data. Random Forests cannot. GBMs rarely output such prediction values but it is still possible to be observed because in contrast to standard tree learners they learn the "residuals", not the values themselves. This is obviously true for individual base learners (we might have positive only data and need to predict negative residuals) but can at times be realised in the final ensemble too. Granted, usually they are "relatively small" steps outside the observed support of the data but they still count. :) $\endgroup$
    – usεr11852
    Mar 3 at 1:04
  • $\begingroup$ Agreed, and never claimed otherwise, the standard version does not extrapolate beyond the training data features (and the version with regression models in the nodes just keeps the regression model constant beyond the training data features). $\endgroup$
    – Björn
    Mar 3 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.