Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
So I am using HistGradientBoostingRegressor (scikit learn) to predict temperature values. After training and testing, the model seems to provide predictions that stagnates after certain values even if the actual values go beyond those values (see following Figure).
Is this a limitation of the model or am I missing something?
This is a limitation that's inherent to the particular version you are using (which is the most common one). What you do there, is that you have multiple trees (the overall prediction is a weighted average of the predictions of all the trees) and each tree has branches. Each observation "goes along" branches and at each branching point either goes right or left. The decision is based on some split of feature values that occur in the training data (i.e. there will not be any splits once you are beyond the feature range of the training data, and there will not be splits at multiple values between two adjacent feature values). After several splits, you eventually reach a leaf node and the prediction is a single value at that leaf node. See e.g. this notebook for some illustrations in a simple setting.
So, inherently, once you go beyond the values seen in training for features, predictions will stay constant (such tree based models basically don't extrapolate beyond the training data, or interpolate between training data points). That's why it's really good, if there is a lot of varied training data and if the training data covers the whole range of the feature space of interest for practical predictions.
Alternative versions might e.g. build some simple model (e.g. something like a (penalized?) linear regression with only certain features) at each leaf node, such models would extrapolate to an extent.
