1
$\begingroup$

It has been said(link , link) that gradient boosting can predict values that fall outside of training domain for $Y$ in a regression problem.

I intuitively sense that there is a distinction between how well gradient boosting can generalize outside $Y$ domain when compared to simple linear model (for simplicity, let's assume that this generalization is correct, despite no data supporting it).

For a simple linear model, this generalization is pretty straightforward. For a simple model $y = 2x$, when learned on domain $[1;10]$, the model will predict $100$ for $x = 50$.

The predictions in gradient boosting though may lie outside training domain, but in general they would be artifacts, lying close to the boundary, despite values of $X$ suggesting otherwise. For the above example, I would expect predictions closer to $10$ than $100$.

Example proving otherwise and/or math proof or navigation to resource based on which I can proof that would be appreciated.

$\endgroup$
2
$\begingroup$

It's possible for gradient boosted trees to make meaningful predictions outside the training domain of $Y$ because each subsequent tree after the first iteration are based on predicting the error ($\epsilon_{n-1}$) of the previous tree(s) and therefore while the initial tree is restricted to the training domain of $Y$, summing across gradient boosted trees ($\bar{Y} = \bar{Y_1}+\bar\epsilon_{1}+... + \bar\epsilon_{n-1}$) is not. ($n$ is the number of iterations/trees.)

Let's take simple example of a two tree scenario with just two regressors. I want to predict house prices ($Y$) and have two regressors, number of bedrooms($X_1$) and area ($X_2$) (either A,B or C).

Now assume the first iteration only uses the bedroom variable($X_1$) to make predictions for $Y$. This first tree works out that as bedrooms increase the price increases, let's assume for simplicity each 1 addition bedroom add \$10,000 to the price of a house and that 1 bedroom houses start at \$50,000. This first iteration generated errors $\epsilon_1 $.

The second iteration then make use of area ($X_2$) to make predictions for $\epsilon_1$.

Area A is a highly desirable area and the second iteration determines that when $X_2==A$ the price of the house should be increased by $50,000.

Now let's assume the highest value house in our training dataset is \$100,000 which is say a 6 bed house in area B. It's also the case that our training data only has 1 to 3 bedroom houses for area A. Now we want to value a newly built 5 bedroom house in area A. The first tree values this at \$90,000. However, the second itteration knows that houses in area A are typically under-valued by the first iteration and that it's likely $\epsilon_1 $ is -\$50,000. Thus, the model will predict \$140,000 for this new 5 bedroom house, which is a meaningful prediction outside of the training domain for $Y$. I'm sure you could even imagine scenarios where the prediction from the model could meaningfully be even further from the training domain but it's all because Gradient Boosted Models are based on correcting errors beyond the first iteration.

$\endgroup$
  • $\begingroup$ This is a very good explanation on why xgboost can fall outside Y training domain, but for my specific question, it makes me think on the role of distribution of $\epsilon$ in the problem. In your example, to fall outside of training domain (even an over-exaggerated 10-fold), one requires the value of -\$50,000, which may only come from an error-averaging process, I think? Hence error itself cannot be greater than $Y_{max} - Y_{min}$? Even for very diverse datasets with respect to area, there appears to exist an upper bound on error, non-existent for linear model. $\endgroup$ – user2530062 Jan 3 at 14:28
  • 1
    $\begingroup$ The error could be greater than $Y_{max}-Y_{min}$ if there are more than 2 iterations/multiple regressors but yes I think I'd agree that the bounds for $\bar{Y}$ are more limited coming from a gradient boosted tree model than a linear model. However, this may be desirable. In my experience, the fact that $\bar{Y}$ from a linear model can tend towards Infinity based on certain levels of $\beta$ /$X$ can cause issues, hence why regularised linear models are popular. $\endgroup$ – Morgan Ball Jan 3 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.