how does predicted median go above 95% prediction interval when using GBM with quantile loss function

Question

I was checking out how to create prediction intervals with Gradient boosted regression trees using Scikit-learn. If you set the alpha at .95 or .05, you can get the 95% prediction interval around the prediction.

When I tried running it on my own data, I realized that sometimes, or often, the predicted mean was ABOVE the upper 95% prediction interval. How is this possible? It happens whether I change the loss function to 'ls' or keep it at 'quantile' but change the alpha to .5.

For an example, look at sklearn's documentation on it. You will notice that one of their predictions sits outside of the prediction intervals.

Note - it is not the observed value that goes beyond the prediction interval (which would be sensible ~5% of the time) but the prediction for that value of x. Said another way, if the quantile is set at .5 the predicted value, the prediction is sometimes higher than when the quantile is set at .95. That doesn't make sense to me.

http://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_quantile.html

Can anyone explain this to me?

Also, should a loss function of 'ls' == a loss function of 'quantile' with an alpha =.5?

Answer 1

I can answer the last part of your question: The 'ls' loss is different to the 'quantile' loss with an alpha of 0.5. While the former yields an estimate for the conditional mean of your response variable (through minimizing the sum of the residual least squares), the latter yields an estimate for the 50% quantile, i.e. the conditional median.

Regarding the first part of your question, maybe this paper helps (they propose a modified version of quantile regression which avoids the crossing quantiles you mentioned): http://www4.stat.ncsu.edu/~hdbondel/noCross.pdf

Answer 2

This is goofy behavior and I remember reading a paper that referred to this kind of "quantile crossover" as an "embarrassing problem" (don't remember the reference). However, when you fit multiple quantile regression models, you can, indeed, wind up with this kind of quantile crossover.

What happens is that the model aims to accomplish a task (minimize loss) and do it well overall, not at any particular place. By making a sacrifice that leads to very high predicted medians in a few places, the model makes better predictions overall than it would by making lower predictions for the median in those places (ditto for making lower-than-would-otherwise-be-desired predictions for the $0.95$ quantile).

Skimming Google Scholar, there appear to be papers that aim to remedy quantile crossover if you want to predict multiple quantiles simultaneously.