how does predicted median go above 95% prediction interval when using GBM with quantile loss function

Question

I was checking out how to create prediction intervals with Gradient boosted regression trees using Scikit-learn. If you set the alpha at .95 or .05, you can get the 95% prediction interval around the prediction.

When I tried running it on my own data, I realized that sometimes, or often, the predicted mean was ABOVE the upper 95% prediction interval. How is this possible? It happens whether I change the loss function to 'ls' or keep it at 'quantile' but change the alpha to .5.

For an example, look at sklearn's documentation on it. You will notice that one of their predictions sits outside of the prediction intervals.

Note - it is not the observed value that goes beyond the prediction interval (which would be sensible ~5% of the time) but the prediction for that value of x. Said another way, if the quantile is set at .5 the predicted value, the prediction is sometimes higher than when the quantile is set at .95. That doesn't make sense to me.

http://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_quantile.html

Can anyone explain this to me?

Also, should a loss function of 'ls' == a loss function of 'quantile' with an alpha =.5?

Answer 1

I can answer the last part of your question: The 'ls' loss is different to the 'quantile' loss with an alpha of 0.5. While the former yields an estimate for the conditional mean of your response variable (through minimizing the sum of the residual least squares), the latter yields an estimate for the 50% quantile, i.e. the conditional median.

Regarding the first part of your question, maybe this paper helps (they propose a modified version of quantile regression which avoids the crossing quantiles you mentioned): http://www4.stat.ncsu.edu/~hdbondel/noCross.pdf