I was checking out how to create prediction intervals with Gradient boosted regression trees using Scikit-learn. If you set the alpha at .95 or .05, you can get the 95% prediction interval around the prediction.
When I tried running it on my own data, I realized that sometimes, or often, the predicted mean was ABOVE the upper 95% prediction interval. How is this possible? It happens whether I change the loss function to 'ls' or keep it at 'quantile' but change the alpha to .5.
For an example, look at sklearn's documentation on it. You will notice that one of their predictions sits outside of the prediction intervals.
Note - it is not the observed value that goes beyond the prediction interval (which would be sensible ~5% of the time) but the prediction for that value of x. Said another way, if the quantile is set at .5 the predicted value, the prediction is sometimes higher than when the quantile is set at .95. That doesn't make sense to me.
Can anyone explain this to me?
Also, should a loss function of 'ls' == a loss function of 'quantile' with an alpha =.5?