i am currently experimenting with quantile-regression in h2o. I am building prediction intervals. For the individual regression models i am looking after R^2, RMSE and also quantile-loss. I performed a grid-search for the two quantile-models that build the lower and upper bound of the interval which improved R^2 (higher) and RMSE (lower). However, quantile-loss did increase for the optimized models, compared to the "base"-models.
I am now thinking of the role of quantile-loss and how to interpret this. I thought that quantile-loss would decrease as well when models are tuned.
Any thoughts or hints on this?