How Do I Build a Quantile Regression Model with GradientBoostingRegressor from sklearn?

Question

I am building a quantile regression model using scikit-learn's GradientBoostingRegressor algorithm.

I was going to use GridSearchCV for hyperparameter optimization.

Two questions:

1. Does it make sense to use gridsearchcv given that I am looking at quantile regression rather than mean-based regression?

2. If yes, what should I be using for the scoring parameter for GridSearchCV?

Answer 1

1. Yes, it makes perfect sense to use GridSearchCV. It is very reasonable way to choose hyper-parameters.
2. The scoring parameter should correspond to the quantile of interest $\alpha$. The mean quantile loss $\text{MQL}_{\alpha} $ for a particular quantile $\alpha$ is: $\frac{1}{N} \sum_{i=1}^N \rho_\alpha(y_i - \hat{y}_i)$. Here $\alpha \in (0,1)$ is a constant and the check function $\rho_{\alpha}(r)$ is: $r(\alpha - \mathbf 1_{r<0})$ or more descriptively: $\mathbf 1_{r<0} (1-\alpha) |r| + \mathbf 1_{r \geq 0} (\alpha) |r|$, with $r$ being our residual $r = y_i - \hat{y}_i$ and $\mathbf 1$ being the indicator function. (CV.SE has some great answer if you want to see more details on the matter here and here).

So our $\text{MQL}_{\alpha} $ loss function would be something like:

def mqloss(y_true, y_pred, alpha):  
  if (alpha > 0) and (alpha < 1):
    residual = y_true - y_pred 
    return np.mean(residual * (alpha - (residual<0)))
  else:
    return np.nan

Some minor final things to note:

1. we would have to use of make_scorer functionality from sklearn.metrics to create this custom loss function. We could then pass it to GridSearchCV as the scoring parameter. (i.e. some like: mqloss_scorer = make_scorer(mqloss, alpha=0.90))
2. we would have to refit our model/rerun GridSearchCV for each different choice of $\alpha$. This is inline with the sklearn's example of using the quantile regression to generate prediction intervals for gradient boosting regression.
3. our choice of $\alpha$ for GradientBoostingRegressor's quantile loss should coincide with our choice of $\alpha$ for mqloss. Otherwise we are training our GBM again one quantile but we are evaluating it against another. It is doable, but most likely incoherent. :)