I have am using Sklearns GradientBoostingRegressor for quantile regression as wells as a linear neural network implemented in Keras. I do however not know how to find the hyperparameters.

For the GradientBoostingRegressor a separate regression is fitted for each quantile. Do I find a new set of hyperparameters for every quantile or do I fit the same set of hyperparameters for every quantile? A possibility is to use the RandomizedSearchCV, but which 'scoring' should I use?

And for Keras how do I decide on the hyperparameters as the way I have implemented the model, it predicts all quantiles at the same time. The hyperparameters that I want to choose are the lr and rho. Below is an example of my implementation in Keras:

def quantile_loss_nn(y_true, y_pred):
    loss = 0
    for q_i, q in enumerate(quantiles):
        e = y_true - y_pred[:, q_i:q_i+1]
        loss += K.mean(K.maximum(q*e, (q-1)*e))
    return loss

def keras_linear_model(input_size, output_size, loss): 
    inputs = Input(shape=(input_size,))
    output = Dense(output_size)(inputs)
    model = Model(inputs=inputs, outputs=output)   
    optimiser = RMSprop(lr=0.01, rho=0.9)
    model.compile(optimizer=optimiser, loss=loss, metrics=['mae'])
    return model

quantiles = [0.025, 0.25, 0.5, 0.75, 0.975]

model_linear_M1 = keras_linear_model(X_train1.shape[1], len(quantiles), quantile_loss_nn)
epochs = 1000
batch_size = 32
model_linear_M1.fit(X_train1, y_train1,

EDIT: I have discovered that in the RandomizedSearchCV it is possible to use the make_scorer to construct your own scorer. I have tried to implement a loss function but it does not work:

def mqloss(y_true, y_pred):
    alpha_ = alpha_global
    e = y_true - y_pred
    return np.maximum(alpha*e, (alpha-1)*e)

Here alpha_global is specified outside of the definition.

  • $\begingroup$ You could do as in the TF model and pick the set of parameters that produce the smallest sum of univariate quantile (validation) losses. $\endgroup$
    – Michael M
    Jun 6, 2021 at 12:03
  • $\begingroup$ @MichaelM can you elaborate on that? :) $\endgroup$
    – andKaae
    Jun 6, 2021 at 12:11
  • $\begingroup$ I like your approach with TensorFlow. If you build multiple boosted trees, each with it's own quantile loss, then their trees will be different. So there is no benefit of using the same hyperparameters across quantiles. In my view, you would tune each model separately per quantile. $\endgroup$
    – Michael M
    Jun 7, 2021 at 17:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.