Which metric to use to evaluate Quantile Regression?

Question

I have a prediction problem for which I want to predict the 75% Quantile using Quantile Regression. I am a little bit confused on how to evaluate this model (and also compare different models).

If I understand this approach correctly, for given prediction y_hat the following is true:

1. With a chance of 75%, the value of y_hat is lower than the actual value of y
2. With a chance of 25%, the value of y_hat is higher than the actual value of y

In order to evaluate this model, intuitively I would would compare the overestimations in the test to the underestimations. An "ideal" model would "underestimate" in 25% of the cases (i.e. the value of y_hat is higher than the actual value of y).

Is this a valid approach?

Because I find it somehow confusing to not evaluate regression metrics like MAE, RMSE, Rsquared, etc. But these metrics do no seem suitable here (because we deliberately overestimate). Is there some asymmetric evaluation metric I am missing for this case? Happy if anyone can help clarify my confusion.

Note: I am aware of this post and the respective papers linked there. But is there a corresponding (python) implementation?

Answer 1

Quantile regression at quantile $\tau$ has a loss function, often called "pinball loss". Let

$$ l_{\tau}(y_i, \hat y_i) = \begin{cases} \tau\vert y_i - \hat y_i\vert, & y_i - \hat y_i \ge 0 \\ (1 - \tau)\vert y_i - \hat y_i\vert, & y_i - \hat y_i < 0 \end{cases} $$

Then we add up each individual $l$ to get a loss $L$ for the whole model. $$ L_{\tau}(y, \hat y) = \sum_{i=1}^n l_{\tau}(y_i, \hat y_i) $$

Use the loss function, same as you would in least squares linear regression.

Answer 2

@Dave's suggestion is known as the pinball loss, and it is precisely the standard loss function for quantile predictions.

For references, take a look at Koenker's textbook Quantile Regression, or Gneiting (2011, "Quantiles as optimal point forecasts", IJF). We also have a number of threads here at CV.

There are Python implementations in scikit-learn and TensorFlow.

Answer 3

You commented that pinball loss might be hard to interpret compared to something like $R^2$. Fortunately, we can draw an analogy to $R^2$ to get a metric that is similar.

In linear regression with square loss, we are estimating the conditional mean of $y$. In the absence of any information that could explain variability in $y$ (no features), a reasonable naïve model is to predict the pooled mean of $y$ every time, so $\bar y$. This gives some kind of baseline performance to which we can compare models that want to tighten up the estimation of the conditional mean, and $R^2$ has an interpretation as this.

$$ R^2=1-\dfrac{\text{ Square loss of model of interest }}{\text{ Square loss of baseline model }} $$

Do something similar with your pinball loss, where the baseline model always predicts the pooled $75$th percentile of $y$.

$$ R^2=1-\dfrac{\text{ Pinball loss of model of interest }}{\text{ Pinball loss of baseline model }} $$

This idea of comparing your performance to the performance of a baseline model is used elsewhere, such as in McFadden’s $R^2$ for logistic regression.