# Which metric to use to evaluate Quantile Regression?

I have a prediction problem for which I want to predict the 75% Quantile using Quantile Regression. I am a little bit confused on how to evaluate this model (and also compare different models).

If I understand this approach correctly, for given prediction y_hat the following is true:

1. With a chance of 75%, the value of y_hat is lower than the actual value of y
2. With a chance of 25%, the value of y_hat is higher than the actual value of y

In order to evaluate this model, intuitively I would would compare the overestimations in the test to the underestimations. An "ideal" model would "underestimate" in 25% of the cases (i.e. the value of y_hat is higher than the actual value of y).

Is this a valid approach?

Because I find it somehow confusing to not evaluate regression metrics like MAE, RMSE, Rsquared, etc. But these metrics do no seem suitable here (because we deliberately overestimate). Is there some asymmetric evaluation metric I am missing for this case? Happy if anyone can help clarify my confusion.

Note: I am aware of this post and the respective papers linked there. But is there a corresponding (python) implementation?

• Quantile regression has a loss function. $$L_{\tau}(y_i, \hat y_i) = \begin{cases} \tau\vert y_i - \hat y_i\vert, & y_i - \hat y_i \ge 0 \\ (1 - \tau)\vert y_i - \hat y_i\vert, & y_i - \hat y_i < 0 \end{cases}$$ (Then you add up the values for each index $i$ to get a loss for the entire regression, same as you would for absolute loss or square loss.) Why not use it?
– Dave
Jun 27, 2022 at 15:48

Quantile regression at quantile $$\tau$$ has a loss function, often called "pinball loss". Let

$$l_{\tau}(y_i, \hat y_i) = \begin{cases} \tau\vert y_i - \hat y_i\vert, & y_i - \hat y_i \ge 0 \\ (1 - \tau)\vert y_i - \hat y_i\vert, & y_i - \hat y_i < 0 \end{cases}$$

Then we add up each individual $$l$$ to get a loss $$L$$ for the whole model. $$L_{\tau}(y, \hat y) = \sum_{i=1}^n l_{\tau}(y_i, \hat y_i)$$

Use the loss function, same as you would in least squares linear regression.

• I am aware of pinball loss...but was a bit hesitant to use it as it is quite hard to interpret imo (compared to MAPE, R2, RMSE, even to MSE)...had this post in mind and was hoping for a more interpretable metric (not loss function), but of course you are right, why not use pinball loss for evaluation.
– bk_
Jun 28, 2022 at 7:11
• @ Dave: great answer! why is it called pinball loss? Apr 21, 2023 at 3:38
• @stats_noob Thanks! As far as your question goes, it might be posted somewhere on here. If not, that sounds like a reasonable question to post. (Hint: graph it and think about the physics of a pinball that ricochets.)
– Dave
Apr 21, 2023 at 4:06

@Dave's suggestion is known as the pinball loss, and it is precisely the standard loss function for quantile predictions.

For references, take a look at Koenker's textbook Quantile Regression, or Gneiting (2011, "Quantiles as optimal point forecasts", IJF). We also have a number of threads here at CV.

There are Python implementations in scikit-learn and TensorFlow.

You commented that pinball loss might be hard to interpret compared to something like $$R^2$$. Fortunately, we can draw an analogy to $$R^2$$ to get a metric that is similar.

In linear regression with square loss, we are estimating the conditional mean of $$y$$. In the absence of any information that could explain variability in $$y$$ (no features), a reasonable naïve model is to predict the pooled mean of $$y$$ every time, so $$\bar y$$. This gives some kind of baseline performance to which we can compare models that want to tighten up the estimation of the conditional mean, and $$R^2$$ has an interpretation as this.

$$R^2=1-\dfrac{\text{ Square loss of model of interest }}{\text{ Square loss of baseline model }}$$

Do something similar with your pinball loss, where the baseline model always predicts the pooled $$75$$th percentile of $$y$$.

$$R^2=1-\dfrac{\text{ Pinball loss of model of interest }}{\text{ Pinball loss of baseline model }}$$

This idea of comparing your performance to the performance of a baseline model is used elsewhere, such as in McFadden’s $$R^2$$ for logistic regression.

• Yes that makes a lot of sense. Thanks for pointing that out. Does this modified $R^2$ have a name or is there literature on it?
– bk_
Jun 28, 2022 at 13:42
• is it the $D^2$ pinball score (see sklearn implementation)?
– bk_
Jun 28, 2022 at 13:49
• @bk_ There are likely differences between how sklearn does it and how I would do it when it comes to out-of-sample assessments, and I disagree with their assertion that it represents the fraction of pinball loss explained ($R^2$ only has its "proportion of variance explained" interpretation under specific situations), but that seems to be getting at this, yes. // My qualms with how sklearn implements the out-of-sample assessments really warrants a distinct question, though the gist is explained here for $R^2$ (analogous argument for $D^2$).
– Dave
Jun 28, 2022 at 14:03
• @bk_ Here is the question I finally posted about how to motivate the $R^2$ definition used by sklearn.
– Dave
Feb 20, 2023 at 16:24