12
$\begingroup$

Quantile regression allows to estimate a conditional quantile for y (like e.g. the median of y,...) from data x.

I do not see any distributional assumptions about y being made. This seems in contrast to maximum likelihood estimation which starts with making an assumption about the distribution of y (e.g gaussian distribution).

Therefore the question: is quantile regression a maximum likelihood method?

If not, what is the broader term for methods like quantile regression?


Additional rewording: What is the rationale of the quantile loss function in quantile regression (of arbitrary complex models)? Does it rely on the specification of the distributional form of response variable? And, specifically, is the quantile loss (somehow) a (log) likelihood function?

$\endgroup$
1

3 Answers 3

15
$\begingroup$

You seem to confuse two closely-related yet very different concepts: a regression model (which is a specification of a statistical model) and a parameter estimation method (which essentially is a data-based objective function formulation and subsequent numerical procedures).

For simplicity, we restrict ourselves to the linear parametric family. A quantile regression (model) models (i.e., approximates) the $\tau$-th conditional quantile of the response $y$ given predictors $x$ as a linear function of parameters: \begin{align} Q_\tau(y|x) = \alpha + \beta'x. \tag{1} \end{align}

Likewise, a mean regression (model) models the conditional mean of the response $y$ given predictors $x$ as a linear function of parameters: \begin{align} E(y|x) = \alpha + \beta'x. \tag{2} \end{align}

In principle (of course, it is somewhat a quite narrow view), $(1)$ standalone is the heart of "quantile regression" -- we do not need to understand how parameters $\alpha$ and $\beta$ will be estimated to specify a quantile regression model.

The parameter estimation problem kicks in when a sample $\{(y_i, x_i): i = 1, \ldots, n\}$ is observed. As you may have already known, typically $\alpha$ and $\beta$ are estimated by minimizing the sum of check function: \begin{align} (\hat{\alpha}, \hat{\beta}) = \operatorname{argmin}_{\alpha, \beta}\sum_{i = 1}^n \rho_\tau(y_i - \alpha - \beta'x_i), \tag{3} \end{align} which is numerically implemented by the linear programming or interior point algorithm. Of course, this is one of many parameter estimation methods (when $\tau = 0.5$, this is usually referred to as Least Absolute Deviation Estimation), which, as you stated, does not require any distributional assumption of $y$.

The MLE, as another parameter estimation method, on the other hand, can be carried out only if one specifies the complete conditional distribution of $y$. That means, to do maximum likelihood estimation, you will need to specify a statistical model that is more granular than regression models such as $(1)$ or $(2)$ (it is more granular because the complete distribution function contains much more information than the quantile function or the mean function only. In fact, both $Q_\tau(y|x)$ and $E(y|x)$ can be derived probabilistically from the distribution of $y|x$). For example, a statistical model like \begin{align} y | x \sim f(\alpha + \beta'x; \theta), \tag{4} \end{align} where $f$ is some known density function with additional parameter $\theta$. Model $(4)$ then entails the likelihood function (assuming the observations are i.i.d.) \begin{align} L(\alpha, \beta; \theta) = \prod_{i = 1}^nf(\alpha + \beta x_i; \theta), \end{align} which can be maximized over the parameter space to determine MLE.

It is well-known that when $f$ is Gaussian, Model $(4)$ implies the mean regression model $(2)$, and when $f$ is (asymmetrical) Laplacian, Model $(4)$ implies the quantile regression model $(1)$. For other conditional distributions $f$, in general $(1)$ or $(2)$ are not nested in $(4)$ (that is, neither $Q_\tau(y|x)$ nor $E(y|x)$ admits the simple linear form $\alpha + \beta'x$ under $(4)$).

In summary, the question "Is quantile regression a maximum likelihood method?" is somewhat ill-posed because the former is a statistical model while the latter is a parameter estimation method that depends on a more granular statistical model than the quantile regression model. In this sense, I do not think these two concepts are comparable. If by "what is the broader term for methods like quantile regression?", you meant "what is the parameter estimation method typically used to estimate $(\alpha, \beta)$ in $(1)$?", then the answer is $(3)$ -- I am not sure if there is a universally accepted term for this minimization problem, but it may be OK to call it "least $L^1$ estimation", in view of $\rho_\tau(t) = t(\tau - I_{(-\infty, 0)}(t))$ is a piecewise linear function.

$\endgroup$
10
  • $\begingroup$ Do you or does anyone else know what distributions correspond to quantiles other than the median? $\endgroup$ Commented Mar 1, 2023 at 2:44
  • 1
    $\begingroup$ @BigBendRegion I couldn't follow your question very well. Can you elaborate? $\endgroup$
    – Zhanxiong
    Commented Mar 1, 2023 at 2:51
  • 3
    $\begingroup$ The asymmetric Laplace distribution has a quantile that is the MLE of a parameter: en.wikipedia.org/wiki/… $\endgroup$ Commented Mar 1, 2023 at 4:11
  • $\begingroup$ Common terms for the loss function used in quantile regression are "pinball loss", or "quantile, "linlin", "hinge", "tick" or "newsvendor loss". Any of these search terms will likely get more hits than "least $L^1$ estimation" or "asymmetric Laplace loss" or similar. $\endgroup$ Commented Mar 1, 2023 at 7:14
  • 2
    $\begingroup$ @Ggjj11 No, like quantile loss function, the squared loss function also does not rely on the normality assumption, it is derived from $\bar{Y} = \operatorname{argmin}_u E[(Y - u)^2]$ for any distribution of $Y$. The relationship between normal and squared loss function is precisely analogous to the relationship between asymmetric Laplace and quantile loss. $\endgroup$
    – Zhanxiong
    Commented Mar 1, 2023 at 22:40
2
$\begingroup$

It depends on the loss function you're trying to minimize. In MLE you're not always assuming that the distribution is gaussian. For instance, there's a relation between the distribution you assume and the loss function you try to minimize

  • Gaussian distribution $\propto e^{-(x-\mu)^2}$ implies $L^2$ loss
  • Laplace distribution $\propto e^{-|x-\mu|}$ implies $L^1$ loss

In the case of Gaussian distribution

$$ P(y|x) = N(y| f(x;\theta), \sigma^2) $$

where $\sigma$ is fixed. Therefore the likelihood estimation is is

$$ \theta^* = \text{arg max}_\theta \prod_i P(y_i|x_i) = \text{arg max}_\theta \sum_i \log P(x_i | y_i) $$

and therefore

$$ \theta^* = \text{arg max} -n \log \sigma - \frac{n}{2} \log (2\pi) -\frac{1}{2} \sum_i \left(\frac{y_i - f(x_i, \theta)}{\sigma}\right)^2 $$

which removing constant terms is

$$ \theta^* = \text{argmax}_\theta -\sum_i \left(y_i - f(x_i, \theta)\right)^2 $$

which is the standard $L^2$ loss function for regression problems. So, as you can see by defining a loss function you're also defining which distribution you assume of your data.

In the case of quantile regression is the same. Depending on your loss function you'll implicitly be specifying an underlying distribution.

$\endgroup$
2
  • $\begingroup$ Thank you! I found that is the asymmetric Laplace distribution which is used to construct the the likelihood function, and therefore the log likelihood function, which is used in quantile regression. I am curious for the reasons why the asymmetric Laplace distribution is useful for modeling quantiles. Which property is exploited here? $\endgroup$
    – Ggjj11
    Commented Mar 1, 2023 at 18:50
  • $\begingroup$ whuber argued here stats.stackexchange.com/questions/251600/… that the quantile loss arises automatically from basic considerations. Where did he (implicitly) assume an asymmetric Laplace distribution? I cannot see this. $\endgroup$
    – Ggjj11
    Commented Mar 1, 2023 at 21:53
1
$\begingroup$

Quantile Regression is not necessarily a maximum likelihood method (while it can be when using a working likelihood like the asymmetric Laplace distribution).

Performing empirical risk minimization with the quantile loss is not (necessarily) a maximum likelihood estimation method.

You can simply minimize this special risk and can show for arbitrary distributions that it result in an estimation of the quantile: https://en.m.wikipedia.org/wiki/Empirical_risk_minimization

The proof for this goes by

  1. defining the risk as the expected loss function: $R_\tau = \int_R dy p(y) w_\tau(y,\hat{y}) |y-\hat{y}|$
  2. splitting each integral explicitly to resolve the absolute value $|y-\hat{y}|$
  3. taking the derivative with respect to $\hat{y}$ and setting it to zero
  4. identify the definition of the $\tau$ quantile

All this works without assuming a special probability density p, so it is definitely not a maximum likelihood estimation.

This is also how we can show that the expected mean squared minimization estimates the (conditional) mean and the mean absolute error estimates the (conditional) median.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.