35
$\begingroup$

I am hoping to get an intuitive, accessible explanation of quantile regression.

Let's say I have a simple dataset of outcome $Y$, and predictors $X_1, X_2$.

If, for example, I run a quantile regression at .25,.5,.75, and get back $\beta_{0,.25},\beta_{1,.25}...\beta_{2,.75}$.

Are the $\beta$ values found by simply ordering the $y$ values, and performing a linear regression based on the examples which are at/near the given quantile?

Or do all of the samples contribute to the $\beta$ estimates, with descending weights as the distance from the quantile increases?

Or is it something totally different? I've yet to find an accessible explanation.

Improve this question
$\endgroup$
2

2 Answers 2

Reset to default
35
$\begingroup$

I recommend Koenker & Hallock (2001, Journal of Economic Perspectives) and Koenker's textbook Quantile Regression.

  1. The starting point is the observation that the median of a data set minimizes the sum of absolute errors. That is, the 50% quantile is a solution to a particular optimization problem (to find the value that minimizes the sum of absolute errors).
  2. From this, it is easy to find that any $\tau$-quantile is the solution to a specific minimization problem, namely to minimize a sum of asymmetrically weighted absolute errors, with weights that depend on $\tau$.
  3. Finally, to make the step to regression, we model the solution to this minimization problem as a linear combination of predictor variables, so now the problem is one of finding not a single value, but a set of regression parameters.

So your intuition is quite correct: all of the samples contribute to the $\beta$ estimates, with asymmetric weights depending on the quantile $\tau$ we aim for.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Regarding your point 1), wouldn't this only be true assuming Y is symmetrically distributed? If Y is skewed like {1, 1, 2, 4, 10}, the median 2 certainly wouldn't minimize absolute error. Does quantile regression always assume Y is symmetrically distributed? Thanks! $\endgroup$
    – Ben
    Commented Jan 10, 2018 at 17:56
  • 2
    $\begingroup$ @Ben: no, symmetry is not required. The key point is that the median minimizes the expected absolute error. If you have a discrete distribution with values 1, 2, 4, 10 and probabilities 0.4, 0.2, 0.2, 0.2, then a point summary of 2 does indeed minimize the expected absolute error. A simulation is just a few lines of R code: foo <- sample(x=c(1,2,4,10),size=1e6,prob=c(.4,.2,.2,.2),replace=TRUE); xx <- seq(1,10,by=.1); plot(xx,sapply(xx,FUN=function(yy)mean(abs(yy-foo))),type="l") $\endgroup$
    – Stephan Kolassa
    Commented Jan 10, 2018 at 20:56
  • $\begingroup$ (And yes, I should have been clearer in my answer, instead of discussing "sums".) $\endgroup$
    – Stephan Kolassa
    Commented Jan 10, 2018 at 21:06
29
$\begingroup$

The basic idea of quantile regression comes from the fact the the analyst is interested in distribution of data rather that just mean of data. Lets start with mean.

Mean regression fits a line of the form of $y=X\beta$ to the mean of data. In other words, $E(Y|X=x)=x\beta$. A general approach to estimate this line is using least square method, $\arg\min_\beta (y-x\beta)'(y-X\beta)$.

On the other hand median regression looks for a line that expect half of the data are on sides. In this case target function is $\arg\min_\beta |y-X\beta|$ where $|.|$ is the first norm.

Extending the idea of median to quantile results in Quantile regression. The idea behind is to find a line that $\alpha$-percent of data are beyond that.

Here you made a small mistake, Q-regression is not like finding a quantile of data then fit a line to that subset (or even the borders that is more challenging).

Q-regression looks for a line that split data into a qroup a $\alpha$ quantile and the rests. Target function, saying check function of Q-regression is $$ \hat\beta_\alpha=\arg\min_\beta \bigg\{\alpha |y-X\beta| I(y>X\beta) + (1-\alpha) |y-X\beta|I(y<X\beta)\bigg\}. $$

As you see this clever target function is nothing more that translating quantile to an optimization problem.

Moreover, as you see, Q-regression is defined for a certain quantie ($\beta_\alpha$) and then can be extended to find all quantiles. In other words, Q-regression can reproduce (conditional) distribution of response.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This answer is brilliant. $\endgroup$
    – Jinhua Wang
    Commented Oct 13, 2019 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.