3
$\begingroup$

I'm trying to find a primer on a topic that I'm sure must have been studied, but I can't find anything on. Suppose we'd like to do regression in a supervised learning setting to learn the expected value of $y$ given $x$ i.e. $\mathbb{E}_{p(y|x)}[y]$. Rather than learning the expected value directly, suppose I instead perform quantile regression to learn $Q$ uniformly distributed quantiles and then combine the quantiles to estimate the desired expected value. Has anyone studied this algorithm? Does it have a name? Is there an advantage to be had by taking this circuitous route? For instance, each quantile may be biased, but their combination might have smaller variance?

$\endgroup$
4
  • $\begingroup$ @whuber , you've been tremendously helpful. Would you happen to know? $\endgroup$ Commented Jun 23, 2020 at 20:04
  • $\begingroup$ Do you mean $\mathbb{E}(y|x)$ instead of $\mathbb{E}(p(y|x))$? Or otherwise, how is the latter the expected value of $y$ given $x$? Also, what does the subscript $_x$ stand for in the expectation? $\endgroup$ Commented Jun 23, 2020 at 20:31
  • $\begingroup$ I think I meant $\mathbb{E}_{p(y|x)}[y]$ $\endgroup$ Commented Jun 23, 2020 at 20:33
  • $\begingroup$ Thank you for the clarification. $\endgroup$ Commented Jun 23, 2020 at 20:34

1 Answer 1

2
$\begingroup$

This is known as model-averaged quantile regression. It was discussed in Koenker "A note on L-estimates for linear models" (1984), though its origins stem from as early as the seminal paper of Koenker & Bassett "Regression quantiles" (1978). The 1984 paper actually proposes composite quantile regression, a natural competitor/counterpart of model-averaged quantile regression; the credit apparently goes to R.V.Hogg rather than Koenker himself.

Both types of quantile regression are discussed quite extensively in Koenker's monograph "Quantile Regression" (2005); for the model-averaged one, see Chapter 5. There is actually an equivalence between the two if unequal weights are allowed for. A recent comparison between the two estimators is provided in Bloznelis et al. "Composite versus model-averaged quantile regression" (2019).

It has been established that the model-averaged as well as the composite estimator can beat the OLS estimator of $\beta$ in a linear regression model in terms of asymptotic efficiency for certain error distributions. Actually, the composite estimator with equal weights has some nice efficiency guarantees; given some assumptions, the asymptotic relative efficiency of the composite quantile estimator with equal weights is at least ~70% that of an OLS estimator, but it can be much more efficient than it (Zou & Yuan "Composite quantile regression and the oracle model selection theory" (2008)).

$\endgroup$
6
  • $\begingroup$ Wonderful. Thank you so much for the references! I'll take a look. $\endgroup$ Commented Jun 23, 2020 at 20:31
  • $\begingroup$ Quick follow-up question - do you have a reference for model-averaged and/or composite expectile regression? $\endgroup$ Commented Jun 24, 2020 at 14:26
  • 1
    $\begingroup$ @RylanSchaeffer, unfortunately, I do not. I guess expectile regression is less popular than quantile regression, so there may be fewer results for it. Also, I wonder if they have anyone like Koenker working on expectile regression; Koenker seems to be really knowledgeable about quantile regression and to have developed the field, together with collaborators, to a quite advanced stage. (I love his monograph, even though I cannot follow all the details.) Please ping me if you find out, I will be curious to hear. $\endgroup$ Commented Jun 24, 2020 at 14:59
  • $\begingroup$ Can I ask for a citation on "It has been established that the model-averaged ... estimator can beat the OLS estimator of 𝛽 in a linear regression model in terms of asymptotic efficiency for certain error distributions. " I think that's exactly what I'm looking for. $\endgroup$ Commented Jun 27, 2020 at 18:57
  • 1
    $\begingroup$ @RylanSchaeffer, the 2008 paper establishes that composite QR beats OLS, while there are multiple references for the fact that composite QR and weighted model-averaged QR are equivalent (or weighted composite QR and model-averaged QR are equivalent); I think Koenker's monograph covers this. There should also be some relevant references in the 2019 paper. $\endgroup$ Commented Jun 27, 2020 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.