I'm trying to find a primer on a topic that I'm sure must have been studied, but I can't find anything on. Suppose we'd like to do regression in a supervised learning setting to learn the expected value of $y$ given $x$ i.e. $\mathbb{E}_{p(y|x)}[y]$. Rather than learning the expected value directly, suppose I instead perform quantile regression to learn $Q$ uniformly distributed quantiles and then combine the quantiles to estimate the desired expected value. Has anyone studied this algorithm? Does it have a name? Is there an advantage to be had by taking this circuitous route? For instance, each quantile may be biased, but their combination might have smaller variance?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ @whuber , you've been tremendously helpful. Would you happen to know? $\endgroup$– Rylan SchaefferJun 23, 2020 at 20:04
-
$\begingroup$ Do you mean $\mathbb{E}(y|x)$ instead of $\mathbb{E}(p(y|x))$? Or otherwise, how is the latter the expected value of $y$ given $x$? Also, what does the subscript $_x$ stand for in the expectation? $\endgroup$– Richard HardyJun 23, 2020 at 20:31
-
$\begingroup$ I think I meant $\mathbb{E}_{p(y|x)}[y]$ $\endgroup$– Rylan SchaefferJun 23, 2020 at 20:33
-
$\begingroup$ Thank you for the clarification. $\endgroup$– Richard HardyJun 23, 2020 at 20:34
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
This is known as model-averaged quantile regression. It was discussed in Koenker "A note on L-estimates for linear models" (1984), though its origins stem from as early as the seminal paper of Koenker & Bassett "Regression quantiles" (1978). The 1984 paper actually proposes composite quantile regression, a natural competitor/counterpart of model-averaged quantile regression; the credit apparently goes to R.V.Hogg rather than Koenker himself.
Both types of quantile regression are discussed quite extensively in Koenker's monograph "Quantile Regression" (2005); for the model-averaged one, see Chapter 5. There is actually an equivalence between the two if unequal weights are allowed for. A recent comparison between the two estimators is provided in Bloznelis et al. "Composite versus model-averaged quantile regression" (2019).
It has been established that the model-averaged as well as the composite estimator can beat the OLS estimator of $\beta$ in a linear regression model in terms of asymptotic efficiency for certain error distributions. Actually, the composite estimator with equal weights has some nice efficiency guarantees; given some assumptions, the asymptotic relative efficiency of the composite quantile estimator with equal weights is at least ~70% that of an OLS estimator, but it can be much more efficient than it (Zou & Yuan "Composite quantile regression and the oracle model selection theory" (2008)).
answered Jun 23, 2020 at 20:30
-
$\begingroup$ Wonderful. Thank you so much for the references! I'll take a look. $\endgroup$ Jun 23, 2020 at 20:31
-
$\begingroup$ Quick follow-up question - do you have a reference for model-averaged and/or composite expectile regression? $\endgroup$ Jun 24, 2020 at 14:26
-
1$\begingroup$ @RylanSchaeffer, unfortunately, I do not. I guess expectile regression is less popular than quantile regression, so there may be fewer results for it. Also, I wonder if they have anyone like Koenker working on expectile regression; Koenker seems to be really knowledgeable about quantile regression and to have developed the field, together with collaborators, to a quite advanced stage. (I love his monograph, even though I cannot follow all the details.) Please ping me if you find out, I will be curious to hear. $\endgroup$ Jun 24, 2020 at 14:59
-
$\begingroup$ Can I ask for a citation on "It has been established that the model-averaged ... estimator can beat the OLS estimator of 𝛽 in a linear regression model in terms of asymptotic efficiency for certain error distributions. " I think that's exactly what I'm looking for. $\endgroup$ Jun 27, 2020 at 18:57
-
1$\begingroup$ @RylanSchaeffer, the 2008 paper establishes that composite QR beats OLS, while there are multiple references for the fact that composite QR and weighted model-averaged QR are equivalent (or weighted composite QR and model-averaged QR are equivalent); I think Koenker's monograph covers this. There should also be some relevant references in the 2019 paper. $\endgroup$ Jun 27, 2020 at 19:35