Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

The solution to the problem:

$$ \min_{m} \; E[|m-X|] $$

is well known to be the median of $X$, but what does the loss function look like for other percentiles? Ex: the 25th percentile of X is the solution to:

$$ \min_{m} \; E[ L(m,X) ] $$

What is $L$ in this case?

share|improve this question

2 Answers 2

up vote 9 down vote accepted

Let $I$ be the indicator function: it is equal to $1$ for true arguments and $0$ otherwise. Pick $0\lt\alpha\lt 1$ and set

$$\Lambda_\alpha(x)=\alpha x\, I(x\ge 0) - (1-\alpha)x\, I(x\lt 0).$$

Figure

This figure plots $\Lambda_{1/5}$. It uses an accurate aspect ratio to help you gauge the slopes, which equal $-4/5$ on the left side and $+1/5$ on the right. In this case excursions above $0$ are heavily downweighted compared to excursions below $0$.

This is a natural function to try because it weights values $x$ that exceed $0$ differently than $x$ that are less than $0$. Let's compute the associated loss and then optimize it.

Writing $F$ for the distribution function of $X$ and setting $L_\alpha(m,x) = \Lambda_\alpha(x-m)$, compute

$$\eqalign{ \mathbb{E}_F(L_\alpha(m,X))&=\int_\mathbb{R} \Lambda_\alpha(x-m)dF(x)\\ &=\alpha\int_\mathbb{R} I(x\ge m)(x-m) dF(x) - (1-\alpha)\int_\mathbb{R} (x-m)I(x\lt m) dF(x)\\ &=\alpha\int_m^\infty(x-m)dF(x) - (1-\alpha)\int_{-\infty}^m(x-m) dF(x). }$$

Figure 2

As $m$ varies in this illustration with the Standard Normal distribution $F$, the total probability-weighted area of $\Lambda_{1/5}$ is plotted. (The curve is the graph of $\Lambda_{1/5}(x-m)dF(x)$.) The right-hand plot for $m=0$ most clearly shows the effect of downweighting the positive values, for without this downweighting the plot would be symmetric about the origin. The middle plot shows the optimum, where the total amount of blue ink (representing $\mathbb{E}_F(L_{1/5}(m,X))\ $) is as small as possible.

This function is differentiable and so its extrema can be found by inspecting the critical points. Applying the Chain Rule and the Fundamental Theorem of Calculus to obtain the derivative with respect to $m$ gives

$$\eqalign{ \frac{\partial}{\partial m}\mathbb{E}_F(L_\alpha(m,X))&=\alpha\left(0-\int_m^\infty dF(x)\right) - (1-\alpha)\left(0 - \int_{-\infty}^m dF(x)\right)\\ &= F(m) - \alpha. }$$

For continuous distributions this always has a solution $m$ which, by definition, is any $\alpha$ quantile of $X$. For non-continuous distributions this might not have a solution but there will be at least one $m$ for which $F(x)-\alpha\lt 0$ for all $x\lt m$ and $F(x)-\alpha\ge 0$ for all $x\ge m$: this also (by definition) is an $\alpha$ quantile of $X$.

Finally, because $\alpha\ne 0$ and $\alpha\ne 1$, it is clear that neither $m\to-\infty$ nor $m\to\infty$ will minimize this loss. That exhausts the inspection of the critical points, showing that $\Lambda_\alpha$ fits the bill.

As a special case, $\mathbb{E}_F(2L_{1/2}(m,X)) = \mathbb{E}_F\left(\left|m-x\right|\right)$ is the loss exhibited in the question.

share|improve this answer
    
I appreciate the effort you put in showing the expected loss is minimized by the correct point $m$. I was wondering how to do that myself for my own answer, but your explanation is good. (+1) –  Matthew Jun 11 at 14:57
2  
You've proved that pictures are worth 1000 words. Thanks @whuber =) –  Cam.Davidson.Pilon Jun 11 at 18:57

This article has your answer. To be specific, $$ L_{0.25}(m,X) = \left| \left( X - m \right) \left(0.25 - \mathbf{1}\{ X > m \} \right) \right|. $$ The loss function can be interpreted as 'balancing out' the different probability mass regions around $0.25$ through the subtraction $0.25 - \mathbf{1}\{ X > m \}$. For the median these mass regions are equal: $$ L_{0.5}(m,X) = \left| \left( X - m \right) \left(0.5 - \mathbf{1}\{ X > m \} \right) \right| = \left| \left( X - m \right) \times \pm 0.5 \right|, $$ making the loss function proportional (in the expectation the constant is neglibible) to $$ \left| X - m\right|, $$ which gives the desired conclusion for the median.

share|improve this answer
    
(+1) Well done! -- it was not obvious where to look for that Wikipedia article; you had to think of quantile regression. –  whuber Jun 11 at 14:56
    
Thanks, @Matthew, this is a great find. I like balancing out interpretation –  Cam.Davidson.Pilon Jun 11 at 18:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.