0
$\begingroup$

What is the value of $a$ and $b$ that minimizes the $L_1$ and $L_2$ norm, respectively? $$ \min_{a} \mathrm{E} \left| X - a \right| $$ I was told that median is the solution. How do you solve it? Why can't it be the mean instead?

$$\mathrm{E} (X - b) ^ 2$$ I was told that mean $\mu$ is the solution to the squared error. How do you solve it?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
2
$\begingroup$

Loss: $$L_1(\hat y,y)=|\hat y-y|$$

Suppose that $$\hat y=y+\varepsilon,$$ where the probability density of error $\varepsilon$ is $f(\varepsilon)$.

Now, we can solve the problem $$\min_{\hat y} E[L_1(\hat y,y)]$$ using first order condition (FOC): $$\partial/\partial\hat y E[L_1(\hat y,y)]=0$$ $$\partial/\partial\hat y E[L_1(\hat y,y)]= E[\partial/\partial\hat y |\hat y-y|]= \int_{-\infty}^{\hat y-y}f(e)de-\int_{\hat y-y}^{\infty}f(e)de =F(\hat y-y)-(1-F(\hat y-y))=2F(\hat y-y)-1=0$$ Where $F(.)$ is the cumulative distribution function. Also I substituted $\hat y-y$ with $e$ in the integrals.

You can see that the FOC is satisfied when $F(\hat y-y)=1/2$, i.e. when the $\hat y$ is at the median of the distribution.

You can do the same for $L_2$ to show that it requires the mean. Loss: $$L_2(\hat y,y)=(\hat y-y)^2$$

Suppose that $$\hat y=y+\varepsilon,$$ where the probability density of error $\varepsilon$ is $f(\varepsilon)$.

Now, we can solve the problem $$\min_{\hat y} E[L_2(\hat y,y)]$$ using first order condition (FOC): $$\partial/\partial\hat y E[L_2(\hat y,y)]=0$$ $$\partial/\partial\hat y E[L_2(\hat y,y)]= E[\partial/\partial\hat y (\hat y-y)^2]= E[2 (\hat y-y)]= 2 (\hat y-E[y])]=0$$

You can see that the FOC is satisfied when $\hat y=E[y]$, i.e. when the $\hat y$ is at the mean of the distribution.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Is there a course I can learn this technique? This is pretty cool. I’m taking a probability course using Casella and Berger textbook. But, the book example do not cover this. $\endgroup$
    – user13985
    Commented Oct 5, 2017 at 4:32
  • $\begingroup$ This type of stuff is taught in forecasting courses but all you need to know is how to setup the problem. The rest is a simple calculus. I'll update the answer with a link to a book or paper later $\endgroup$
    – Aksakal
    Commented Oct 5, 2017 at 12:12
  • $\begingroup$ Granger's paper is a good intro into how cost function are used in forecasting to come up with optimal forecasts, he has example for quadratic loss (cost) leading to the mean $\endgroup$
    – Aksakal
    Commented Oct 5, 2017 at 18:37
  • $\begingroup$ Can you explain why the middle number minimizes the absolute difference? Why not the mean? $\endgroup$
    – user13985
    Commented Oct 5, 2017 at 18:57
  • $\begingroup$ @user13985 $F(x)=1/2$ is a definition of the median, i.e. when the probability below and above x is the same 50%. For symmetrical distributions such as Gaussian, median and mean are the same $\endgroup$
    – Aksakal
    Commented Oct 5, 2017 at 19:18
1
$\begingroup$

The squared error is the simpler case, and a classic argument, so I will reproduce it here.

Let $\mu = E[X]$. Then we can use an add and subtract trick

$$ E[(X - b)^2] = E[(X - \mu + \mu - b)^2] = E[(X - \mu)^2] + 2 E[(X - \mu)(\mu - b)] + 2 E[(\mu - b)^2] $$

Now in the middle term, $\mu - b$ is a constant, so it can come outside of the expectation

$$ E[(X - \mu)(\mu - b)] = (\mu - b) E[X - \mu] = (\mu - b)(E[X] - \mu) = 0 $$

So, all together

$$ E[(X - b)^2] = E[(X - \mu)^2] + 2 E[(\mu - b)^2] $$

This is clearly minimized when $b = \mu$.

The median case is more elusive, but still elementary, and also deserving of classic status. You can look here for some proofs. Both André's and Brian's answers are simple and intuitive demonstrations.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Solution in link is using ordered statistics to get the median. I read it. It was very long. I was completely lost when I reached the end. $\endgroup$
    – user13985
    Commented Oct 5, 2017 at 17:30
  • $\begingroup$ Ok... That tends to happen with mathematics. You should always anticipate that you will need to read something multiple times and maybe take a walk and think about it. I think the idea there is simple and beautiful, but you are free to disagree! $\endgroup$
    – Matthew Drury
    Commented Oct 5, 2017 at 18:47
  • $\begingroup$ Why does the median minimize the absolute difference? What's the intuition in simple words? $\endgroup$
    – user13985
    Commented Oct 5, 2017 at 19:01
  • $\begingroup$ Im not sure there is an intuition in simple words for either case, why do you believe a simple reason exists? $\endgroup$
    – Matthew Drury
    Commented Oct 5, 2017 at 19:20
  • $\begingroup$ I read this post. They used median to explain it. math.stackexchange.com/questions/85448/… $\endgroup$
    – user13985
    Commented Oct 5, 2017 at 19:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.