16
$\begingroup$

This question already has an answer here:

I do not understand the intuition behind why the median is the best estimate if we are going to judge prediction accuracy using the Mean Absolute Error. Let's say you have a random variable X and you want to predict what the next X is. Let's denote your prediction as d.

Under Mean Squared Error, which is:

MSE = (X - d)^2

We know that expected MSE, or sum of MSEs, is minimized when d is equal to the mean or E[X]. This makes sense intuitively. The best predictor of a random variable is its mean.

However, under Mean Absolute Error, which is:

MAE = |X - d|

The expected MAE or sum of MAEs is minimized when d is equal to the median of the random variable. While the book I am reading has a fancy proof to show why this is the case, intuitively I don't understand why the median would be the best predictor. I also don't understand why the mean (or median) wouldn't be the best choice for both.

$\endgroup$

marked as duplicate by Stephan Kolassa, mkt, Sycorax, mdewey, Frans Rodenburg Sep 25 at 2:20

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Why does it make intuitive sense that the mean minimizes MSE? You’re right, but if I didn’t know that, I could believe that median does it. $\endgroup$ – Dave Sep 22 at 0:22
  • 1
    $\begingroup$ As a simple concrete example, try minimizing the absolute error for the dataset $\{-1, 0, 3 \}$, which has median 0 and mean 1. By "sliding" your estimate around from left to right, you should gain some intuition for why the median is the best choice. $\endgroup$ – tddevlin Sep 22 at 6:31
  • $\begingroup$ @Dave I guess they way I thought about it is, if on average X produces E[X], and if you estimate is E[X], then on average the MSE will be 0 so it is minimized. Not sure if that's the right intuition but you can see why when I use that logic on MAE, it doesn't make sense. $\endgroup$ – confused Sep 22 at 13:57
29
$\begingroup$

Here is an intuitive argument with light math. Let's say we have a $d$ claiming to be minimizing the MAE of points $x_i$. And, let's say we have $n_l$ and $n_r$ points on its left and right. If we move $d$ slightly left, i.e. an amount of $\Delta$, then all the absolute differences on the left will decrease by $\Delta$, and all the absolute differences on the right will increase by $\Delta$, leading to a net decrease of $(n_l-n_r)\Delta$ in MAE. If $n_l\neq n_r$, $d$ always have incentive to move either left or right, because each move either decreases or increases the MAE. For example, if $n_r<n_l$, then we move left because the net decrease in MAE is $(n_l-n_r)\Delta$, and if $n_l<n_r$ we move right because the net decrease will be $(n_r-n_l)\Delta$. This continues until we reach $n_l=n_r$, which is satisfied by the median.

$\endgroup$
6
$\begingroup$

gunes has already presented a wonderful answer with simple formulas. Here is my a numerical example to test it: consider the set {1, 1, 1, 1, 1, 1, 1, 1, 1, 11}; that is, nine 1s and a single 11. The mean is 2, the median is 1.

When you consider the sum of absolute values as the sum of distances, the median will have 0 distance to nine values but a distance of 10 to the final value, making a total sum of 10. By moving our comparison value a single unit to 2 (the mean), we will increase the sum of distances by 1 from each of the nine values and decrease it by 1 from the single final value, making a total sum of 17.

Applying the formulas from gunes, when you move right from the median by any small value of $\Delta$, you add $9 * \Delta$ and subtract $1 * \Delta$, which means the sum increases by a total of $8 * \Delta$. If you move left then the sum increases by a total of $10 * \Delta$.

By intuition, you stop moving left or right when the number of values to the left and to the right are equal. This can be the median (middle value) for an odd number of values or anywhere between the two middle values when there are an even number of values.

To demonstrate the final point: consider the set {1, 2, 3, 4}. Here the median is 2.5 by definition. But you can use an point between the two middle values (inclusive) of 2 and 3. The sum of distances would be 4 for the [2, 3] range (for 2, 2.5, 3, or anything in between).

$\endgroup$
  • $\begingroup$ I thought of that example too, but then I wondered why doesn't MSE follow the same logic. 9/10 the squared deviations will be 0 while 1/10 the squared deviation will be 100. I guess "squaring" it does something that magnifies the differences, and so really I should be digging into understanding the properties of exponents to understand the intuition, maybe. $\endgroup$ – confused Sep 22 at 14:02
  • $\begingroup$ +1 for the examples. $\endgroup$ – gunes Sep 22 at 20:51

Not the answer you're looking for? Browse other questions tagged or ask your own question.