Can someone give the intuition behind Mean Absolute Error and the Median? [duplicate]

Question

I do not understand the intuition behind why the median is the best estimate if we are going to judge prediction accuracy using the Mean Absolute Error. Let's say you have a random variable $X$ and you want to predict what the next $X$ is. Let's denote your prediction as d.

Under Mean Squared Error, which is:

$\text{MSE} = (X - d)^2$

We know that expected MSE, or sum of MSEs, is minimized when $d$ is equal to the mean or $E[X]$. This makes sense intuitively. The best predictor of a random variable is its mean.

However, under Mean Absolute Error, which is:

$\text{MAE} = |X - d|$

The expected MAE or sum of MAEs is minimized when $d$ is equal to the median of the random variable. While the book I am reading has a fancy proof to show why this is the case, intuitively I don't understand why the median would be the best predictor. I also don't understand why the mean (or median) wouldn't be the best choice for both.

Answer 1

Here is an intuitive argument with light math. Let's say we have a $d$ claiming to be minimizing the MAE of points $x_i$. And, let's say we have $n_l$ and $n_r$ points on its left and right. If we move $d$ slightly left, i.e. an amount of $\Delta$, then all the absolute differences on the left will decrease by $\Delta$, and all the absolute differences on the right will increase by $\Delta$, leading to a net decrease of $(n_l-n_r)\Delta$ in MAE. If $n_l\neq n_r$, $d$ always have incentive to move either left or right, because each move either decreases or increases the MAE. For example, if $n_r<n_l$, then we move left because the net decrease in MAE is $(n_l-n_r)\Delta$, and if $n_l<n_r$ we move right because the net decrease will be $(n_r-n_l)\Delta$. This continues until we reach $n_l=n_r$, which is satisfied by the median.

Answer 2

gunes has already presented a wonderful answer with simple formulas. Here is my a numerical example to test it: consider the set {1, 1, 1, 1, 1, 1, 1, 1, 1, 11}; that is, nine 1s and a single 11. The mean is 2, the median is 1.

When you consider the sum of absolute values as the sum of distances, the median will have 0 distance to nine values but a distance of 10 to the final value, making a total sum of 10. By moving our comparison value a single unit to 2 (the mean), we will increase the sum of distances by 1 from each of the nine values and decrease it by 1 from the single final value, making a total sum of 17.

Applying the formulas from gunes, when you move right from the median by any small value of $\Delta$, you add $9 * \Delta$ and subtract $1 * \Delta$, which means the sum increases by a total of $8 * \Delta$. If you move left then the sum increases by a total of $10 * \Delta$.

By intuition, you stop moving left or right when the number of values to the left and to the right are equal. This can be the median (middle value) for an odd number of values or anywhere between the two middle values when there are an even number of values.

To demonstrate the final point: consider the set {1, 2, 3, 4}. Here the median is 2.5 by definition. But you can use an point between the two middle values (inclusive) of 2 and 3. The sum of distances would be 4 for the [2, 3] range (for 2, 2.5, 3, or anything in between).