Why is the Median Less Sensitive to Extreme Values Compared to the Mean? I am sure we have all heard the following argument stated in some way or the other:

*

*For a given set of measurements (e.g. heights of students), the mean of these measurements is more "prone" to be influenced by outliers compared to the median of these same measurements.
Conceptually, the above argument is straightforward to understand. The median is not directly calculated using the "value" of any of the measurements, but only using the "ranked position" of the measurements. On the other hand, the mean is directly calculated using the "values" of the measurements, and not by using the "ranked position" of the measurements. Therefore, a statistically larger number of outlier points should be required to influence the median of these measurements - compared to influence of fewer outlier points on the mean. For example: the average weight of a blue whale and 100 squirrels will be closer to the blue whale's weight, but the median weight of a blue whale and 100 squirrels will be closer to the squirrels.
Using the R programming language, we can see this argument manifest itself on simulated data:
library(plotly)
    set.seed(123)


d = data.frame(data = rnorm(20, 5,50), col = "non outlier")

dd = data.frame(data = rnorm(5,150, 10), col = "outlier")

my_data = rbind(d,dd)

> mean(d$data)
[1] 10.08877

> median(d$data)
[1] 17.11447

We can also plot this to get a better idea:
d1 = data.frame(data = mean(my_data$data), col = "mean")
# add "1" to the median so that it becomes visible in the plot
d2 = data.frame(data = median(my_data$data) +1 , col = "median")

new_data = rbind(my_data, d1, d2)


plot_ly(type = "scatter", mode = "markers", data = new_data, x = ~data, y = " ", color = ~col ) %>% layout(title = 'Effect of Outliers on Median vs Mean')


My Question:  In the above example, we can see that the median is less influenced by the outliers compared to the mean - but in general, are there any "statistical proofs" that shed light on this inherent "vulnerability" of the mean compared to the median?
Apart from the logical argument of measurement "values" vs. "ranked positions" of measurements - are there any theoretical arguments behind why the median requires larger valued and a larger number of outliers to be influenced towards the extremas of the data compared to the mean?
I am aware of related concepts such as Cooke's Distance (https://en.wikipedia.org/wiki/Cook%27s_distance) which can be used to estimate the effect of removing an individual data point on a regression model - but are there any formulas which show some relation between the number/values of outliers on the mean vs. the median?
Are there any theoretical statistical arguments that can be made to justify this logical argument regarding the number/values of outliers on the mean vs. the median?
 A: Changing an outlier doesn't change the median; as long as you have at least three data points, making an extremum more extreme doesn't change the median, but it does change the mean by the amount the outlier changes divided by n.
Adding an outlier, or moving a "normal" point to an extreme value, can only move the median to an adjacent central point. For instance, if you start with the data [1,2,3,4,5], and change the first observation to 100 to get [100,2,3,4,5], the median goes from 3 to 4. So not only is the a maximum amount a single outlier can affect the median (the mean, on the other hand, can be affected an unlimited amount), the effect is to move to an adjacently ranked point in the middle of the data, and the data points tend to be more closely packed close to the median. So, for instance, if you have nine points evenly spaced in Gaussian percentile, such as [-1.28, -0.84, -0.52, -0.25, 0, 0.25, 0.52, 0.84, 1.28]. The average separation between observations is 0.32, but changing one observation can change the median by at most 0.25. The median of a bimodal distribution, on the other hand, could be very sensitive to change of one observation, if there are no observations between the modes.
A: Extreme values influence the tails of a distribution and the variance of the distribution. This also influences the mean of a sample taken from the distribution.
Extreme values do not influence the center portion of a distribution. This means that the median of a sample taken from a distribution is not influenced so much.

Example
Below is an illustration with a mixture of three normal distributions with different means.
The mixture is 90% a standard normal distribution making the large portion in the middle and two times 5% normal distributions with means at $+ \mu$ and $-\mu$.
The value of $\mu$ is varied giving distributions that mostly change in the tails.

The consequence of the different values of the extremes is that the distribution of the mean (right image) becomes a lot more variable.


For large sample sizes
The sample variance of the mean will relate to the variance of the population:
$$Var[mean(x_n)] \approx \frac{1}{n} Var[x]$$
The sample variance of the median will relate to the slope of the cumulative distribution (and the height of the distribution density near the median)
$$Var[median(x_n)] \approx  \frac{1}{n} \frac{1}{4f(median(x))^2}$$
Example where the mean is less influenced by outliers
In general we have that large outliers influence the variance $Var[x]$ a lot, but not so much the density at the median $f(median(x))$.
But, it is possible to construct an example where this is not the case. If we mix/add some percentage $\phi$ of outliers to a distribution with a variance of the outliers that is relative $v$ larger than the variance of the distribution (and consider that these outliers do not change the mean and median), then the new mean and variance will be approximately
$$Var[mean(x_n)] \approx \frac{1}{n}  (1-\phi + \phi v) Var[x]$$
$$Var[mean(x_n)] \approx \frac{1}{n}  \frac{1}{4((1-\phi)f(median(x))^2}$$
So the relative change (of the sample variance of the statistics) are for the mean $\delta_\mu = (v-1)\phi$ and for the median $\delta_m = \frac{2\phi-\phi^2}{(1-\phi)^2}$. And we have $\delta_m > \delta_\mu$ if $$v < 1+ \frac{2-\phi}{(1-\phi)^2}$$
An example here is a continuous uniform distribution with point masses at the end as 'outliers'. The variance of a continuous uniform distribution is 1/3 of the variance of a Bernoulli distribution with equal spread. So $v=3$ and for any small $\phi>0$ the condition is fulfilled and the median will be relatively more influenced than the mean.
This is a contrived example in which the variance of the outliers is relatively small. This is done by using a continuous uniform distribution with point masses at the ends. So the outliers are very tight and relatively close to the mean of the distribution (relative to the variance of the distribution).
A: I'm going to say no, there isn't a proof the median is less sensitive than the mean since it's not always true. At least not if you define "less sensitive" as a simple "always changes less under all conditions". I'm told there are various definitions of sensitivity, going along with rules for well-behaved data for which this is true.
Say our data is 5000 ones and 5000 hundreds, and we add an outlier of -100 (or we change one of the hundreds to -100). The median jumps by 50 while the mean barely changes.
That seems like very fake data. So say our data is only multiples of 10, with lots of duplicates. It could even be a proper bell-curve. Then it's possible to choose outliers which consistently change the mean by a small amount (much less than 10), while sometimes changing the median by 10. Or simply changing a value at the median to be an appropriate outlier will do the same.
Or we can abuse the notion of outlier without the need to create artificial peaks. Take the 100 values 1,2 ... 100. Mean and median both 50.5. Then add an "outlier" of -0.1 -- median shifts by exactly 0.5 to 50, mean (5049.9/101) drops by almost 0.5 but not quite. Of course we already have the concepts of "fences" if we want to exclude these barely outlying outliers.
If feels as if we're left claiming the rule is always true for sufficiently "dense" data where the gap between all consecutive values is below some ratio based on the number of data points, and with a sufficiently strong definition of outlier.
A: An outlier is not precisely defined, a point can more or less of an outlier. You might say outlier is a fuzzy set where membership depends on the distance  $d$ to the pre-existing average. Call such a point a $d$-outlier. The key difference in mean vs median is that the effect on the mean of a introducing a $d$-outlier depends on $d$, but the effect on the median does not. Using Big-0 notation, the effect on the mean is $O(d)$, and the effect on the median is $O(1)$.
Btw "the average weight of a blue whale and 100 squirrels will be closer to the blue whale's weight"--this is not true.
A: Mathematical description/proof/viewpoint for special case
There is a short mathematical description/proof in the special case of

*

*Comparing the sensitivity in terms of the variance of the sample statistic. (this can be generalized to the 3rd central moment, and possibly other cost functions)

*Symmetric distributions in which case we can express the distribution of the median and mean in terms of integrals of the quantile functions in a similar way.

Let's assume that the distribution is centered at $0$ and the sample size $n$ is odd (such that the median is easier to express as a beta distribution).
Then in terms of the quantile function $Q_X(p)$ we can express
$$\begin{array}{rcrr}
Var[mean(X_n)] &=& \frac{1}{n}\int_0^1& 1 \cdot Q_X(p)^2 \, dp \\
Var[median(X_n)] &=&  \frac{1}{n}\int_0^1& f_n(p) \cdot Q_X(p)^2 \, dp
\end{array}$$
where $f(p) = \frac{n}{Beta(\frac{n+1}{2}, \frac{n+1}{2})} p^{\frac{n-1}{2}}(1-p)^{\frac{n-1}{2}}$
Below is a plot of $f_n(p)$ when $n = 9$ and it is compared to the constant value of $1$ that is used to compute the variance of the sample mean.

What the plot shows is that the contribution of the squared quantile function to the variance of the sample statistics (mean/median) is for the median larger in the center and lower at the edges.

Adding outliers versus changing outliers.
When we change outliers, then the quantile function $Q_X(p)$ changes only at the edges where the factor $f_n(p) < 1$ and so the mean is more influenced than the median.
When we add outliers, then the quantile function $Q_X(p)$ is changed in the entire range. So the median might in some particular cases be more influenced than the mean.
Example: Say we have a mixture of two normal distributions with different variances and mixture proportions. Then the change of the quantile function is of a different type when we change the variance in comparison to when we change the proportions.
Below is an example of different quantile functions where we mixed two normal distributions.

*

*The black line is the quantile function for the mixture of $90\%$ a distribution with $\sigma = 1$ and $\phi = 10\%$ a distribution with $\sigma_{outlier} = 2$.

*On the left we changed the proportion of outliers $\phi \in \lbrace 20 \%, 30 \%, 40 \% \rbrace$.

*On the right we changed the variance of outliers with $ \sigma_{outlier} \in \lbrace 4, 8, 16 \rbrace$.


The quantile function of a mixture is a sum of two components in the horizontal direction. Whether we add more of one component or whether we change the component will have different effects on the sum.

Generalizations

*

*The conditions that the distribution is symmetric and that the distribution is centered at 0 can be lifted. It will make the integrals more complex. $$\begin{array}{rcrr}
Var[mean(X_n)] &=& \frac{1}{n}\int_0^1& 1 \cdot (Q_X(p)-Q_(p_{mean}))^2 \, dp \\
Var[median(X_n)] &=&  \frac{1}{n}\int_0^1& f_n(p) \cdot (Q_X(p) - Q_X(p_{median}))^2 \, dp
\end{array}$$ now these 2nd terms in the integrals are different. We have $(Q_X(p)-Q_(p_{mean}))^2$ and $(Q_X(p) - Q_X(p_{median}))^2$. But we still have that the factor in front of it is the constant $1$ versus the factor $f_n(p)$ which goes towards zero at the edges.


*The condition that we look at the variance is more difficult to relax. I have made a new question that looks for simple analogous cost functions. But we could imagine with some intuitive handwaving that we could eventually express the cost function as a sum of multiple expressions $$mean: E[S(X_n)] =  \sum_{i}g_i(n) \int_0^1 1 \cdot h_{i,n}(Q_X) \, dp  \\ median: E[S(X_n)] =  \sum_{i}g_i(n) \int_0^1 f_n(p) \cdot h_{i,n}(Q_X) \, dp $$ where we can not solve it with a single term but in each of the terms we still have the $f_n(p)$ factor, which goes towards zero at the edges.
A: A helpful concept when considering the sensitivity/robustness of mean vs. median (or other estimators in general) is the breakdown point. This is the proportion of (arbitrarily wrong) outliers that is required for the estimate to become arbitrarily wrong itself.
Using this definition of "robustness", it is easy to see how the median is less sensitive:
At least HALF your samples have to be outliers for the median to break down (meaning it is maximally robust), while a SINGLE sample is enough for the mean to break down.
A: Actually, there are a large number of illustrated distributions for which the statement can be wrong!
Background for my colleagues, per Wikipedia on Multimodal distributions:

Bimodal distributions have the peculiar property that – unlike the unimodal distributions – the mean may be a more robust sample estimator than the median.[15] This is clearly the case when the distribution is U shaped like the arcsine distribution. It may not be true when the distribution has one or more long tails.

Here is another educational reference (from Douglas College) which is certainly accurate for large data scenarios:

In symmetrical, unimodal datasets, the mean is the most accurate measure of central tendency. For asymmetrical (skewed), unimodal datasets, the median is likely to be more accurate. For bimodal distributions, the only measure that can capture central tendency accurately is the mode.

So, evidently, in the case of said distributions, the statement is incorrect (lacking a specificity to the class of unimodal distributions).
A: A reasonable way to quantify the "sensitivity" of the mean/median to an outlier is to use the absolute rate-of-change of the mean/median as we change that data point.  To that end, consider a subsample $x_1,...,x_{n-1}$ and one more data point $x$ (the one we will vary).  If we denote the sample mean of this data by $\bar{x}_n$ and the sample median of this data by $\tilde{x}_n$ then we have:
$$\begin{align}
\text{Sensitivity of mean}
&\equiv \bigg| \frac{d\bar{x}_n}{dx} \bigg|
= \frac{1}{n}, \\[12pt]
\text{Sensitivity of median (} n \text{ odd)}
&\equiv \bigg| \frac{d\tilde{x}_n}{dx} \bigg| 
= \mathbb{I}(x = x_{((n+1)/2)} < x_{((n+3)/2)}), \\[12pt]
\text{Sensitivity of median (} n \text{ even)}
&\equiv \bigg| \frac{d\tilde{x}_n}{dx} \bigg| 
= \frac{1}{2} \cdot \mathbb{I}(x_{(n/2)} \leqslant x \leqslant x_{(n/2+1)} < x_{(n/2+2)}). \\[12pt]
\end{align}$$
In the trivial case where $n \leqslant 2$ the mean and median are identical and so they have the same sensitivity.  In the non-trivial case where $n>2$ they are distinct.  In this latter case the median is more sensitive to the internal values that affect it (i.e., values within the intervals shown in the above indicator functions) and less sensitive to the external values that do not affect it (e.g., an "outlier").
A: Ironically, you are asking about a generalized truth (i.e., normally true but not always) and wonder about a proof for it. If you want a reason for why outliers TYPICALLY affect mean more so than median, just run a few examples. Your light bulb will turn on in your head after that.
Step 1: Take ANY random sample of 10 real numbers for your example.
Step 2: Identify the outlier with a value that has the greatest absolute value.
Step 3: Add a new item (eleventh item) to your sample set and assign it a positive value number that is 1000 times the magnitude of the absolute value you identified in Step 2.
Step 4: Add a new item (twelfth item) to your sample set and assign it a negative value number that is 1000 times the magnitude of the absolute value you identified in Step 2.
Step 5: Calculate the mean and median of the new data set you have. Compare the results to the initial mean and median. Which one changed more, the mean or the median.
Step 6. Repeat the exercise starting with Step 1, but use different values for the initial ten-item set.
Again, did the median or mean change more? No matter what ten values you choose for your initial data set, the median will not change AT ALL in this exercise! You can use a similar approach for item removal or item replacement, for which the mean does not even change one bit. Clearly, changing the outliers is much more likely to change the mean than the median.
Others with more rigorous proofs might be satisfying your urge for rigor, but the question relates to generalities but allows for exceptions. So, you really don't need all that rigor. There are exceptions to the rule, so why depend on rigorous proofs when the end result is, "Well, 'typically' this rule works but not always...". The example I provided is simple and easy for even a novice to process.
