Why is the Median Less Sensitive to Extreme Values Compared to the Mean?

Question

I am sure we have all heard the following argument stated in some way or the other:

• For a given set of measurements (e.g. heights of students), the mean of these measurements is more "prone" to be influenced by outliers compared to the median of these same measurements.

Conceptually, the above argument is straightforward to understand. The median is not directly calculated using the "value" of any of the measurements, but only using the "ranked position" of the measurements. On the other hand, the mean is directly calculated using the "values" of the measurements, and not by using the "ranked position" of the measurements. Therefore, a statistically larger number of outlier points should be required to influence the median of these measurements - compared to influence of fewer outlier points on the mean. For example: the average weight of a blue whale and 100 squirrels will be closer to the blue whale's weight, but the median weight of a blue whale and 100 squirrels will be closer to the squirrels.

Using the R programming language, we can see this argument manifest itself on simulated data:

library(plotly)
    set.seed(123)


d = data.frame(data = rnorm(20, 5,50), col = "non outlier")

dd = data.frame(data = rnorm(5,150, 10), col = "outlier")

my_data = rbind(d,dd)

> mean(d$data)
[1] 10.08877

> median(d$data)
[1] 17.11447

We can also plot this to get a better idea:

d1 = data.frame(data = mean(my_data$data), col = "mean")
# add "1" to the median so that it becomes visible in the plot
d2 = data.frame(data = median(my_data$data) +1 , col = "median")

new_data = rbind(my_data, d1, d2)


plot_ly(type = "scatter", mode = "markers", data = new_data, x = ~data, y = " ", color = ~col ) %>% layout(title = 'Effect of Outliers on Median vs Mean')

My Question: In the above example, we can see that the median is less influenced by the outliers compared to the mean - but in general, are there any "statistical proofs" that shed light on this inherent "vulnerability" of the mean compared to the median?

Apart from the logical argument of measurement "values" vs. "ranked positions" of measurements - are there any theoretical arguments behind why the median requires larger valued and a larger number of outliers to be influenced towards the extremas of the data compared to the mean?

I am aware of related concepts such as Cooke's Distance (https://en.wikipedia.org/wiki/Cook%27s_distance) which can be used to estimate the effect of removing an individual data point on a regression model - but are there any formulas which show some relation between the number/values of outliers on the mean vs. the median?

Are there any theoretical statistical arguments that can be made to justify this logical argument regarding the number/values of outliers on the mean vs. the median?

Answer 1

if you write the sample mean $\bar x$ as a function of an outlier $O$, then its sensitivity to the value of an outlier is $d\bar x(O)/dO=1/n$, where $n$ is a sample size. the same for a median is zero, because changing value of an outlier doesn't do anything to the median, usually.

example to demonstrate the idea: 1,4,100. the sample mean is $\bar x=35$, if you replace 100 with 1000, you get $\bar x=335$. the median stays the same 4.

this is assuming that the outlier $O$ is not right in the middle of your sample, otherwise, you may get a bigger impact from an outlier on the median compared to the mean.

TL;DR;

adding the outlier

you may be tempted to measure the impact of an outlier by adding it to the sample instead of replacing a valid observation with na outlier. it can be done, but you have to isolate the impact of the sample size change. if you don't do it correctly, then you may end up with pseudo counter factual examples, some of which were proposed in answers here. I'll show you how to do it correctly, then incorrectly.

The mean $x_n$ changes as follows when you add an outlier $O$ to the sample of size $n$: $$\bar x_{n+O}-\bar x_n=\frac {n \bar x_n +O}{n+1}-\bar x_n$$ Now, let's isolate the part that is adding a new observation $x_{n+1}$ from the outlier value change from $x_{n+1}$ to $O$. We have to do it because, by definition, outlier is an observation that is not from the same distribution as the rest of the sample $x_i$. Remember, the outlier is not a merely large observation, although that is how we often detect them. It is an observation that doesn't belong to the sample, and must be removed from it for this reason. Here's how we isolate two steps: $$\bar x_{n+O}-\bar x_n=\frac {n \bar x_n +x_{n+1}}{n+1}-\bar x_n+\frac {O-x_{n+1}}{n+1}\\ =(\bar x_{n+1}-\bar x_n)+\frac {O-x_{n+1}}{n+1}$$

Now, we can see that the second term $\frac {O-x_{n+1}}{n+1}$ in the equation represents the outlier impact on the mean, and that the sensitivity to turning a legit observation $x_{n+1}$ into an outlier $O$ is of the order $1/(n+1)$, just like in case where we were not adding the observation to the sample, of course. Note, that the first term $\bar x_{n+1}-\bar x_n$, which represents additional observation from the same population, is zero on average.

If we apply the same approach to the median $\bar{\bar x}_n$ we get the following equation: $$\bar{\bar x}_{n+O}-\bar{\bar x}_n=(\bar{\bar x}_{n+1}-\bar{\bar x}_n)+0\times(O-x_{n+1})\\=(\bar{\bar x}_{n+1}-\bar{\bar x}_n)$$ In other words, there is no impact from replacing the legit observation $x_{n+1}$ with an outlier $O$, and the only reason the median $\bar{\bar x}_n$ changes is due to sampling a new observation from the same distribution.

a counter factual, that isn't

The analysis in previous section should give us an idea how to construct the pseudo counter factual example: use a large $n\gg 1$ so that the second term in the mean expression $\frac {O-x_{n+1}}{n+1}$ is smaller that the total change in the median. Here's one such example: "... our data is 5000 ones and 5000 hundreds, and we add an outlier of -100..."

Let's break this example into components as explained above. As an example implies, the values in the distribution are 1s and 100s, and -100 is an outlier. So, we can plug $x_{10001}=1$, and look at the mean: $$\bar x_{10000+O}-\bar x_{10000} =\left(50.5-\frac{505001}{10001}\right)+\frac {-100-\frac{505001}{10001}}{10001}\\\approx 0.00495-0.00150\approx 0.00345$$ The term $-0.00150$ in the expression above is the impact of the outlier value. It's is small, as designed, but it is non zero.

The same for the median: $$\bar{\bar x}_{10000+O}-\bar{\bar x}_{10000}=(\bar{\bar x}_{10001}-\bar{\bar x}_{10000})\\= (1-50.5)=-49.5$$

Voila! We manufactured a giant change in the median while the mean barely moved. However, if you followed my analysis, you can see the trick: entire change in the median is coming from adding a new observation from the same distribution, not from replacing the valid observation with an outlier, which is, as expected, zero.

a counter factual, that is

Now, what would be a real counter factual? In all previous analysis I assumed that the outlier $O$ stands our from the valid observations with its magnitude outside usual ranges. These are the outliers that we often detect. What if its value was right in the middle?

Let's modify the example above:"... our data is 5000 ones and 5000 hundreds, and we add an outlier of ..." 20!

Let's break this example into components as explained above. As an example implies, the values in the distribution are 1s and 100s, and 20 is an outlier. So, we can plug $x_{10001}=1$, and look at the mean: $$\bar x_{10000+O}-\bar x_{10000} =\left(50.5-\frac{505001}{10001}\right)+\frac {20-\frac{505001}{10001}}{10001}\\\approx 0.00495-0.00305\approx 0.00190$$ The term $-0.00305$ in the expression above is the impact of the outlier value. It's is small, as designed, but it is non zero.

The break down for the median is different now! $$\bar{\bar x}_{10000+O}-\bar{\bar x}_{10000}=(\bar{\bar x}_{10001}-\bar{\bar x}_{10000})\\= (1-50.5)+(20-1)=-49.5+19=-30.5$$

In this example we have a nonzero, and rather huge change in the median due to the outlier that is 19 compared to the same term's impact to mean of -0.00305! This shows that if you have an outlier that is in the middle of your sample, you can get a bigger impact on the median than the mean.

conclusion

Note, there are myths and misconceptions in statistics that have a strong staying power. For instance, the notion that you need a sample of size 30 for CLT to kick in. Virtually nobody knows who came up with this rule of thumb and based on what kind of analysis. So, it is fun to entertain the idea that maybe this median/mean things is one of these cases. However, it is not. Indeed the median is usually more robust than the mean to the presence of outliers.

Answer 2

A reasonable way to quantify the "sensitivity" of the mean/median to an outlier is to use the absolute rate-of-change of the mean/median as we change that data point. To that end, consider a subsample $x_1,...,x_{n-1}$ and one more data point $x$ (the one we will vary). If we denote the sample mean of this data by $\bar{x}_n$ and the sample median of this data by $\tilde{x}_n$ then we have:

$$\begin{align} \text{Sensitivity of mean} &\equiv \bigg| \frac{d\bar{x}_n}{dx} \bigg| = \frac{1}{n}, \\[12pt] \text{Sensitivity of median (} n \text{ odd)} &\equiv \bigg| \frac{d\tilde{x}_n}{dx} \bigg| = \mathbb{I}(x = x_{((n+1)/2)} < x_{((n+3)/2)}), \\[12pt] \text{Sensitivity of median (} n \text{ even)} &\equiv \bigg| \frac{d\tilde{x}_n}{dx} \bigg| = \frac{1}{2} \cdot \mathbb{I}(x_{(n/2)} \leqslant x \leqslant x_{(n/2+1)} < x_{(n/2+2)}). \\[12pt] \end{align}$$

In the trivial case where $n \leqslant 2$ the mean and median are identical and so they have the same sensitivity. In the non-trivial case where $n>2$ they are distinct. In this latter case the median is more sensitive to the internal values that affect it (i.e., values within the intervals shown in the above indicator functions) and less sensitive to the external values that do not affect it (e.g., an "outlier").

Answer 3

Changing an outlier doesn't change the median; as long as you have at least three data points, making an extremum more extreme doesn't change the median, but it does change the mean by the amount the outlier changes divided by n.

Adding an outlier, or moving a "normal" point to an extreme value, can only move the median to an adjacent central point. For instance, if you start with the data [1,2,3,4,5], and change the first observation to 100 to get [100,2,3,4,5], the median goes from 3 to 4. So not only is the a maximum amount a single outlier can affect the median (the mean, on the other hand, can be affected an unlimited amount), the effect is to move to an adjacently ranked point in the middle of the data, and the data points tend to be more closely packed close to the median. So, for instance, if you have nine points evenly spaced in Gaussian percentile, such as [-1.28, -0.84, -0.52, -0.25, 0, 0.25, 0.52, 0.84, 1.28]. The average separation between observations is 0.32, but changing one observation can change the median by at most 0.25. The median of a bimodal distribution, on the other hand, could be very sensitive to change of one observation, if there are no observations between the modes.

Answer 4

Extreme values influence the tails of a distribution and the variance of the distribution. This also influences the mean of a sample taken from the distribution.

Extreme values do not influence the center portion of a distribution. This means that the median of a sample taken from a distribution is not influenced so much.

---

Example

Below is an illustration with a mixture of three normal distributions with different means.

The mixture is 90% a standard normal distribution making the large portion in the middle and two times 5% normal distributions with means at $+ \mu$ and $-\mu$.

The value of $\mu$ is varied giving distributions that mostly change in the tails.

The consequence of the different values of the extremes is that the distribution of the mean (right image) becomes a lot more variable.

---

For large sample sizes

The sample variance of the mean will relate to the variance of the population:

$$Var[mean(x_n)] \approx \frac{1}{n} Var[x]$$

The sample variance of the median will relate to the slope of the cumulative distribution (and the height of the distribution density near the median)

$$Var[median(x_n)] \approx \frac{1}{n} \frac{1}{4f(median(x))^2}$$

Example where the mean is less influenced by outliers

In general we have that large outliers influence the variance $Var[x]$ a lot, but not so much the density at the median $f(median(x))$.

But, it is possible to construct an example where this is not the case. If we mix/add some percentage $\phi$ of outliers to a distribution with a variance of the outliers that is relative $v$ larger than the variance of the distribution (and consider that these outliers do not change the mean and median), then the new mean and variance will be approximately

$$Var[mean(x_n)] \approx \frac{1}{n} (1-\phi + \phi v) Var[x]$$

$$Var[mean(x_n)] \approx \frac{1}{n} \frac{1}{4((1-\phi)f(median(x))^2}$$

So the relative change (of the sample variance of the statistics) are for the mean $\delta_\mu = (v-1)\phi$ and for the median $\delta_m = \frac{2\phi-\phi^2}{(1-\phi)^2}$. And we have $\delta_m > \delta_\mu$ if $$v < 1+ \frac{2-\phi}{(1-\phi)^2}$$

An example here is a continuous uniform distribution with point masses at the end as 'outliers'. The variance of a continuous uniform distribution is 1/3 of the variance of a Bernoulli distribution with equal spread. So $v=3$ and for any small $\phi>0$ the condition is fulfilled and the median will be relatively more influenced than the mean.

This is a contrived example in which the variance of the outliers is relatively small. This is done by using a continuous uniform distribution with point masses at the ends. So the outliers are very tight and relatively close to the mean of the distribution (relative to the variance of the distribution).

Answer 5

An outlier is not precisely defined, a point can more or less of an outlier. You might say outlier is a fuzzy set where membership depends on the distance $d$ to the pre-existing average. Call such a point a $d$-outlier. The key difference in mean vs median is that the effect on the mean of a introducing a $d$-outlier depends on $d$, but the effect on the median does not. Using Big-0 notation, the effect on the mean is $O(d)$, and the effect on the median is $O(1)$.

Btw "the average weight of a blue whale and 100 squirrels will be closer to the blue whale's weight"--this is not true.

Answer 6

Mathematical description/proof/viewpoint for special case

There is a short mathematical description/proof in the special case of

• Comparing the sensitivity in terms of the variance of the sample statistic. (this can be generalized to the 3rd central moment, and possibly other cost functions)
• Symmetric distributions in which case we can express the distribution of the median and mean in terms of integrals of the quantile functions in a similar way.

Let's assume that the distribution is centered at $0$ and the sample size $n$ is odd (such that the median is easier to express as a beta distribution).

Then in terms of the quantile function $Q_X(p)$ we can express

$$\begin{array}{rcrr} Var[mean(X_n)] &=& \frac{1}{n}\int_0^1& 1 \cdot Q_X(p)^2 \, dp \\ Var[median(X_n)] &=& \frac{1}{n}\int_0^1& f_n(p) \cdot Q_X(p)^2 \, dp \end{array}$$

where $f(p) = \frac{n}{Beta(\frac{n+1}{2}, \frac{n+1}{2})} p^{\frac{n-1}{2}}(1-p)^{\frac{n-1}{2}}$

Below is a plot of $f_n(p)$ when $n = 9$ and it is compared to the constant value of $1$ that is used to compute the variance of the sample mean.

What the plot shows is that the contribution of the squared quantile function to the variance of the sample statistics (mean/median) is for the median larger in the center and lower at the edges.

---

Adding outliers versus changing outliers.

When we change outliers, then the quantile function $Q_X(p)$ changes only at the edges where the factor $f_n(p) < 1$ and so the mean is more influenced than the median.

When we add outliers, then the quantile function $Q_X(p)$ is changed in the entire range. So the median might in some particular cases be more influenced than the mean.

Example: Say we have a mixture of two normal distributions with different variances and mixture proportions. Then the change of the quantile function is of a different type when we change the variance in comparison to when we change the proportions.

Below is an example of different quantile functions where we mixed two normal distributions.

• The black line is the quantile function for the mixture of $90\%$ a distribution with $\sigma = 1$ and $\phi = 10\%$ a distribution with $\sigma_{outlier} = 2$.
• On the left we changed the proportion of outliers $\phi \in \lbrace 20 \%, 30 \%, 40 \% \rbrace$.
• On the right we changed the variance of outliers with $ \sigma_{outlier} \in \lbrace 4, 8, 16 \rbrace$.

The quantile function of a mixture is a sum of two components in the horizontal direction. Whether we add more of one component or whether we change the component will have different effects on the sum.

---

Generalizations

• The conditions that the distribution is symmetric and that the distribution is centered at 0 can be lifted. It will make the integrals more complex. $$\begin{array}{rcrr} Var[mean(X_n)] &=& \frac{1}{n}\int_0^1& 1 \cdot (Q_X(p)-Q_(p_{mean}))^2 \, dp \\ Var[median(X_n)] &=& \frac{1}{n}\int_0^1& f_n(p) \cdot (Q_X(p) - Q_X(p_{median}))^2 \, dp \end{array}$$ now these 2nd terms in the integrals are different. We have $(Q_X(p)-Q_(p_{mean}))^2$ and $(Q_X(p) - Q_X(p_{median}))^2$. But we still have that the factor in front of it is the constant $1$ versus the factor $f_n(p)$ which goes towards zero at the edges.

• The condition that we look at the variance is more difficult to relax. I have made a new question that looks for simple analogous cost functions. But we could imagine with some intuitive handwaving that we could eventually express the cost function as a sum of multiple expressions $$mean: E[S(X_n)] = \sum_{i}g_i(n) \int_0^1 1 \cdot h_{i,n}(Q_X) \, dp \\ median: E[S(X_n)] = \sum_{i}g_i(n) \int_0^1 f_n(p) \cdot h_{i,n}(Q_X) \, dp $$ where we can not solve it with a single term but in each of the terms we still have the $f_n(p)$ factor, which goes towards zero at the edges.

Answer 7

A helpful concept when considering the sensitivity/robustness of mean vs. median (or other estimators in general) is the breakdown point. This is the proportion of (arbitrarily wrong) outliers that is required for the estimate to become arbitrarily wrong itself.

Using this definition of "robustness", it is easy to see how the median is less sensitive: At least HALF your samples have to be outliers for the median to break down (meaning it is maximally robust), while a SINGLE sample is enough for the mean to break down.

Answer 8

I'm going to say no, there isn't a proof the median is less sensitive than the mean since it's not always true. At least not if you define "less sensitive" as a simple "always changes less under all conditions". I'm told there are various definitions of sensitivity, going along with rules for well-behaved data for which this is true.

Say our data is 5000 ones and 5000 hundreds, and we add an outlier of -100 (or we change one of the hundreds to -100). The median jumps by 50 while the mean barely changes.

That seems like very fake data. So say our data is only multiples of 10, with lots of duplicates. It could even be a proper bell-curve. Then it's possible to choose outliers which consistently change the mean by a small amount (much less than 10), while sometimes changing the median by 10. Or simply changing a value at the median to be an appropriate outlier will do the same.

Or we can abuse the notion of outlier without the need to create artificial peaks. Take the 100 values 1,2 ... 100. Mean and median both 50.5. Then add an "outlier" of -0.1 -- median shifts by exactly 0.5 to 50, mean (5049.9/101) drops by almost 0.5 but not quite. Of course we already have the concepts of "fences" if we want to exclude these barely outlying outliers.

If feels as if we're left claiming the rule is always true for sufficiently "dense" data where the gap between all consecutive values is below some ratio based on the number of data points, and with a sufficiently strong definition of outlier.

Answer 9

Actually, there are a large number of illustrated distributions for which the statement can be wrong!

Background for my colleagues, per Wikipedia on Multimodal distributions:

Bimodal distributions have the peculiar property that – unlike the unimodal distributions – the mean may be a more robust sample estimator than the median.[15] This is clearly the case when the distribution is U shaped like the arcsine distribution. It may not be true when the distribution has one or more long tails.

Here is another educational reference (from Douglas College) which is certainly accurate for large data scenarios:

In symmetrical, unimodal datasets, the mean is the most accurate measure of central tendency. For asymmetrical (skewed), unimodal datasets, the median is likely to be more accurate. For bimodal distributions, the only measure that can capture central tendency accurately is the mode.

So, evidently, in the case of said distributions, the statement is incorrect (lacking a specificity to the class of unimodal distributions).

Answer 10

Ironically, you are asking about a generalized truth (i.e., normally true but not always) and wonder about a proof for it. If you want a reason for why outliers TYPICALLY affect mean more so than median, just run a few examples. Your light bulb will turn on in your head after that.

Step 1: Take ANY random sample of 10 real numbers for your example. Step 2: Identify the outlier with a value that has the greatest absolute value. Step 3: Add a new item (eleventh item) to your sample set and assign it a positive value number that is 1000 times the magnitude of the absolute value you identified in Step 2. Step 4: Add a new item (twelfth item) to your sample set and assign it a negative value number that is 1000 times the magnitude of the absolute value you identified in Step 2. Step 5: Calculate the mean and median of the new data set you have. Compare the results to the initial mean and median. Which one changed more, the mean or the median. Step 6. Repeat the exercise starting with Step 1, but use different values for the initial ten-item set.

Again, did the median or mean change more? No matter what ten values you choose for your initial data set, the median will not change AT ALL in this exercise! You can use a similar approach for item removal or item replacement, for which the mean does not even change one bit. Clearly, changing the outliers is much more likely to change the mean than the median.

Others with more rigorous proofs might be satisfying your urge for rigor, but the question relates to generalities but allows for exceptions. So, you really don't need all that rigor. There are exceptions to the rule, so why depend on rigorous proofs when the end result is, "Well, 'typically' this rule works but not always...". The example I provided is simple and easy for even a novice to process.