25
$\begingroup$

Section 1.7.2 of Discovering Statistics Using R by Andy Field, et al., while listing virtues of mean vs median, states:

... the mean tends to be stable in different samples.

This after explaining median's many virtues, e.g.

... The median is relatively unaffected by extreme scores at either end of the distribution ...

Given that the median is relatively unaffected by extreme scores, I'd have thought it to be more stable across samples. So I was puzzled by the authors' assertion. To confirm I ran a simulation — I generated 1M random numbers and sampled 100 numbers 1000 times and computed mean and median of each sample and then computed the sd of those sample means and medians.

nums = rnorm(n = 10**6, mean = 0, sd = 1)
hist(nums)
length(nums)
means=vector(mode = "numeric")
medians=vector(mode = "numeric")
for (i in 1:10**3) { b = sample(x=nums, 10**2); medians[i]= median(b); means[i]=mean(b) }
sd(means)
>> [1] 0.0984519
sd(medians)
>> [1] 0.1266079
p1 <- hist(means, col=rgb(0, 0, 1, 1/4))
p2 <- hist(medians, col=rgb(1, 0, 0, 1/4), add=T)

As you can see the means are more tightly distributed than medians.

enter image description here

In the attached image the red histogram is for medians — as you can see it is less tall and has fatter tail which also confirms the assertion of the author.

I’m flabbergasted by this, though! How can median which is more stable tends to ultimately vary more across samples? It seems paradoxical! Any insights would be appreciated.

$\endgroup$
6
  • 2
    $\begingroup$ Yeah, but try it by sampling from nums <- rt(n = 10**6, 1.1). That t1.1 distribution will give a bunch of extreme values, not necessarily balanced between positive and negative (just as good a chance of getting another positive extreme value as a negative extreme value to balance), that will cause a gigantic variance in $\bar{x}$. This is what median shields against. The normal distribution is unlikely to give any especially extreme values to stretch out the $\bar{x}$ distribution wider than median. $\endgroup$ – Dave Jul 2 '19 at 20:48
  • 11
    $\begingroup$ The author's statement is not generally true. (We have received many questions here related to errors in this author's books, so this is not a surprise.) The standard counterexamples are found among the "stable distributions", where the mean is anything but "stable" (in any reasonable sense of the term) and the median is far more stable. $\endgroup$ – whuber Jul 2 '19 at 21:00
  • 1
    $\begingroup$ "... the mean tends to be stable in different samples." is a nonsense statement. "stability" is not well defined. The (sample) mean is indeed quite stable in a single sample because it is a nonrandom quantity. If the data are "instable" (highly variable?) the mean is "instable" too. $\endgroup$ – AdamO Jul 2 '19 at 22:04
  • 1
    $\begingroup$ This question is likely answered by the detailed analyses offered at stats.stackexchange.com/questions/7307, wherein the same question is asked in a specific way (where the sense of "stable" is well defined). $\endgroup$ – whuber Jul 3 '19 at 18:59
  • 2
    $\begingroup$ Try replacing rnorm with rcauchy. $\endgroup$ – Eric Towers Jul 4 '19 at 7:20
4
$\begingroup$

The median is maximally robust to outliers, but highly susceptible to noise. If you introduce a small amount of noise to each point, it will enter the median undampened as long as the noise is small enough to not change the relative order of the points. For the mean it's the other way around. Noise is averaged out, but a single outlier can change the mean arbitrarily.

                                        mean  median   
original   [1.0, 2.0, 3.0, 4.0, 5.0]       3       3
noise      [1.1, 1.9, 3.1, 4.1, 4.9]    3.02     3.1
outlier    [100, 2.0, 3.0, 4.0, 5.0]    22.8       4

Your test mostly measures robustness to noise, but you can easily create a sample where the median performs better. If you want an estimator that is robust to both outliers and noise, just throw away the top and bottom third and average the remainder.

$\endgroup$
1
  • 1
    $\begingroup$ Is there a more specific name for this algorithm than "the 33% trimmed mean" ? $\endgroup$ – David Cary Oct 26 '19 at 18:36
26
$\begingroup$

As @whuber and others have said, the statement is not true in general. And if you’re willing to be more intuitive — I can’t keep up with the deep math geeks around here — you might look at other ways mean and median are stable or not. For these examples, assume an odd number of points so I can keep my descriptions consistent and simple.

  1. Imagine you have spread of points on a number line. Now imagine you take all of the points above the middle and move them up to 10x their values. The median is unchanged, the mean moved significantly. So the median seems more stable.

  2. Now imagine these points are fairly spread out. Move the center point up and down. A one-unit move changes the median by one, but barely moved the mean. The median now seems less stable and more sensitive to small movements of a single point.

  3. Now imagine taking the highest point and moving it smoothly from the highest to the lowest point. The mean will also smoothly move. But the median will not move continuously: it won’t move at all until your high point becomes lower than the previous median, then it starts following the point until it goes below the next point, then the median sticks to that point and again doesn’t move as you continue moving your point downwards. [Edited per comment]

So different transformations of your points cause either mean or median to look less smooth or stable in some sense. The math heavy-hitters here have shown you distributions from which you can sample, which more closely matches your experiment, but hopefully this intuition helps as well.

$\endgroup$
2
  • 1
    $\begingroup$ Regarding item 3: Wouldn't the median also move smoothly? Say the initial set of points is [1, 3, 5, 7, 9]. Initially the median is 5. That will remain the median until the fifth point (initially 9) drops below 5, at which point the median will smoothly follow the fifth point as it decreases, until it hits 3, at which point the median will stay at 3. So even though the point which defines the median is "jumping" (from the third point, to the fifth point, to the second point), the actual value of the median has no jump/discontinuity. $\endgroup$ – Scott M Jul 3 '19 at 16:30
  • $\begingroup$ @ScottM You seem right. Not sure why I thought it would jump. I’ll reword when I get a chance. $\endgroup$ – Wayne Jul 3 '19 at 17:25
19
$\begingroup$

Suppose you have $n$ data points from some underlying continuous distribution with mean $\mu$ and variance $\sigma^2 < \infty$. Let $f$ be the density function for this distribution and let $m$ be its median. To simplify this result further, let $\tilde{f}$ be the corresponding standardised density function, given by $\tilde{f}(z) = \sigma \cdot f(\mu+\sigma z)$ for all $z \in \mathbb{R}$. The asymptotic variance of the sample mean and sample median are given respectively by:

$$\mathbb{V}(\bar{X}_n) = \frac{\sigma^2}{n} \quad \quad \quad \quad \quad \mathbb{V}(\tilde{X}_n) \rightarrow \frac{\sigma^2}{n} \cdot \frac{1}{4} \cdot \tilde{f}\Big( \frac{m-\mu}{\sigma} \Big)^{-2}.$$

We therefore have:

$$\frac{\mathbb{V}(\bar{X}_n)}{\mathbb{V}(\tilde{X}_n)} \rightarrow 4 \cdot \tilde{f}\Big( \frac{m-\mu}{\sigma} \Big)^2.$$

As you can see, the relative size of the variance of the sample mean and sample median is determined (asymptotically) by the standardised density value at the true median. Thus, for large $n$ we have the asymptotic correspondence:

$$\mathbb{V}(\bar{X}_n) < \mathbb{V}(\tilde{X}_n) \quad \quad \iff \quad \quad f_* \equiv \tilde{f} \Big( \frac{m-\mu}{\sigma} \Big) < \frac{1}{2}.$$

That is, for large $n$, and speaking asymptotically, the variance of the sample mean will be lower than the variance of the sample median if and only if the standardised density at the standardised median value is less than one-half. The data you used in your simulation example was generated from a normal distribution, so you have $f_* = 1 / \sqrt{2 \pi} = 0.3989423 < 1/2$. Thus, it is unsurprising that you found a higher variance for the sample median in that example.

$\endgroup$
1
  • $\begingroup$ Awesome! Thanks. $\endgroup$ – Alok Lal Jul 19 '19 at 2:11
4
$\begingroup$

Comment: Just to echo back your simulation, using a distribution for which SDs of means and medians have the opposite result:

Specifically, nums are now from a Laplace distribution (also called 'double exponential'), which can be simulated as the difference of two exponential distribution with the same rate (here the default rate 1). [Perhaps see Wikipedia on Laplace distributions.]

set.seed(2019)
nums = rexp(10^6) - rexp(10^6)
means=vector(mode = "numeric")
medians=vector(mode = "numeric")
for (i in 1:10^3) { b = sample(x=nums, 10^2); 
  medians[i]= median(b); means[i]=mean(b) }
sd(means)
[1] 0.1442126
sd(medians)
[1] 0.1095946   # <-- smaller

hist(nums, prob=T, br=70, ylim=c(0,.5),  col="skyblue2")
 curve(.5*exp(-abs(x)), add=T, col="red")

enter image description here

Note: Another easy possibility, explicitly mentioned in @whuber's link, is Cauchy, which can be simulated as Student's t distribution with one degree of freedom, rt(10^6, 1). However, its tails are so heavy that making a nice histogram is problematic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.