0
$\begingroup$

If a data set is in the presence of some outliers, then the sample average could not be a "fair" measure of the centrality of that data set. Instead, the sample median could be a better choice. However, it is not too unthinkable that, if a sample mean is not very informative, then the corresponding sample standard deviation is not very informative as well. So, in the presence of outliers that affect the empirical distribution sufficiently significant in a given sense, it seems more advisable to use a measure for dispersion such as $\sum_{1}^{n}|x_{i} - \hat{{\text{med}}}(x)|$ where $x := (x_{1}, \dots, x_{n})'$ is the data at hand and $\hat{\text{med}}(x)$ is the sample median of $x$; the intuition is that it is then a measure of dispersion with respect to a more "impartial" measure of centrality.

Under the assumption that such a measure is not as widely used as the usual standard deviation, it follows that such a measure has certain properties that are not very desirable. So my question is what are the pros and cons such a measure possesses comparing to the canonical measure of dispersion. Thanks.

$\endgroup$
  • $\begingroup$ That formula gives you something proportional to the "mean absolute deviation", which uses a robust central tendency measurement but is not robust about outliers in the deviation from that central tendency. A measurement in wider use is the median absolute deviation, which just goes all the way -- it's the median deviation from the median. $\endgroup$ – user8948 Oct 19 '18 at 19:09
  • 1
    $\begingroup$ @user8948 why don't you expand your comment into an answer that more directly addresses the question as posed? Explain, for example, how the proposed measure of dispersion would suffer in the presence of outliers and how the related median absolute deviation is robust to large numbers of outliers. This site is most useful for future reference (an important part of our goal) when questions have formal posted answers rather than answers hiding in comments. $\endgroup$ – EdM Oct 19 '18 at 21:27
0
$\begingroup$

One way of assessing the robustness of statistics to outliers is the breakdown point -- roughly, how much contamination in the data it can handle.

To simplify, let's restrict ourselves to scalar-valued data (this is what you were thinking anyway) and outliers as arbitrarily large values. "Arbitrarily large" means that for any finite number we choose, this value can be higher. The breakdown point is then the (smallest) fraction of arbitrarily large values that leads to an arbitrarily large estimate.

Consider, for example, the arithmetic mean. The true mean of $(x_1, \cdots, x_n)$ is $$\mu^* = \frac{x_1 +\cdots + x_n}{n};$$

but if we were to replace $x_n$ by some very large number $M$ by error, we would calculate

$$ \mu = \frac{x_1 + \cdots + x_{n-1} + M}n = \frac{x_1 + \cdots + x_{n-1} + x_n + M - x_n}n = \mu^* + \frac{M-x_n}n$$

Because $M$ is allowed to be arbitrarily large, this error would lead us to overestimate $\mu$ by an arbitrarily large amount. You can see this concretely on something like Excel or Python by taking $x_1, \cdots, x_n$ in the hundreds and replacing $x_n$ by a value of millions or billions.

How does replacing $x_n$ affect the median $m$? It doesn't: half of the values are still smaller than $m$. How does replacing $x_n$ and $x_{n-1}$ affect the median? (Note that our values aren't ordered; we're not simply replacing the largest or smallest values) Continuing this reasoning tells you that you need to replace a full half of your sample before the median is affected. The median has a breakdown point of 50%.


To your question: let's say we have again values $x_1, \cdots, x_n$. Your proposal is to measure dispersion by the mean deviation from the median:

$$S^* = |x_1 - m| + \cdots + |x_n - m| = n\tilde\mu $$

where $\tilde \mu$ is the arithmetic mean of the $|x_i - m|$. As before, replacing $|x_n - m|$ with some arbitrarily large number $M$ can lead to an arbitrarily large overestimate of $S$. Can we make $|x_n - m|$ arbitrarily large? Of course; replacing $M$ with some number larger than $\max(x_n,m)$ will do.

But: the median deviation from the median, also as before, is such that half of the $|x_i - m|$ could be replaced without changing the dispersion. This is called the median absolute deviation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.