Skip to main content
2 of 11
added 57 characters in body
user603
  • 23k
  • 3
  • 88
  • 155

Three facts will help you here. I will denote $n$ the sample size.

  • What you discovered is called the exact fit property. If a proportion $\alpha > 0.5$ of the observations in your sample have the same value, the mad of those observations will be 0.
  • This is not a property of the mad in particular, but of all robust estimators of scale. More precisely: any robust estimator of scale with a breakdown point of $0< \alpha < 0.5$ will have an exact fit property at the level of $1-\alpha$ (see section 3 of Croux et al., 2006, [0], for example).
  • Your first proposals amount to replacing the value of $M_i$ by arbitrary numbers in case of exact fit (setting $M_i=0$ in the former and $M_i=O(1/\sigma)$ --where $\sigma$ is the amount by which you perturb the data-- in the latter).

In both cases, you seem to think that the appearance of exact fit means the MAD=0 means the MAD becomes somehow useless for describing the sample and the resulting outlier's detection rule does not work (it doesn't quiet follow from this that you would still prefer to set $M_i$ to an arbitrary number, but let us set that aside for now). But this assessment is not warranted. To wit: In an exact fit situation where half or more of the data is tied at an arbitrary value $x$, all numbers in your sample that are different from $x$ are severe outliers.

These numbers are, after all, infinitely divergent from the pattern of the bulk of the data.

The last point is the salient one. If you use the convention $0/0:=0$ then you can use the outliers detection rule as before (values of $M_i>3.5$ are the outliers).

  • Algorithms for projection-pursuit robust principal component analysis. (2006). Croux, C. Filzmoser, P. and Oliveira, M. R.
user603
  • 23k
  • 3
  • 88
  • 155