1
$\begingroup$

My question arises because initially I wanted to make a short study comparing the Median Absolute Deviation $\rm{MAD}$ as an approximation of the standard deviation, $\sigma$.

The $\rm{MAD}$ is a good way of estimating the standard deviation in data where one has outliers, weighting them less strongly. The normal arithmetic standard deviation will treat all values in the sample equally. The $\rm{MAD}$ defined standard deviation is

$$\sigma_\rm{MAD} = k \cdot \rm{MAD} = k \cdot \rm{Median}\left(\left|x_i - \bar{x} \right|\right)$$

where $k$ is a scaling parameter and is dependant on the parent distribution. For Gaussian statistics this is $1.4826...$

So for a sample size of $N$ I wanted to see how well that $\sigma_\rm{MAD}$ agrees with the true $\sigma$ of the distribution - from which the sample is drawn.

I define a parent distribution which is Gaussian with parameters $\mu_{ref} = 0$ and $\sigma_{ref} = 1$. Then for sample sizes of $4,8,16,64,512$ I compare the difference between the true standard deviation of the parents distribution, $\sigma_{ref}$, and the standard deviation calculated from the normal arithmetic definition and the $\rm{MAD}$ approach. I.e. $\sigma_{ref} - \sigma_{SD}$ and $\sigma_{ref} - 1.4826 \cdot\rm{MAD}$. I repeat this process $100,000$ times for each sample size to get decent statistics.

What I see if I look at the histograms for what I have described above is the following: enter image description here

What I find interesting, and what my questions are, is two fold:

  1. the distributions for low statistics of $\sigma_{ref} - \sigma_{test}$ are very skewed. This is not surprising in the sense that for small samples far away points have more of an impact. I am curious as to what this distribution is. As you can see, as the sample size increases, Gaussian statistics is recovered.
  2. The second interesting thing is that you can see the MAD approach lags behind the arithmetic standard deviation in terms of the distribution shapes. If I can find an answer to one it will be easier to parametrise and understand this. I suppose it is related to the fact that the MAD suppresses the effect of outliers, but what I find surprising is that it is behind rather than in front of the other methods of determining the standard deviation of the sample.
$\endgroup$
10
  • 1
    $\begingroup$ The title is somewhat misleading, not sure in what sense this is an "undersampled bootstrap", but I surely expected a very different kind of question under that title. You may want to change it to "simulated distribution of the MAD" or something like that. $\endgroup$ Feb 22, 2020 at 16:09
  • $\begingroup$ Well that is my motivation, the principle question I am interested in, is what is the distribution of the histograms for a bootstrap where sample sizes are small and the parent distribution is Gaussian. The MAD business is just context - although you did provide a nice answer regarding the difference present in the MAD, thus my upvote. $\endgroup$
    – user27119
    Feb 22, 2020 at 17:10
  • 2
    $\begingroup$ But really this depends on what you "bootstrap". With MAD it's one thing, with other statistics it'll be another thing. Actually in my use of terminology (which I think is pretty standard) this wouldn't even be bootstrap, rather (Monte Carlo) simulation. There's parametric and nonparametric bootstrap; nonparametric bootstrap means sampling from an empirical distribution, parametric bootstrap means sampling from a parametric distribution with estimated parameters. You're sampling from a distribution with fixed parameters, despite actually simulating the distribution of estimators. $\endgroup$ Feb 22, 2020 at 17:38
  • 1
    $\begingroup$ And by the way this isn't "undersampled" either; the bootstrap/simulation principle doesn't require minimum underlying sample sizes, you basically can use this to find out what goes on at any sample size (which you actually successfully do here). Your sample sizes are small, fair enough, but I don't see what's "undersampled" here. $\endgroup$ Feb 22, 2020 at 17:40
  • $\begingroup$ Maybe my terminology is poor, I am an experimental physicist by trade -- not a statistician by any means. But I say undersampled because 4 points for example can hardly be representative of the parent distribution? I mean it also must be because in some sense because as my histograms shows my $\sigma_{ref} -\sigma_{test}$ get more Gaussian the larger the sample size -- for both arithmetic and MAD based SD. $\endgroup$
    – user27119
    Feb 22, 2020 at 17:45

1 Answer 1

2
$\begingroup$

This addresses question 2 only, I don't know the answer to 1. You generated data from a proper Gaussian distribution. A Gaussian distribution has a very, very low probability to generate outliers, because of is (squared) exponentially decreasing tails. For the Gaussian distribution, the standard way of estimating the standard deviation is optimal (actually not in every possible sense, but its square is minimum variance unbiased for the normal variance). The MAD with suitable choice of $k$ has the same expected value, but is worse in terms of variance and MSE. This is not surprising, because it is a median of deviations from the median, meaning that information from the half of observations further away from the median is not used, at least not the full quantitative information of their size (just the fact that they are "further away" influences the computation). Therefore the MAD makes less efficient use of the information in the observations and can be expected to be worse.

The advantage of the MAD is that if you don't know that data stem indeed from a Gaussian distribution, and there may be a danger that there are more and/or more extreme outliers than under the Gaussian, the MAD will hardly be affected, whereas the "arithmetic" standard deviation will be affected a lot (you may simulate data from a t-distribution with low degrees of freedom to check that). The MAD is at the same time still OKish (if not quite as good as the arithmetic standard deviation) for Gaussian data, and therefore recommended from a robustness point of view. However if you are sure your data are Gaussian (which of course you never can be) or even just light tailed without outliers, it is not surprising that the MAD does somewhat worse.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.