What distribution do statistics callculated from small samples follow, when drawn from a Gaussian parents distribtution? My question arises because initially I wanted to make a short study comparing the Median Absolute Deviation $\rm{MAD}$ as an approximation of the standard deviation, $\sigma$.
The $\rm{MAD}$ is a good way of estimating the standard deviation in data where one has outliers, weighting them less strongly. The normal arithmetic standard deviation will treat all values in the sample equally. The $\rm{MAD}$ defined standard deviation is
$$\sigma_\rm{MAD} = k \cdot \rm{MAD} = k \cdot \rm{Median}\left(\left|x_i - \bar{x} \right|\right)$$
where $k$ is a scaling parameter and is dependant on the parent distribution. For Gaussian statistics this is $1.4826...$
So for a sample size of $N$ I wanted to see how well that $\sigma_\rm{MAD}$ agrees with the true $\sigma$ of the distribution - from which the sample is drawn. 
I define a parent distribution which is Gaussian with parameters $\mu_{ref} = 0$ and $\sigma_{ref} = 1$. Then for sample sizes of $4,8,16,64,512$ I compare the difference between the true standard deviation of the parents distribution, $\sigma_{ref}$, and the standard deviation calculated from the normal arithmetic definition and the $\rm{MAD}$ approach. I.e. $\sigma_{ref} - \sigma_{SD}$ and  $\sigma_{ref} - 1.4826 \cdot\rm{MAD}$. I repeat this process $100,000$ times for each sample size to get decent statistics. 
What I see if I look at the histograms for what I have described above is the following:

What I find interesting, and what my questions are, is two fold:


*

*the distributions for low statistics of $\sigma_{ref} - \sigma_{test}$ are very skewed. This is not surprising in the sense that for small samples far away points have more of an impact. I am curious as to what this distribution is. As you can see, as the sample size increases, Gaussian statistics is recovered. 

*The second interesting thing is that you can see the MAD approach lags behind the arithmetic standard deviation in terms of the distribution shapes. If I can find an answer to one it will be easier to parametrise and understand this. I suppose it is related to the fact that the MAD suppresses the effect of outliers, but what I find surprising is that it is behind rather than in front of the other methods of determining the standard deviation of the sample.

 A: This addresses question 2 only, I don't know the answer to 1.
You generated data from a proper Gaussian distribution. A Gaussian distribution has a very, very low probability to generate outliers, because of is (squared) exponentially decreasing tails. For the Gaussian distribution, the standard way of estimating the standard deviation is optimal (actually not in every possible sense, but its square is minimum variance unbiased for the normal variance). The MAD with suitable choice of $k$ has the same expected value, but is worse in terms of variance and MSE. This is not surprising, because it is a median of deviations from the median, meaning that information from the half of observations further away from the median is not used, at least not the full quantitative information of their size (just the fact that they are "further away" influences the computation). Therefore the MAD makes less efficient use of the information in the observations and can be expected to be worse.
The advantage of the MAD is that if you don't know that data stem indeed from a Gaussian distribution, and there may be a danger that there are more and/or more extreme outliers than under the Gaussian, the MAD will hardly be affected, whereas the "arithmetic" standard deviation will be affected a lot (you may simulate data from a t-distribution with low degrees of freedom to check that). The MAD is at the same time still OKish (if not quite as good as the arithmetic standard deviation) for Gaussian data, and therefore recommended from a robustness point of view. However if you are sure your data are Gaussian (which of course you never can be) or even just light tailed without outliers, it is not surprising that the MAD does somewhat worse.  
