5
$\begingroup$

I have tried to find a solution to my question on Google, but I can’t seem to find much information about error bars and median absolute deviation and I do not know much about statistical error analysis so any help would be greatly appreciated.

I am creating a semi-log plot for my astronomy research that splits the data into 5 equally spaced bins in log base 10 (x-axis) and plots the average value for each bin (y-axis). Thus each bin has a larger sample size as the x-axis increases. Since it’s a statistical study there is a lot of potential for uncertainty and there is a high presence of outliers (non-Gaussian error distributions). As a result, I would like to use median absolute deviation (MAD) error bars since MAD is less susceptible to outliers.

So my question is: Much like standard deviation and standard error, where standard error is just stdev/sqrt(N) and N is the size of the sample – is there an analog to this for median absolute deviation? The error bars I get when I use MAD/sqrt(N) look correct, but I am not confident in explaining my reasoning for using this approach. Is dividing MAD by the square root of the sample size an acceptable means to produce error bars? And if not, do you have any other suggestions? Also, from research I’ve done it seems that median absolute deviation is a better estimator than mean absolute deviation…would you agree with this?

UPDATE:

Here is my original plot with error bars from median absolute deviation/sqrt(N) (NOTE: title should read median, not mean!) The first bin contains 39 samples, second contains 146, third 454, fourth 1287, and fifth 2371 samples. It looks nice, but the method for producing error bars does not seem very accurate. enter image description here

I've created error bars using the bootstrap method as suggested by @Glen_b (attached below). It looks good to me, but my advisor feels that the error bars have been overestimated in this method.

enter image description here

$\endgroup$
  • $\begingroup$ Do you need to use bins equally spaced in terms of log10(x)? Could you instead compare y among the quintiles with respect to the x values? Then standard boxplots would display well, specifically show outliers, and the problems of unequal-sized groups would go away. $\endgroup$ – EdM Dec 12 '13 at 18:34
  • $\begingroup$ Hi @EdM - As this plot will be compared to similar plots in the field, my advisor would like me to keep it as a semi-log plot as I attached. A boxplot is a good idea that I have not thought about though - it could be easier to provide information about outliers and error. I will have to look more into this and bring it up to my advisor. Thank you for the suggestion! $\endgroup$ – Joe Dec 12 '13 at 18:54
  • 1
    $\begingroup$ "believes they have been overestimated" sounds like a way of saying "considers they look uncomfortably large". That aside "abs mean deviation" or even "abs median deviation" needs to be fixed; mentioning "abs" first implies it's the last operation, which is quite wrong. $\endgroup$ – Nick Cox Dec 13 '13 at 10:40
4
$\begingroup$

It sounds like you're talking about what's sometimes called a regressogram, with a log-scaled x-variable.

There are a number of issues here, not necessarily in logical order:

  • the quantity you're plotting is a mean, so if you want to plot median absolute deviation, it's the MAD of the means you want.

  • your suggestion $\text{MAD}/\sqrt n$ leads to the question "when is the MAD of the mean equal to the MAD of the data divided by $\sqrt n$?"

  • when you say "it seems that median absolute deviation is a better estimator than mean absolute deviation" ... that depends what we're talking about - a better estimator of what?, and under what circumstances?


So, "when is the MAD of the mean equal to the MAD of the data divided by $\sqrt n$?"

The answer is, unlike the situation with standard deviation, this is not generally the case. The reason why standard deviations of averages scale as they do is that variances of independent random variables add (more precisely, the variance of the sum is the sum of the variances when the variables are independent), irrespective of the distributions of the components (as long as the variances all exist). It is this particular property that largely accounts for the popularity of variances and standard deviations.

Neither the median deviation, nor the mean deviation have that property in general.

However, when the data are normal, they will in effect inherit that property, since the ratio of the population mean deviation or median deviation to the standard deviation at a normal will be a constant, normals are closed under convolution, and standard deviations scale that way.

If the data were reasonably close to normal, it could perhaps be adequate.


What else might be done? One way to estimate the standard error of a statistic is via the bootstrap; for the mean deviation - being a mean - this should do well in large samples. Unfortunately, medians don't do so well under the bootstrap, and this issue will carry over to median absolute deviations.

If you have some probability model for your data, there's also simulation as a way of approaching the problem.

$\endgroup$
  • $\begingroup$ Hi Glen_b, thanks so much for your input. I have produced error bars using the bootstrap method but my advisor thinks that the error bars are too large with this method. Do you think that using the jackknife method instead of bootstrap would produce any difference? $\endgroup$ – Joe Dec 12 '13 at 18:46
  • 1
    $\begingroup$ Joe, error bars are always too small (our models never account for all possible sources of variation or uncertainty). When somebody says that they are too large when a legitimate method has been used to calculate them, that raises a big red flag because it suggests their intent is to use statistical analysis to sanctify preconceived results rather than to reveal anything about the data. $\endgroup$ – whuber Dec 12 '13 at 19:44
  • $\begingroup$ Joe, can you say more about how you produced the results with the bootstrap? While notionally simple enough, I know from personal experience that there are a number of places one might go astray. Note that one way to test your implementation is to apply it to simulated data in the normal case (simulated exactly from the model, which is assumed to be constant+noise in each bin, but you may need only simulate a single bin to begin with to identify the most likely sources of any errors). If the samples aren't too small, your original approach and the bootstrap should give quite similar answers. $\endgroup$ – Glen_b Dec 12 '13 at 20:12
  • $\begingroup$ @Glen_b I used an IDL routine I found online to produce the bootstrap statistics (found here: code.google.com/p/sdssidl/source/browse/trunk/pro/statistics/…) My original approach and the bootstrap approach do give similar answers (both plots have been added to original question posted) I think that the bootstrap approach is the best bet thus far, however I'd still like to check out the jackknife approach. $\endgroup$ – Joe Dec 12 '13 at 22:52
  • $\begingroup$ Sorry, excuse my slowness. How does that code give you the bootstrap distribution for the mean deviation of the sample mean? $\endgroup$ – Glen_b Dec 13 '13 at 3:02
3
$\begingroup$

A standard error means something. You don't just take any old statistic and divide by sqrt(n). Why not just plot your MAD and have your error bar a representation of variability in the data? If you want something to represent the quality of your median estimate then just calculate a confidence interval of the median.

$\endgroup$
  • $\begingroup$ Hi John, thanks for the response and I now realize that my original plan is not too accurate. The reason I do not want to just plot MAD for my error bars is because this provides similarly sized error bars for each bin. Since my first bin has less than 100 samples and my 5th has over 2000,shouldn't the error bars for the last bin be much smaller than the first due to the larger sample size? $\endgroup$ – Joe Dec 12 '13 at 18:25
  • $\begingroup$ MAD is not consistently influenced by sample size one way or the other. If you believe there should be equal variability among the groups then you can use the best estimate you have and larger N's generally have better estimates. If it's not, then use the estimates for each group. Regardless, I see that your questions has evolved quite substantially beyond my initial answer. $\endgroup$ – John Dec 12 '13 at 20:20
3
$\begingroup$

Whatever you do, plot your raw data or minimally make them available some how.

If you choose median absolute deviation (MAD), do make it absolutely clear whether it is of deviations from the mean or the median, as I've seen MAD used as an abbreviation for both and in any case any ambiguity benefits no-one.

Plotting +/- MAD as error bars has a loose connection to the widely used box plots in which median and quartiles are shown in a box and there are various different recipes for what is shown outside the box.

MAD is approximately |quartile $-$ median| in a symmetric distribution. For a symmetric distribution it's immaterial whether MAD is MAD from median and or from mean or "quartile" is the upper or lower quartile. MAD will be similar to (upper q. $-$ median) and (median $-$ lower q.) even in many asymmetric distributions. There are various slightly different rules for quartiles, which may cause minor puzzles, but is not central here.

A bigger question is this: if outliers make your standard errors dubious, how come you want to show means, because they will be affected too? As @John implies, a median is clearly a possibility. Also, would you be better off on a logarithmic or other transformed scale for your y variable too?

$\endgroup$
  • $\begingroup$ Hi Nick, thank you for the input. I need to give this problem more thought...after finals! I would like to try out your suggestion of median/quartile errors. My advisor would like to stick to a log scale only in x since this plot is being compared to other similar plots that have been released. Thanks again. $\endgroup$ – Joe Dec 12 '13 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.