# Error bars on error bars?

Inspired by my recent attendance at an environmental toxicology conference, I have the following question about error bars:

Let's say that I'm drawing samples from some unknown distribution, with finite mean and variance. I want to present the sample mean, and add some error bars. Since I don't know much about the underlying distribution, I just add error bars showing +/- the standard devaition of the samples.

My question is, is there any way I could meaningfully indicate how certain I am of those error bars? Adding error bars to the error bars, so to speak.

As as example, I have drawn 5 samples from some distribution, and I have repeated this 5 times. The sample means, and error bars of +/- the sample standard deviations, are shown below.

We can see that by chance, these sample means and error bars look quite different, and not really mutually compatible. Of course 5 samples isn't very much, but if my samples are obtained via some convoluted experimental procedure (capturing a wild animal and taking a blood sample, for example), it might not be an easy option to get more samples.

Update:

Just to add some notes on how I was thinking:

Coming from a computational physics background myself, I'm used to Monte Carlo methods, and the $$1/\sqrt{N}$$-error which follows from the central limit theorem. So just like the error in the sample mean has an expected distribution, I thought perhaps it would make sense to ask about the expected error in the sample standard deviation. Of course, the problem is that the distribution of the error in the sample mean is expressed in terms of the (unknown) variance of the underlying distribution, and hence I am left taking the standard deviation of the sample, or something along those lines.

But still, I thought there ought to be some way of indicating that my sample standard deviation is itself quite uncertain, due to the small $$N$$. But perhaps the only way is simply to list $$N$$, and be explicit about what the error bars show.

• Found on XKCD: xkcd.com/2110 Commented Jun 12, 2022 at 20:36
• You could look at the distribution of the sample variance, which relates the variance of the sample variance to the fourth central moment of the samples. I've used this quantity in the past to estimate error bars on quantum noise (where the variance is the signal). Commented Jun 13, 2022 at 20:04
• "Great fleas have little fleas upon their backs to bite 'em, And little fleas have lesser fleas, and so ad infinitum. And the great fleas themselves, in turn, have greater fleas to go on; While these again have greater still, and greater still, and so on." Augustus de Morgan, A Budget of Paradoxes. Also see It's turtles all the way down. How to resolve the issue? Renormalization.
– whuber
Commented Jun 13, 2022 at 21:04
• See also: How can I estimate meta uncertainty? on Physics SE. Commented Jun 14, 2022 at 11:51

You are interested in standard errors, which describe the variability in a parameter estimate, and are related to your sampling approach. This is distinct from the parameters themselves (e.g. mean and standard deviation), which are functions of the underlying population only, and are not dependent on how large your sample is.

Your current plot shows two values per group, the sample mean and sample standard deviation, about which there is no uncertainty (it is whatever you observe it to be). Assuming appropriate random sampling, you can use these values to make inference about the unobservable quantities of the population mean and population standard deviation for each group. You can use common tools like standard error or 95% confidence intervals to estimate the precision of your parameter estimates.

It would be odd to try to represent this as error bars on error bars, but it would be perfectly reasonable to list the mean and standard deviation for each group, along with the 95% CI of each parameter estimate. This can help you to decide if the means/standard deviations observed in Groups C and D, for example, represent true differences in the underlying population parameters, or if the apparent differences represent normal variation that would be expected with a small sample size.

• Thanks for your answer. Could you perhaps clarify your last paragraph a little? How the mentioned statistics can help me decide if C and D are truly different? (Also, note that in the situation I was thinking of, I would only have one small set of samples, not multiple. I just made the plot to illustrate the point that with a small sample size, the mean and standard deviation can come out quite different.)
– Tor
Commented Jun 13, 2022 at 18:55
• @Tor You can compute the means and standard deviations from the observed data, and along with the sample size, you can compute the standard error of the sample mean and the standard error of the sample standard deviation. From there, calculate the standard error/confidence interval of the difference of means/standard deviations - if the 95% CI of the difference does not contain 0, you can be reasonably sure that the groups were not drawn from distributions with identical means/SDs. Commented Jun 13, 2022 at 19:08
• Ok, I can estimate the standard error of the sample mean, $\sigma/\sqrt{N}$, by using the sample standard deviation as an estimate of the true standard deviation. But how do I calculate the standard error of the sample standard deviation?
– Tor
Commented Jun 13, 2022 at 19:19
• @Tor Good question, this isn't too commonly done, but there seems to be an answer here: stats.stackexchange.com/questions/156518/… Commented Jun 13, 2022 at 19:24

The objects we use to make inferences (e.g., estimates, confidence intervals, error bars, test statistics, p-values, etc.) are statistics, meaning that they are functions of the observed data. Since they are already functions of the observed data, these objects do not have any uncertainty in them --- they represent inferences about uncertain values, but there no uncertainty in the statistics themselves. We do not form error bars on error bars because there is no uncertainty in the error bars to begin with, because they are formed as a function of the observed data.

As a minor point, it is generally suboptimal practice to use error bars to show a deviation of plus/minus one (estimated) standard deviation. Usually you are better off using these values and other statistics to form an appropriate confidence interval for the uncertain value of interest, and using the error bars to show the confidence interval. In either case you should label your plot appropriately so that the reader understands what the error bars represent.

• It is common practice in physics to use the standard deviation in error bars. Not sure where this conventions stems from, but I guess it was used because for this choice the error bars (very) roughly represent the probable error. Using confidence intervals (i.e. $2\sigma$) would be misleading to the readers. One might use box plots, though. Commented Jun 13, 2022 at 9:40
• I agree there is no uncertainty in the observed standard deviation of a dataset, but we generally want to use this value to reason about the unobserved standard deviation of the population the data was drawn from, which is uncertain. It would be odd to show these graphically as error bars on the same plot, but calculating the standard error of the mean or the 95% confidence interval around the standard deviation seems rather commonplace. This answer makes it sound like computing the 95% CI of the standard deviation is nonsense. Commented Jun 13, 2022 at 14:01
• I don't really see anything in the answer that would imply that there is anything wrong with forming a CI for a population standard deviation. Whatever CI the OP wishes to form should already take account of the sampling variation in all relevant quantities.
– Ben
Commented Jun 13, 2022 at 22:53
• This is the right answer. FWIW, I think it's fine to mark off +/- 1 SD with lines on a plot. This needs to be made clear, of course, eg, in a figure caption. When doing so, these lines are descriptive, not inferential. It may also be worth mentioning that it's no big deal to use sample SDs that are subject to error to create a CI. If you can be confident the population is normal, you just use quantiles from the $t$ distribution to compute the limits; if you cannot be confident, Chebychev's inequality can provide outer bounds. Commented Jun 14, 2022 at 19:23
• Regarding the first paragraph: mean value is also a function of the data, and we do calculate uncertainty on it, i.e. uncertainty of how well it estimates certain parameter of the source distribution (its first moment in this case). The same way you can calculate uncertainty of any other function of the data. Commented Jun 15, 2022 at 2:01

However you construct your error bars, they are a rule. You cannot be unsure of them. Let us imagine that they are confidence intervals. There are multiple standard ways to create confidence intervals. They are different rules with slightly different properties. However, they are a chosen rule.

Other ways to construct error bars exist as well, such as adding plus or minus one standard deviation. It is still a rule.

You know the answer exactly. They are not uncertain.

What they are reflecting is the random elements of the samples seen. If they are a $$1-\alpha$$ percent confidence interval, there is a guarantee that the confidence intervals cover the parameter at least $$1-\alpha$$ percent of the time. There is no guarantee that it covers it for this sample. Even with a set of five samples, none of them may cover the parameter, the guarantee is over infinite repetition.

Each way you could construct an error bar has some form of optimality principle behind it. So, error bars satisfy some optimality condition that is good on average.

All of them are a statement of the best estimator of the range in which a parameter sits, given a model and a loss function.

Your error bars are a statement of uncertainty.

• Seeing some observed standard deviation in a sample yields zero uncertainty in the sample standard deviation - it is what it is. That does not, however, imply zero uncertainty in the population standard deviation. If all the groups in the OP's original plot were of different sizes, it would be perfectly reasonable to estimate a 95% CI for each group's SD to get a sense of how precise each group's sample SD is. This can allow inference of whether the observed SD difference between groups C and D, for example, represents a true difference in group SDs, or if it's merely sampling variation. Commented Jun 13, 2022 at 16:36

The traditional design of error bars gives an unfortunate impression of some linear distribution of uncertainty, and places a lot of visual emphasis on the the end of the bar, which is where the distribution of the location of your estimate is least likely. Clause Wilke (in his book Fundamentals of Data Visualization, in the chapter Visualizing uncertainty) shows some graphical alternatives to traditional error bars that convey something of the distribution of uncertainty in an estimate:

Image by Claus Wilke, used under an Attribution-NonCommercial-NoDerivatives 4.0 International licence. Original available at https://clauswilke.com/dataviz/visualizing-uncertainty.html

The "graded error bars" in (a) and (b) are formed by plotting the 90%, 95% and 99% CIs simultaneously. Thom Baguley discusses a similar approach he terms "tiered error bars" and provides example R code here: https://seriousstats.wordpress.com/2012/06/21/confidence-intervals-with-tiers/ , although I first saw such an approach being used by Andrew Gelman in his textbook Data Analysis Using Regression and Multilevel/Hierarchical Models.

• Thanks for posting, this is interesting to know about. I should probably check out this book. (Note, however, that this does not really solve my original problem, since my small sample size leads to enormous uncertainty in the distribution that would give rise to the tiered error bars.)
– Tor
Commented Jun 15, 2022 at 4:29
• deemphasizing the ends of the CI is very interesting!
– Ben
Commented Jun 16, 2022 at 0:23

## Review of confidence intervals

Let $$\theta \in \mathbb{R}$$ be a parameter of interest which we study based on a random variable $$X$$. An exact $$1-\alpha$$ confidence interval $$(L(X),U(X)$$ is defined by the property that $$\begin{equation*} \mathbb{P}\left[ L(X) < \theta < U(X) \right] = 1-\alpha, \end{equation*}$$ where $$L$$ is the lower endpoint and $$R$$ is the upper endpoint of the confidence interval.

The plot shown in the question illustrates that $$L$$ and $$U$$ are random variables. This is certainly the case, as they depend on the random variable $$X$$. However, a fraction of the confidence intervals $$(L(X),U(X))$$ contain $$\theta$$. By construction, the fraction is exactly $$1-\alpha$$. When $$\alpha=0.05$$, this is $$95\%$$ of the confidence intervals.

## Error bars on error bars

This procedure makes perfect sense if the target of inference is $$\theta$$ - which is what we stated above. However, you may also be interested in the endpoints $$L(X)$$ and $$U(X)$$ themselves. Then you can construct a "confidence intervals" $$(L^L(X), U^L(X))$$ and $$(L^U(X), U^U(X))$$ such that $$\begin{equation*} \mathbb{P} \left[L^L(X) < L(X) < U^L(X) \right] = 1-\alpha \end{equation*}$$ and $$\begin{equation*} \mathbb{P} \left[L^U(X) < U(X) < U^U(X) \right] = 1-\alpha. \end{equation*}$$ For example, the "confidence interval" $$(L^L(X), U^L(X))$$ contains the random variable $$L(X)$$ a fraction $$1-\alpha$$ of the time.

Based on all these confidence intervals, we could extend the original confidence interval to $$(L^L(X), U^U(X))$$. I'm not sure what the utility of this is, though.

### TLDR;

Below is a simulation where we repeated an experiment of estimating the mean of a normal distribution with $$\mu = 0$$ and $$\sigma = 1$$. We did 200 repetitions with samples of size 10.

We can indeed see that the estimate of the standard deviation is different each experiment. We are not certain about the exact value of the standard deviation.

But one thing is more or less a constant, that is the probability that the true mean is inside the interval depicted by the error bar.

In this example we made 61 times (30.5%) a wrong estimate of the interval (coloured red/blue when we underestimate or overestimate the mean). For large samples it will become approximately 32% error (see https://en.m.wikipedia.org/wiki/68–95–99.7_rule)

When we interpret the error bars more in this way, as an interval that contains the parameter some amount of time, then the error on error bars is sort of baked into it and is in this expression of error of containing the parameter.

### Interval estimation

Error bars can be seen as a graphical representation of interval estimation. So this discussion about uncertainty in the uncertainty estimates themselves and can be seen as similar to a more general discussion about intervals.

### Standard deviation, standard error, are simple indicators of intervals

When the error bars represent the estimated standard deviation then indeed the error bars themselves have some uncertainty as well. Standard deviations are just a simple way of expressing the uncertainty.

You can have all kinds of intervals like, credible intervals or confidence intervals, in which case the uncertainty is tackled in some way or another.

### Alternative example: Confidence intervals

For instance a confidence interval will contain the correct data point $$\alpha\%$$ of the time. This is one way to represent the certainty and precision in the data. The more precise the data is the smaller that we can make the intervals.

But note that confidence intervals represent the uncertainty in a peculiar way. See Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean? The confidence interval contains the true data point with $$\alpha\%$$ probability when we condition on the model parameters, and not when we condition on the observation.

For particular observations the intervals will be more often wrong than for other observations and the intervals may differ in size (like in your graph). So there is still uncertainty about the intervals. But this uncertainty is already expressed by stating that it is an interval with $$\alpha\%$$ probability.

The error bars based on the 'simple' standard deviation is often very close to a 68% confidence interval (see https://en.m.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule).

### How the 'problem' is solved.

• In the case of the confidence interval the problem can be solved by computing a statistic that is a pivotal quantity.

For instance a t statistic is a ratio of the mean and the observed standard deviation, because both the numerator and denominator in this ratio depend on the variance of the original distribution, the ratio becomes independent on this variance. In this way the uncertainty about the variance of the distribution has been 'eliminated'.

• In the case of the credible interval we use a prior distribution to express the uncertainty about the entire system. In the final computation of the interval based on the posterior distribution, the uncertainty about the interval is included.