Error bars on error bars?

Question

Inspired by my recent attendance at an environmental toxicology conference, I have the following question about error bars:

Let's say that I'm drawing samples from some unknown distribution, with finite mean and variance. I want to present the sample mean, and add some error bars. Since I don't know much about the underlying distribution, I just add error bars showing +/- the standard devaition of the samples.

My question is, is there any way I could meaningfully indicate how certain I am of those error bars? Adding error bars to the error bars, so to speak.

As as example, I have drawn 5 samples from some distribution, and I have repeated this 5 times. The sample means, and error bars of +/- the sample standard deviations, are shown below.

We can see that by chance, these sample means and error bars look quite different, and not really mutually compatible. Of course 5 samples isn't very much, but if my samples are obtained via some convoluted experimental procedure (capturing a wild animal and taking a blood sample, for example), it might not be an easy option to get more samples.

Update:

Just to add some notes on how I was thinking:

Coming from a computational physics background myself, I'm used to Monte Carlo methods, and the $1/\sqrt{N}$-error which follows from the central limit theorem. So just like the error in the sample mean has an expected distribution, I thought perhaps it would make sense to ask about the expected error in the sample standard deviation. Of course, the problem is that the distribution of the error in the sample mean is expressed in terms of the (unknown) variance of the underlying distribution, and hence I am left taking the standard deviation of the sample, or something along those lines.

But still, I thought there ought to be some way of indicating that my sample standard deviation is itself quite uncertain, due to the small $N$. But perhaps the only way is simply to list $N$, and be explicit about what the error bars show.

Answer 1

You are interested in standard errors, which describe the variability in a parameter estimate, and are related to your sampling approach. This is distinct from the parameters themselves (e.g. mean and standard deviation), which are functions of the underlying population only, and are not dependent on how large your sample is.

Your current plot shows two values per group, the sample mean and sample standard deviation, about which there is no uncertainty (it is whatever you observe it to be). Assuming appropriate random sampling, you can use these values to make inference about the unobservable quantities of the population mean and population standard deviation for each group. You can use common tools like standard error or 95% confidence intervals to estimate the precision of your parameter estimates.

It would be odd to try to represent this as error bars on error bars, but it would be perfectly reasonable to list the mean and standard deviation for each group, along with the 95% CI of each parameter estimate. This can help you to decide if the means/standard deviations observed in Groups C and D, for example, represent true differences in the underlying population parameters, or if the apparent differences represent normal variation that would be expected with a small sample size.

Answer 2

The objects we use to make inferences (e.g., estimates, confidence intervals, error bars, test statistics, p-values, etc.) are statistics, meaning that they are functions of the observed data. Since they are already functions of the observed data, these objects do not have any uncertainty in them --- they represent inferences about uncertain values, but there no uncertainty in the statistics themselves. We do not form error bars on error bars because there is no uncertainty in the error bars to begin with, because they are formed as a function of the observed data.

As a minor point, it is generally suboptimal practice to use error bars to show a deviation of plus/minus one (estimated) standard deviation. Usually you are better off using these values and other statistics to form an appropriate confidence interval for the uncertain value of interest, and using the error bars to show the confidence interval. In either case you should label your plot appropriately so that the reader understands what the error bars represent.

Answer 3

The short answer is "no."

However you construct your error bars, they are a rule. You cannot be unsure of them. Let us imagine that they are confidence intervals. There are multiple standard ways to create confidence intervals. They are different rules with slightly different properties. However, they are a chosen rule.

Other ways to construct error bars exist as well, such as adding plus or minus one standard deviation. It is still a rule.

You know the answer exactly. They are not uncertain.

What they are reflecting is the random elements of the samples seen. If they are a $1-\alpha$ percent confidence interval, there is a guarantee that the confidence intervals cover the parameter at least $1-\alpha$ percent of the time. There is no guarantee that it covers it for this sample. Even with a set of five samples, none of them may cover the parameter, the guarantee is over infinite repetition.

Each way you could construct an error bar has some form of optimality principle behind it. So, error bars satisfy some optimality condition that is good on average.

All of them are a statement of the best estimator of the range in which a parameter sits, given a model and a loss function.

Your error bars are a statement of uncertainty.

Answer 4

The traditional design of error bars gives an unfortunate impression of some linear distribution of uncertainty, and places a lot of visual emphasis on the the end of the bar, which is where the distribution of the location of your estimate is least likely. Clause Wilke (in his book Fundamentals of Data Visualization, in the chapter Visualizing uncertainty) shows some graphical alternatives to traditional error bars that convey something of the distribution of uncertainty in an estimate:

Image by Claus Wilke, used under an Attribution-NonCommercial-NoDerivatives 4.0 International licence. Original available at https://clauswilke.com/dataviz/visualizing-uncertainty.html

The "graded error bars" in (a) and (b) are formed by plotting the 90%, 95% and 99% CIs simultaneously. Thom Baguley discusses a similar approach he terms "tiered error bars" and provides example R code here: https://seriousstats.wordpress.com/2012/06/21/confidence-intervals-with-tiers/ , although I first saw such an approach being used by Andrew Gelman in his textbook Data Analysis Using Regression and Multilevel/Hierarchical Models.

Answer 5

Review of confidence intervals

Let $\theta \in \mathbb{R}$ be a parameter of interest which we study based on a random variable $X$. An exact $1-\alpha$ confidence interval $(L(X),U(X)$ is defined by the property that \begin{equation*} \mathbb{P}\left[ L(X) < \theta < U(X) \right] = 1-\alpha, \end{equation*} where $L$ is the lower endpoint and $R$ is the upper endpoint of the confidence interval.

The plot shown in the question illustrates that $L$ and $U$ are random variables. This is certainly the case, as they depend on the random variable $X$. However, a fraction of the confidence intervals $(L(X),U(X))$ contain $\theta$. By construction, the fraction is exactly $1-\alpha$. When $\alpha=0.05$, this is $95\%$ of the confidence intervals.

Error bars on error bars

This procedure makes perfect sense if the target of inference is $\theta$ - which is what we stated above. However, you may also be interested in the endpoints $L(X)$ and $U(X)$ themselves. Then you can construct a "confidence intervals" $(L^L(X), U^L(X))$ and $(L^U(X), U^U(X))$ such that \begin{equation*} \mathbb{P} \left[L^L(X) < L(X) < U^L(X) \right] = 1-\alpha \end{equation*} and \begin{equation*} \mathbb{P} \left[L^U(X) < U(X) < U^U(X) \right] = 1-\alpha. \end{equation*} For example, the "confidence interval" $(L^L(X), U^L(X))$ contains the random variable $L(X)$ a fraction $1-\alpha$ of the time.

Based on all these confidence intervals, we could extend the original confidence interval to $(L^L(X), U^U(X))$. I'm not sure what the utility of this is, though.

Answer 6

TLDR;

Below is a simulation where we repeated an experiment of estimating the mean of a normal distribution with $\mu = 0$ and $\sigma = 1$. We did 200 repetitions with samples of size 10.

We can indeed see that the estimate of the standard deviation is different each experiment. We are not certain about the exact value of the standard deviation.

But one thing is more or less a constant, that is the probability that the true mean is inside the interval depicted by the error bar.

In this example we made 61 times (30.5%) a wrong estimate of the interval (coloured red/blue when we underestimate or overestimate the mean). For large samples it will become approximately 32% error (see https://en.m.wikipedia.org/wiki/68–95–99.7_rule)

When we interpret the error bars more in this way, as an interval that contains the parameter some amount of time, then the error on error bars is sort of baked into it and is in this expression of error of containing the parameter.

Interval estimation

Error bars can be seen as a graphical representation of interval estimation. So this discussion about uncertainty in the uncertainty estimates themselves and can be seen as similar to a more general discussion about intervals.

Standard deviation, standard error, are simple indicators of intervals

When the error bars represent the estimated standard deviation then indeed the error bars themselves have some uncertainty as well. Standard deviations are just a simple way of expressing the uncertainty.

You can have all kinds of intervals like, credible intervals or confidence intervals, in which case the uncertainty is tackled in some way or another.

Alternative example: Confidence intervals

For instance a confidence interval will contain the correct data point $\alpha\%$ of the time. This is one way to represent the certainty and precision in the data. The more precise the data is the smaller that we can make the intervals.

But note that confidence intervals represent the uncertainty in a peculiar way. See Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean? The confidence interval contains the true data point with $\alpha\%$ probability when we condition on the model parameters, and not when we condition on the observation.

For particular observations the intervals will be more often wrong than for other observations and the intervals may differ in size (like in your graph). So there is still uncertainty about the intervals. But this uncertainty is already expressed by stating that it is an interval with $\alpha\%$ probability.

The error bars based on the 'simple' standard deviation is often very close to a 68% confidence interval (see https://en.m.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule).

How the 'problem' is solved.

• In the case of the confidence interval the problem can be solved by computing a statistic that is a pivotal quantity.

  For instance a t statistic is a ratio of the mean and the observed standard deviation, because both the numerator and denominator in this ratio depend on the variance of the original distribution, the ratio becomes independent on this variance. In this way the uncertainty about the variance of the distribution has been 'eliminated'.

• In the case of the credible interval we use a prior distribution to express the uncertainty about the entire system. In the final computation of the interval based on the posterior distribution, the uncertainty about the interval is included.