# Why not using confidence interval to calculate the repeatability?

When calculating the repeatability, like when you want to know how good the measurement of the length of a ruler is, I often read that it is sufficient to calculate the standard deviation.

I wonder about why not using a confidence interval? Or also not the standard error?

• What confidence interval or standard error do you have in mind? What parameter would it estimate and why would that be related to repeatability of a measurement?
– whuber
Commented May 4, 2020 at 11:21
• There are more? Is the second question something like an answer so when I can answer this I also answer my own question?
– Ben
Commented May 4, 2020 at 11:54

I'm not familiar with measures of repeatability, but what I think you're getting at is measuring uncertainty. I think this is a good opportunity to talk about uncertainty at various levels.

Let's say were are measuring the precision of some instrument, like a ruler, where we make measurements and compare with some ground truth. We can estimate the average error (which is a parameter) of the ruler by using the individual error measurements we make.

There are different kinds of uncertainty in this experiment. There is:

1. Uncertainty in the data: What is the size of the next error I will observe?, and

2. Uncertainty in the parameters: Were I to perform my experiment again, what would my estimated average error be?

Confidence intervals, which use the standard error, give a measure of uncertainty for the second kind. They tell you, given the experiment you just performed, what are some of the values of the average error consistent with the data you just observed (I'm being a bit fast and loose with the definition of the confidence interval). What a confidence interval does not tell you is where you are likely to see new observations.

This is easily demonstrated. Consider 1000 data points from a standard normal. A confidence interval for data from this distribution might be $$[-2/\sqrt{1000}, 2/\sqrt{1000}]$$. However, we know we can (and often will) observe data outside this interval.

On the other hand, the standard deviation gives a measure of uncertainty of the first kind. The standard deviation is the square root of the variance, which is the expectation of the squared distance of observations from the mean (wow, that's a mouthful). If you wanted an interval estimator of uncertainty of the first kind, you could make use of tolerance interval or similar estimators.

So, in conclusion, confidence intervals don't measure uncertainty in the observations. They measure uncertainty in the parameters. If you want uncertainty in the observations, its best to use something on the same scale as the observations, and that would be the standard deviation.

• Thank you, I got that! So a confidence interval makes more sense when I want to know what the length of the ruler is, so I measure it 10 times and the CI would tell me how where 95% of the measurements will fall. Contrary to that, the standard deviation tells me how good I measure. Is that correct?
– Ben
Commented May 4, 2020 at 12:00
• That isn't quite correct. A confidence interval tells you what values of the parameter (e.g. the mean) are consistent with the data you see. The standard deviation is a measure of spread, not a measure of quality. I would encourage you to perhaps revisit some of the material from an introductory resource on statistics. Commented May 4, 2020 at 12:15
• ok, thanks, I will look them up! Another a-bit-far-fetched question: I stumbled over the qualityTools for R and saw it can be used to check repeatability (also for experiment designs), explained here: rdrr.io/cran/qualityTools Are you familiar with that? I just want to know if it makes sense to use the cg-function where the target would be the mean of the data? It sounds legit for me..
– Ben
Commented May 4, 2020 at 12:25
• I'm not familiar with that library. I'm sorry. Commented May 4, 2020 at 12:51