Usefulness of the confidence interval

Question

Confidence interval (CI) of the true value of a parameter is estimated using a sample. The interpretation of that CI is that many (say 100) such samples are taken, about 95% of the time the confidence interval formed from those samples would contain the true parameter value. A variation of this definition can be seen. This link has some answers.

I was wondering if there is any use of such a definition given that I would have to repeat the sampling/experiment many times (say 100 times). The explanation I settled to is that if a random and highly representative sample is taken/obtained, the CI that is formed from it would be a representative of those many samples (say 100). I cannot say the true value will be in the middle of that CI since true value is unknown.

But what else can we say? What is the use of the one CI I have constructed?

Answer 1

A confidence interval is typically more useful than a hypothesis test. A hypothesis test tells you if you can rule out a specific null hypothesis (typically, $0$). On the other hand a confidence interval demarcates an infinite set of values that, if they had been your null, would have been rejected similarly. (Likewise, it gives the set of potential null values that would not have been rejected.) For example, consider a 95% confidence interval for a mean $(.1, .9)$. The p-value for the (nil) null is $<.05$, but the confidence interval also lets you know that if your null value had been $1.0$, it would have been rejected as well.

A confidence interval also helps you differentiate between high level of confidence and a large effect. People are often impressed by an effect that is highly significant (e.g., $p<.0001$), and conclude that it must be really important. However, p-values conflate the size of the effect with the clarity of the effect. You can get a low p-value because the effect is large or because the effect is small, but you have very many data. This isn't ambiguous if you're looking at a confidence interval that is, say, $(.05, .15)$ versus $(5, 15)$.

In addition, a confidence interval is usually more informative than a point estimate. Although the point estimate returned by some fitting function will typically be the single most likely value (conditional on your data and your model), it isn't actually very likely to be the true value. There is, as you mention, no guarantee that the true value lies within a, say, 95% confidence interval (for instance, there isn't a 95% chance that the true value is in a 95% confidence interval¹). That said, it is more likely that the true value lies within the interval than it is that the point estimate is the true value—this should be obvious since the point estimate is within the interval. In fact, you could think of a point estimate as a $0\%$ confidence interval.

_{1. Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?}

Answer 2

I think it's best understood with a simple example.

Imagine you are on a farm that raises sheep. The farm has a lot of sheep but you only observe 5. Out of those 5 sheep, 1 sheep was black and 4 sheep were white. And you are interested in the real proportion of black/white sheep within the farm. What can you tell about that proportion based on the 5-sheep sample you just saw?

One question might be - is it reasonable to think that the true proportion of black/white sheep on the farm is equal (50/50)? To answer this question you can calculate the probability of seeing 4 white and 1 black sheep (or a more extreme difference) if the true proportion of black sheep is 0.5. This is a p-value.

Another question is the inverse of the first one - given the sample of sheep you just saw - what proportions are not unreasonable to consider? You can dismiss the possibility of the farm having only white sheep, since you already saw one black sheep. We can say that you dismiss it because the probability to see 1 black and 4 white sheep, if all the sheep are white, is 0. But how to go further? Well, you can calculate all the proportions for which the chance of seeing 1 black and 4 white sheep is greater than 5%. Those would be the "reasonable proportions" based on your observation. This is a 95% confidence interval.

So you can think about the confidence interval as a sort of philosophical tool that, given certain assumptions and under certain conditions, allows you to expand your inductive reasoning - going from observations to generalizations. As you can see, there is no need to repeat anything multiple times at all.

---

Disclaimer: the above example is simplified for brevity. In particular - it doesn't mention the assumption that your observations of sheep are independent of one another. And that in two-tailed scenarios you would also have to consider 1 white / 4 black cases.

Answer 3

The answer by gung-Reinstate Monica is fine. I add the following. In reality there is no such thing as a true model. There is no true parameter either (as such a parameter is only defined within a model). What we do is we use models to think about a reality that is different, however we don't have better tools than artificial formal models to make quantitative statements.

So let's imagine that what we observe in reality behaves like data generating process, potentially infinitely repeatable, modelled by some distribution with a parameter $\mu$, say, that we identify, in our brains, with some real quantity we are interested in. What we want to use the model for is to quantify the uncertainty, because we think of the real process as having some random variation, i.e., we will get other numbers when we do the same thing next time, the precise explanation of which is either unobservable or not of interest or not worth the effort finding out. What we want is some indication how far away from reality we might be with our best guess (the parameter estimate), because we are convinced, normally from experience, that the data that we have will not tell us precisely what is going on, however they hint at it, with some possible variation. The confidence interval is a way to use model-thinking to quantify this. It asks: If the model is true, which parameter values could have given rise to the data that we have observed?

The confidence interval gives us a set of parameter values that, were the model true, are all compatible with what was observed, i.e., what was observed is a realistic, at least fairly typical thing if any of the values in the confidence interval were true, and pretty atypical if other values were true. Therefore it gives a set of "realistic" parameter values. That said, as I wrote before, none of these is really true, however as long as we think of the real situation in terms of the model, it makes sense to think of the model taking one of these parameter values. That may look disappointingly far away from reality, but it is hard to do better really. That's the nature of models. (Epistemic Bayesian logic would be an alternative, but it comes with problems that turn out to be fairly similar if you look at them in the right way.)

The positive side of this modest way of interpreting things is that it doesn't rely on the model setup being literally fulfilled. Particularly there is no need to indeed repeat the experiment to give the result a meaning. This is an imagination anyway, a tool for thinking about the situation, the possibility of which can be more or less close to reality. (Obviously the advantage of indeed being able to repeat this several times is that we have better ways to assess if the model world is reasonably in line with the real world.)

Issues: (1) As I wrote, the model is in fact not true. This is not generally a problem (it's the nature of models actually), but it is a problem if it is violated in ways that make thinking in terms of the specific model strongly misleading. A typical issue is if data are strongly positively correlated if in fact the model assumes them independent - you'll end up with a far too narrow confidence interval.

(2) Confidence intervals oversimplify things in the sense that you specify a confidence level, and then parameters are either in or out. However, the difference in compatibility with the data of parameters that are borderline in or borderline out, respectively, is not that big. If your confidence interval is, say, $[5,10]$, it is not really appropriate to think that 5.1 is a perfectly realistic value whereas 4.9 is totally not. Rather 4.9 is just slightly more unrealistic than 5.1, which is conceivable as true but may (depending on the exact model, statistic used etc.) be substantially less realistic than, say, 7.