Formulating a statement about an estimated confidence interval

Question

Consider a data generating process with a parameter of interest $\theta$. I would like to estimate $\theta$ as precisely as possible and also quantify the estimation imprecision / uncertainty. I obtain an $n$-size i.i.d. sample from which a confidence interval (CI) for parameter $\theta$ is estimated. The estimator of the CI guarantees that in the long run, 95% of such CIs will include the true parameter value. However, I do not know whether the estimate I got from this particular sample includes the true parameter value or not; nor can I make a statement like the probability that $\theta$ lies in this concrete interval is 95%. Not quite a satisfactory answer for me. I would like to say more about the estimated confidence interval (the particular realization of the respective estimator).

But is this all that the frequentist paradigm has to offer on this question? For example, couldn't I say that I am 95% confident that this particular interval includes the true value, based on the long-run properties of the estimator? (For that matter, I would bet 95 against 5 that the interval actually includes the true value and consider this to to be a fair bet.) Is there a "legitimate" way to express this confidence within the frequentist paradigm, and if so, how should I phrase it?

P.S. Yes, I am aware of Bayesian statistics and the corresponding problem and answer formulations. Nevertheless, this question is intentionally about the frequentist paradigm.

Answer 1

The important thing about frequentist statistics is that parameters are not random variables. Rather, they are fixed quantities which happen to be unknown. It's therefore not legitimate to make probability statements about parameters. For example, the statement "the probability that the parameter is between 500 and 507 is 95%" is not a valid statement for a frequentist.

However. That does not rule out any statement about randomness which involves the parameter in some way. To be specific: there could be other sources of randomness in the statement. In my view some authors are too hawkish about banning this (it's much easier to do that than explain the nuance of the issue to students); but frequentists can actually make probability statements about confidence intervals, as long as they are made properly and very carefully.

To wit: you have to be absolutely clear that by "confidence interval", you are talking about the estimator (as you made clear in your question). Estimators are statistics, which means they are functions of the data, where the data comes from a random sample. They are, therefore, random variables.

The statement "the parameter is in the confidence interval" is therefore a random event in the technical sense that it describes a subset of the sample space (i.e. a subset of the possible outcomes of random sampling). Sometimes this event happens; sometimes it does not. And therefore, even for a frequentist, this statement can be assigned a probability; that is, you can say "the probability that the parameter is in the confidence interval is 95%".

Though it's a little more informative to phrase this as the following equivalent statement: "the probability that the confidence interval contains the parameter is 95%". This just makes it linguistically more clear that the confidence interval is the thing "doing" the randomness here, to which a probability is being assigned.

The danger in this statement is that you calculate a specific confidence interval for a specific sample (and get [500, 507] for example), and your reference to "confidence interval" is taken to refer to this interval. But [500, 507] is not a statistic: it is the realisation of a statistic. It's just a pair of numbers. It's not random. And "the probability that the confidence interval contains the parameter is 95%" is not, under that interpretation, a valid statement.

To summarise: yes, a frequentist actually can make statements about the probability of a parameter falling within a confidence interval, as long as it is absolutely clear that by "confidence interval" we are talking about an estimator / a statistic, not a specific estimate / the realisation of a statistic.

Answer 2

But is this all that the frequentist paradigm has to offer on this question?

Yes.

Your two statements:

1. The estimator of the CI guarantees that in the long run, 95% of such CIs will include the true parameter value.

2. I am 95% confident that this particular interval includes the true value, based on the long-run properties of the estimator?

Are exactly equivalent, because the "confident that" of statement 2 is synonymous with the long run probabilities of statement 1, even without the dependent clause of statement 2. It is what it is. That said, the first clause of statement 2 is in my opinion a more concise and readable plain-language expression of a CI for a technically literate audience.

When I motivate CIs for my students I stress the meh of their formal meaning. However, I also offer as an explicitly loosey-goosey alternative definition that CIs provide a plausible range of values for $\theta$. "Loosey-goosey" because that's not the precise definition, and because the meaning of confidence level is, I feel, somewhat obscured in that interpretation. But it helps somewhat.