Why do these statements not follow logically from a 95% CI for the mean?

Question

I've been reading Hoekstra et al's 2014 paper on "Robust misinterpretation of confidence intervals", which I downloaded from Wagenmakers's website.

On the penultimate page the following image appears.

According to the authors, False is the correct answer to all these statements. I am not very sure why the statements are false, and as far as I can tell the rest of the paper does not attempt to explain this.

I believe that 1-2 and 4 aren't correct because they assert something about the probable value of the true mean, when the true mean has a definite value that is unknown. Is this a convincing distinction?

Regarding 3, I understand that one is not meant to make assertions about the likelihood the null hypothesis is incorrect, though I'm not so sure of the reason why.

Similarly 6 can't be true because it implies that the true mean is changing from experiment to experiment.

The one I really don't understand at all is 5. Why is that one wrong? If I have a process that 95% of the time produces CIs that contain the true mean, why shouldn't I say I have 95% confidence the population value is between 0.1 and 0.4? Is it because we might have some special information about the sample we just took that would make us think it's likely to be one of the 5% that does not contain the true mean? For example, 0.13 is included in the confidence interval and for some reason 0.13 is not considered a plausible value within some specific research context, e.g. because that value would conflict with previous theory.

What does confidence mean in this context, anyway?

Answer 1

The very meaning of question (5) depends on some undisclosed interpretation of "confidence." I searched the paper carefully and found no attempt to define "confidence" or what it might mean in this context. The paper's explanation of its answer to question (5) is

"... [it] mentions the boundaries of the CI whereas ... a CI can be used to evaluate only the procedure and not a specific interval."

This is both specious and misleading. First, if you cannot evaluate the result of the procedure, then what good is the procedure in the first place? Second, the statement in the question is not about the procedure, but about the reader's "confidence" in its results.

The authors defend themselves:

"Before proceeding, it is important to recall the correct definition of a CI. A CI is a numerical interval constructed around the estimate of a parameter. Such an interval does not, however, directly indicate a property of the parameter; instead, it indicates a property of the procedure, as is typical for a frequentist technique."

Their bias emerges in the last phrase: "frequentist technique" (written, perhaps, with an implicit sneer). Although this characterization is correct, it is critically incomplete. It fails to notice that a confidence interval is also a property of the experimental methods (how samples were obtained and measured) and, more importantly, of nature herself. That is the only reason why anyone would be interested in its value.

I recently had the pleasure of reading Edward Batschelet's Circular Statistics in Biology (Academic Press, 1981). Batschelet writes clearly and to the point, in a style directed at the working scientist. Here is what he says about confidence intervals:

"An estimate of a parameter without indications of deviations caused by chance fluctuations has little scientific value. ...

"Whereas the parameter to be estimated is a fixed number, the confidence limits are determined by the sample. They are statistics and, therefore, dependent on chance fluctuations. Different samples drawn from the same population lead to different confidence intervals."

[The emphasis is in the original, at pp 84-85.]

Notice the difference in emphasis: whereas the paper in question focuses on the procedure, Batschelet focuses on the sample and specifically on what it can reveal about the parameter and how much that information can be affected by "chance fluctuations." I find this unabashedly practical, scientific approach far more constructive, illuminating, and--ultimately--useful.

A fuller characterization of confidence intervals than offered by the paper therefore would have to proceed something like this:

A CI is a numerical interval constructed around the estimate of a parameter. Anyone agreeing with the assumptions underlying the CI construction is justified in saying they are confident that the parameter lies within the interval: this is the meaning of "confident." This meaning is broadly in accord with conventional non-technical meanings of confidence because under many replications of the experiment (whether or not they actually take place) the CI, although it will vary, is expected to contain the parameter most of the time.

In this fuller, more conventional, and more constructive sense of "confidence," the answer to question (5) is true.

Answer 2

Questions 1-2, 4: in frequentist analysis, the true mean is not a random variable, thus thes probabilities are not defined, whereas in Bayesian analysis the probabilities would depend on the prior.

Question 3: For example, consider a case where we know for sure It would still be possible to get these results, but rather unreasonable to say that the null hypothesis is 'unlikely' to be true. We obtained data that is unlikely to occur if the null hypothesis is true, but this does not imply that the null hypothesis is unlikely to be true.

Question 5: This is a bit questionable as this depends on the definition of "we can be p % confident." If we define the statement to mean the thing that is inferred from p % confidence intervals, the statement is by definition correct. The typical pro-Bayesian argument states that people tend to interpret these statements intuitively to mean "the probability is p %", which would be false (compare answers to 1-2,4).

Question 6: Your explanation "it implies that the true mean is changing from experiment to experiment" is exactly correct.

The article was recently discussed in Andrew Gelman's blog (http://andrewgelman.com/2014/03/15/problematic-interpretations-confidence-intervals/). For example, the issue regarding the interpretation of the statement in question 5 is discussed in the comments.

Answer 3

Without any formal definition of what it means to be "95% confident", what justification is there for labelling #5 true or false? A layman would doubtless misinterpret it as synonymous with a 95% probability of the mean's being in that interval: but some people do use it in the sense of having used an interval-generating method whose intervals contain the true mean 95% of the time, precisely to avoid talking about the probability distribution of an unknown parameter; which seems a natural enough extension of the terminology.

The similar structure of the preceding statement (#4) might have encouraged respondents to try to draw a distinction between "we can be 95% confident" & "there is a 95% probability" even if they hadn't entertained the idea before. I had expected this tricksiness to lead to #5 having the highest proportion in agreement—looking at the paper, I found out I was wrong, but noticed that at least 80% read the questionnaire in a Dutch version, which perhaps should raise questions about the pertinence of the English translation.

Answer 4

Here is the definition of a confidence interval, from B. S. Everitt's Dictionary of Statistics:

"A range of values, calculated from the sample observations, that are believed, with a certain probability, to contain the true parameter value. A 95% CI, for example, implies that were the estimation process repeated again and again, then 95% of the calculated intervals would be expected to contain the true parameter value. Note that the stated probability level refers to properties of the interval and not to the parameter itself, which is not considered a random variable"

A very common misconception is to confuse the meaning of a confidence interval with that of a credible interval, AKA "Bayesian confidence interval", which does make statements similar to those in the questions.

I have heard that confidence intervals are often similar to credible intervals that were derived from an uninformative prior, but that was told to me anecdotally (albeit by a guy I respect a lot), and I don't have details or a cite.

Answer 5

Regarding the intuition for the falsehood of Question 5, I obtain the following discussion on this topic from here

It is correct to say that there is a 95% chance that the confidence interval you calculated contains the true population mean. It is not quite correct to say that there is a 95% chance that the population mean lies within the interval.

What's the difference? The population mean has one value. You don't know what it is (unless you are doing simulations) but it has one value. If you repeated the experiment, that value wouldn't change (and you still wouldn't know what it is). Therefore it isn't strictly correct to ask about the probability that the population mean lies within a certain range. In contrast, the confidence interval you compute depends on the data you happened to collect. If you repeated the experiment, your confidence interval would almost certainly be different. So it is OK to ask about the probability that the interval contains the population mean.

Now to your specific questions about 5. Why is it wrong...

1. Is it because we might have some special information about the sample we just took that would make us think it's likely to be one of the 5% that does not contain the true mean? No, rather, I think it is because the true mean is not a random variable, but the confidence interval is a function of the data.
2. What does confidence mean in this context, anyway? A confidence interval enables (confidently - if you trust your assumptions) you to make a claim that the interval covers the true parameter. The interpretation reflects the uncertainty in the sampling procedure; a confidence interval of $100(1-\alpha)$% asserts that [you can be confident that], in the long run, $100(1-\alpha)$% of the realized confidence intervals cover the true parameter.

As a side note (mentioned in other answers to this question), a credible interval, a concept from Bayesian statistics, do predict that the true value of the parameter has a particular probability of being in the confidence interval given the data actually obtained. Perhaps you can obtain more background on this from Gelman's blog.