How to interpret confidence interval of the difference in means in one sample T-test?

Question

SPSS provides the output "confidence interval of the difference means." I have read in some places that it means "95 times out of 100, our sample mean difference will be between between these bounds" I find this unclear. Can anyone suggest clearer wording to explain "confidence interval of the difference in means"? This output appears in the context of a one-sample t-test.

Answer 1

This is not an easy thing, even for respected statisticians. Look at one recent attempt by Nate Silver:

... if I asked you to tell me how often your commute takes 10 minutes longer than average — something that requires some version of a confidence interval — you’d have to think about that a little bit, ...

(from the FiveThirtyEight blog in the New York Times, 9/29/10.) This is not a confidence interval. Depending on how you interpret it, it's either a tolerance interval or a prediction interval. (Otherwise there's nothing the matter with Mr. Silver's excellent discussion of estimating probabilities; it's a good read.) Many other web sites (particularly those with an investment focus) similarly confuse confidence intervals with other kinds of intervals.

The New York Times has made efforts to clarify the meaning of the statistical results it produces and reports on. The fine print beneath many polls includes something like this:

In theory, in 19 cases out of 20, results based on such samples of all adults will differ by no more than three percentage points in either direction from what would have been obtained by seeking to interview all American adults.

(e.g., How the Poll Was Conducted, 5/2/2011.)

A little wordy, perhaps, but clear and accurate: this statement characterizes the variability of the sampling distribution of the poll results. That's getting close to the idea of confidence interval, but it is not quite there. One might consider using such wording in place of confidence intervals in many cases, however.

When there is so much potential confusion on the internet, it is useful to turn to authoritative sources. One of my favorites is Freedman, Pisani, & Purves' time-honored text, Statistics. Now in its fourth edition, it has been used at universities for over 30 years and is notable for its clear, plain explanations and focus on classical "frequentist" methods. Let's see what it says about interpreting confidence intervals:

The confidence level of 95% says something about the sampling procedure...

[at p. 384; all quotations are from the third edition (1998)]. It continues,

If the sample had come out differently, the confidence interval would have been different. ... For about 95% of all samples, the interval ... covers the population percentage, and for the other 5% it does not.

[p. 384]. The text says much more about confidence intervals, but this is enough to help: its approach is to move the focus of discussion onto the sample, at once bringing rigor and clarity to the statements. We might therefore try the same thing in our own reporting. For instance, let's apply this approach to describing a confidence interval of [34%, 40%] around a reported percentage difference in a hypothetical experiment:

"This experiment used a randomly selected sample of subjects and a random selection of controls. We report a confidence interval from 34% to 40% for the difference. This quantifies the reliability of the experiment: if the selections of subjects and controls had been different, this confidence interval would change to reflect the results for the chosen subjects and controls. In 95% of such cases the confidence interval would include the true difference (between all subjects and all controls) and in the other 5% of cases it would not. Therefore it is likely--but not certain--that this confidence interval includes the true difference: that is, we believe the true difference is between 34% and 40%."

(This is my text, which surely can be improved: I invite editors to work on it.)

A long statement like this is somewhat unwieldy. In actual reports most of the context--random sampling, subjects and controls, possibility of variability--will already have been established, making half of the preceding statement unnecessary. When the report establishes that there is sampling variability and exhibits a probability model for the sample results, it is usually not difficult to explain a confidence interval (or other random interval) as clearly and rigorously as the audience needs.

Answer 2

From a pedantic technical viewpoint, I personally don't think there is a "clear wording" of the interpretation of confidence intervals.

I would interpret a confidence interval as: there is a 95% probability that the 95% confidence interval covers the true mean difference

An interpretation of this is that if we were to repeat the whole experiment $N$ times, under the same conditions, then we would have $N$ different confidence intervals. The confidence level is the proportion of these intervals which contain the true mean difference.

My own personal quibble with the logic of such reasoning is that this explanation of confidence intervals requires us to ignore the other $N-1$ samples when calculating our confidence interval. For instance if you had a sample size of 100, would you then go and calculate 100 "1-sample" 95% confidence intervals?

But note that this is all in the philosophy. Confidence intervals are best left vague in the explanation I think. They give good results when used properly.

Answer 3

The rough answer to the question is that a 95% confidence interval allows you to be 95% confident that the true parameter value lies within the interval. However, that rough answer is both incomplete and inaccurate.

The incompleteness lies in the fact that it is not clear that "95% confident" means anything concrete, or if it does, then that concrete meaning would not be universally agreed upon by even a small sample of statisticians. The meaning of confidence depends on what method was used to obtain the interval and on what model of inference is being used (which I hope will become clearer below).

The inaccuracy lies in the fact that many confidence intervals are not designed to tell you anything about the location of the true parameter value for the particular experimental case that yielded the confidence interval! That will be surprising to many, but it follows directly from the Neyman-Pearson philosophy that is clearly stated in this quote from their 1933 paper "On the Problem of the Most Efficient Tests of Statistical Hypotheses":

We are inclined to think that as far as a particular hypothesis is concerned, no test based upon the theory of probability can by itself provide any valuable evidence of the truth or falsehood of that hypothesis.

But we may look at the purpose of tests from another view-point. Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.

Intervals that are based on the 'inversion' of N-P hypothesis tests will therefore inherit from that test the nature of having known long-run error properties without allowing inference about the properties of the experiment that yielded them! My understanding is that this protects against inductive inference, which Neyman apparently considered to be an abomination.

Neyman explicitly lays claim to the term ‘confidence interval’ and to the origin of the theory of confidence intervals in his 1941 Biometrika paper “Fiducial argument and the theory of confidence intervals”. In a sense, then, anything that is properly a confidence interval plays by his rules and so the meaning of an individual interval can only be expressed in terms of the long run rate at which intervals calculated by that method contain (cover) the relevant true parameter value.

We now need to fork the discussion. One strand follows the notion of ‘coverage’, and the other follows non-Neymanian intervals that are like confidence intervals. I will defer the former so that I can complete this post before it becomes too long.

There are many different approaches that yield intervals that could be called non-Neymanian confidence intervals. The first of these is Fisher’s fiducial intervals. (The word ‘fiducial’ may scare many and elicit derisive smirks from others, but I will leave that aside...) For some types of data (e.g. normal with unknown population variance) the intervals calculated by Fisher’s method are numerically identical to the intervals that would be calculated by Neyman’s method. However, they invite interpretations that are diametrically opposed. Neymanian intervals reflect only long run coverage properties of the method, whereas Fisher’s intervals are intended to support inductive inference concerning the true parameter values for the particular experiment that was performed.

The fact that one set of interval bounds can come from methods based on either of two philosophically distinct paradigms leads to a really confusing situation--the results can be interpreted in two contradictory ways. From the fiducial argument there is a 95% likelihood that a particular 95% fiducial interval will contain the true parameter value. From Neyman’s method we know only that 95% of intervals calculated in that manner will contain the true parameter value, and have to say confusing things about the probability of the interval containing the true parameter value being unknown but either 1 or 0.

To a large extent, Neyman’s approach has held sway over Fisher’s. That is most unfortunate, in my opinion, because it does not lead to a natural interpretation of the intervals. (Re-read the quote above from Neyman and Pearson and see if it matches your natural interpretation of experimental results. Most likely it does not.)

If an interval can be correctly interpreted in terms of global error rates but also correctly in local inferential terms, I don’t see a good reason to bar interval users from the more natural interpretation afforded by the latter. Thus my suggestion is that the proper interpretation of a confidence interval is BOTH of the following:

• Neymanian: This 95% interval was constructed by a method that yields intervals that cover the true parameter value on 95% of occasions in the long run (...of our statistical experience).

• Fisherian: This 95% interval has a 95% probability of covering the true parameter value.

(Bayesian and likelihood methods will also yield intervals with desirable frequentist properties. Such intervals invite slightly different interpretations that will both probably feel more natural than the Neymanian.)

Answer 4

The meaning of a confidence interval is: if you were to repeat your experiment in the exact same way (i.e.: the same number of observations, drawing from the same population, etc.), and if your assumptions are correct, and you would calculate that interval again in each repetition, then this confidence interval would contain the true prevalence in 95% of the repetitions (on average).

So, you could say you are 95% certain (if your assumptions are correct etc.) that you have now constructed an interval that contains the true prevalence.

This is typically stated as: with 95% confidence, between 4.5 and 8.3% of children of mothers who smoked throughout pregnancy become obese.

Note that this is typically not interesting in itself: you probably want to compare this to prevalence in children of mothers who didn't smoke (odds ratio, relative risk, etc.)

Answer 5

If the true mean difference is outside of this interval, then there is only a 5% chance that the mean difference from our experiment would be so far away from the true mean difference.

Answer 6

My Interpretation: If you conduct the experiment N times ( where N tends to infinity) then out of these large number of experiments 95% of the experiments will have confidence intervals which lie within these 95% limits. More clearly, lets say those limits are "a" and "b" then 95 out of 100 times your sample mean difference will lie between "a" and "b".I assume that you understand that different experiment can have different samples to cover out of the whole population.