# How to interpret confidence interval of the difference in means in one sample T-test?

SPSS provides the output "confidence interval of the difference means." I have read in some places that it means "95 times out of 100, our sample mean difference will be between between these bounds" I find this unclear. Can anyone suggest clearer wording to explain "confidence interval of the difference in means"? This output appears in the context of a one-sample t-test.

-
What is your intepretation? –  mpiktas Oct 7 '11 at 12:18
Note that there is nothing special about this being a proportion: a CI for the estimate of anything will be interpreted in a similar manner. (However, different procedures may be used to construct the CI, depending on what is being estimated.) Consequently, this question is exactly the same as previous questions asking for interpretations of CIs. –  whuber Oct 7 '11 at 14:34

This is not an easy thing, even for respected statisticians. Look at one recent attempt by Nate Silver:

... if I asked you to tell me how often your commute takes 10 minutes longer than average — something that requires some version of a confidence interval — you’d have to think about that a little bit, ...

(from the FiveThirtyEight blog in the New York Times, 9/29/10.) This is not a confidence interval. Depending on how you interpret it, it's either a tolerance interval or a prediction interval. (Otherwise there's nothing the matter with Mr. Silver's excellent discussion of estimating probabilities; it's a good read.) Many other web sites (particularly those with an investment focus) similarly confuse confidence intervals with other kinds of intervals.

The New York Times has made efforts to clarify the meaning of the statistical results it produces and reports on. The fine print beneath many polls includes something like this:

In theory, in 19 cases out of 20, results based on such samples of all adults will differ by no more than three percentage points in either direction from what would have been obtained by seeking to interview all American adults.

(e.g., How the Poll Was Conducted, 5/2/2011.)

A little wordy, perhaps, but clear and accurate: this statement characterizes the variability of the sampling distribution of the poll results. That's getting close to the idea of confidence interval, but it is not quite there. One might consider using such wording in place of confidence intervals in many cases, however.

When there is so much potential confusion on the internet, it is useful to turn to authoritative sources. One of my favorites is Freedman, Pisani, & Purves' time-honored text, Statistics. Now in its fourth edition, it has been used at universities for over 30 years and is notable for its clear, plain explanations and focus on classical "frequentist" methods. Let's see what it says about interpreting confidence intervals:

The confidence level of 95% says something about the sampling procedure...

[at p. 384; all quotations are from the third edition (1998)]. It continues,

If the sample had come out differently, the confidence interval would have been different. ... For about 95% of all samples, the interval ... covers the population percentage, and for the other 5% it does not.

[p. 384]. The text says much more about confidence intervals, but this is enough to help: its approach is to move the focus of discussion onto the sample, at once bringing rigor and clarity to the statements. We might therefore try the same thing in our own reporting. For instance, let's apply this approach to describing a confidence interval of [34%, 40%] around a reported percentage difference in a hypothetical experiment:

"This experiment used a randomly selected sample of subjects and a random selection of controls. We report a confidence interval from 34% to 40% for the difference. This quantifies the reliability of the experiment: if the selections of subjects and controls had been different, this confidence interval would change to reflect the results for the chosen subjects and controls. In 95% of such cases the confidence interval would include the true difference (between all subjects and all controls) and in the other 5% of cases it would not. Therefore it is likely--but not certain--that this confidence interval includes the true difference: that is, we believe the true difference is between 34% and 40%."

(This is my text, which surely can be improved: I invite editors to work on it.)

A long statement like this is somewhat unwieldy. In actual reports most of the context--random sampling, subjects and controls, possibility of variability--will already have been established, making half of the preceding statement unnecessary. When the report establishes that there is sampling variability and exhibits a probability model for the sample results, it is usually not difficult to explain a confidence interval (or other random interval) as clearly and rigorously as the audience needs.

-
Thanks Whuber, I understand confidence intervals for a mean quite well. It is the CI for a difference in means (between a sample and pop) where I become confused. –  Anne Jun 13 '11 at 14:50
@Anne What are you referring to? Neither your question nor any of the replies refers to a difference between a sample mean and a population mean, as far as I can tell. Your question appears to refer to the difference between two sample means (perhaps between the mean of a group of experimental subjects and a group of controls). –  whuber Jun 13 '11 at 14:55
The example I am thinking of is where you are looking a difference between a sample and population mean. In this case what exactly does the CI between sample and pop mean, mean. We have used the sample mean to estimate the pop standard deviation and thus from that we are estimating the CI around the mean estimate. The difference of means isn't the difference between the pop mean we have provided and the sample mean. So what is it? –  Anne Jun 13 '11 at 15:02
@Anne Is the "population mean" the hypothetical, unknown mean of the population being sampled or is it the measured mean of another population that has been exhaustively sampled? Also, in what sense did you use the "sample mean" to estimate the population standard deviation? Is that perhaps a typo? –  whuber Jun 13 '11 at 15:06
@whuber thanks. Your line "The CIs computed for 95% of all samples (that is, 95% of all possible replications) will cover that true difference." is clearer to me than "95 times out of 100, our sample mean difference will be between between these bounds" and your explanation makes logical sense. –  Anne Jun 17 '11 at 16:10

From a pedantic technical viewpoint, I personally don't think there is a "clear wording" of the interpretation of confidence intervals.

I would interpret a confidence interval as: there is a 95% probability that the 95% confidence interval covers the true mean difference

An interpretation of this is that if we were to repeat the whole experiment $N$ times, under the same conditions, then we would have $N$ different confidence intervals. The confidence level is the proportion of these intervals which contain the true mean difference.

My own personal quibble with the logic of such reasoning is that this explanation of confidence intervals requires us to ignore the other $N-1$ samples when calculating our confidence interval. For instance if you had a sample size of 100, would you then go and calculate 100 "1-sample" 95% confidence intervals?

But note that this is all in the philosophy. Confidence intervals are best left vague in the explanation I think. They give good results when used properly.

-
Thanks for your response. –  Anne Jun 13 '11 at 5:19
Starting a new sentence after "N different confidence intervals." doesn't flow well with "you can further interpret this as saying...". I suggest modifying the third paragraph. –  Theta30 Jun 15 '11 at 20:11
Your third paragraph is much better than the second. Conditional on the observed data, the confidence interval either contains the true parameter value or it doesn't. –  cardinal Jun 16 '11 at 18:10
@probabilityislogic: Since this answer has been accepted, please consider editing your second paragraph. Also, can you please clarify what you mean in your second to last paragraph? As it reads, I'm not quite sure what argument you are making. –  cardinal Jun 17 '11 at 23:03
There is a whole theory, largely parallel to optimal decision theory, on uniformly most accurate confidence sets. Maybe that is the piece of the puzzle missing for you. (?) –  cardinal Jun 18 '11 at 2:51

The rough answer to the question is that a 95% confidence interval allows you to be 95% confident that the true parameter value lies within the interval. However, that rough answer is both incomplete and inaccurate.

The incompleteness lies in the fact that it is not clear that "95% confident" means anything concrete, or if it does, then that concrete meaning would not be universally agreed upon by even a small sample of statisticians. The meaning of confidence depends on what method was used to obtain the interval and on what model of inference is being used (which I hope will become clearer below).

The inaccuracy lies in the fact that many confidence intervals are not designed to tell you anything about the location of the true parameter value for the particular experimental case that yielded the confidence interval! That will be surprising to many, but it follows directly from the Neyman-Pearson philosophy that is clearly stated in this quote from their 1933 paper "On the Problem of the Most Efficient Tests of Statistical Hypotheses":

We are inclined to think that as far as a particular hypothesis is concerned, no test based upon the theory of probability can by itself provide any valuable evidence of the truth or falsehood of that hypothesis.

But we may look at the purpose of tests from another view-point. Without hoping to know whether each separate hypothesis is true or false, we may search for rules to govern our behaviour with regard to them, in following which we insure that, in the long run of experience, we shall not be too often wrong.

Intervals that are based on the 'inversion' of N-P hypothesis tests will therefore inherit from that test the nature of having known long-run error properties without allowing inference about the properties of the experiment that yielded them! My understanding is that this protects against inductive inference, which Neyman apparently considered to be an abomination.

Neyman explicitly lays claim to the term ‘confidence interval’ and to the origin of the theory of confidence intervals in his 1941 Biometrika paper “Fiducial argument and the theory of confidence intervals”. In a sense, then, anything that is properly a confidence interval plays by his rules and so the meaning of an individual interval can only be expressed in terms of the long run rate at which intervals calculated by that method contain (cover) the relevant true parameter value.

We now need to fork the discussion. One strand follows the notion of ‘coverage’, and the other follows non-Neymanian intervals that are like confidence intervals. I will defer the former so that I can complete this post before it becomes too long.

There are many different approaches that yield intervals that could be called non-Neymanian confidence intervals. The first of these is Fisher’s fiducial intervals. (The word ‘fiducial’ may scare many and elicit derisive smirks from others, but I will leave that aside...) For some types of data (e.g. normal with unknown population variance) the intervals calculated by Fisher’s method are numerically identical to the intervals that would be calculated by Neyman’s method. However, they invite interpretations that are diametrically opposed. Neymanian intervals reflect only long run coverage properties of the method, whereas Fisher’s intervals are intended to support inductive inference concerning the true parameter values for the particular experiment that was performed.

The fact that one set of interval bounds can come from methods based on either of two philosophically distinct paradigms leads to a really confusing situation--the results can be interpreted in two contradictory ways. From the fiducial argument there is a 95% likelihood that a particular 95% fiducial interval will contain the true parameter value. From Neyman’s method we know only that 95% of intervals calculated in that manner will contain the true parameter value, and have to say confusing things about the probability of the interval containing the true parameter value being unknown but either 1 or 0.

To a large extent, Neyman’s approach has held sway over Fisher’s. That is most unfortunate, in my opinion, because it does not lead to a natural interpretation of the intervals. (Re-read the quote above from Neyman and Pearson and see if it matches your natural interpretation of experimental results. Most likely it does not.)

If an interval can be correctly interpreted in terms of global error rates but also correctly in local inferential terms, I don’t see a good reason to bar interval users from the more natural interpretation afforded by the latter. Thus my suggestion is that the proper interpretation of a confidence interval is BOTH of the following:

• Neymanian: This 95% interval was constructed by a method that yields intervals that cover the true parameter value on 95% of occasions in the long run (...of our statistical experience).

• Fisherian: This 95% interval has a 95% probability of covering the true parameter value.

(Bayesian and likelihood methods will also yield intervals with desirable frequentist properties. Such intervals invite slightly different interpretations that will both probably feel more natural than the Neymanian.)

-
@Micheal - the place where they will differ is that a fudicial interval must be based on a sufficient statistic, and condition on all ancillary quantities. Neymans confidence interval does not require this property, and so are subject to the "95% confidence interval" having varying coverage for particular sub-classes of samples. –  probabilityislogic Jun 23 '11 at 7:25
@probability - Can you expand on that? Do you mean that there are circumstances where a 95% Neymanian confidence interval is a confidence interval but it is not a 95% interval? What would those circumstances be? Would the Fisherian interval have the same bounds in those circumstances? –  Michael Lew Jun 24 '11 at 3:57
You can show cases where you can tell from the sample, that a "95%" confidence interval doesn't contain the true value. example 5 and example 6 in Jaynes' paper gives two cases where not using sufficient statistics in CIs will give the long run coverage, but the coverage will vary over certain classes of samples. It is analogous to having two variables with the same average (long run coverage) but different variance (coverage in specific case) –  probabilityislogic Jun 24 '11 at 11:30

The meaning of a confidence interval is: if you were to repeat your experiment in the exact same way (i.e.: the same number of observations, drawing from the same population, etc.), and if your assumptions are correct, and you would calculate that interval again in each repetition, then this confidence interval would contain the true prevalence in 95% of the repetitions (on average).

So, you could say you are 95% certain (if your assumptions are correct etc.) that you have now constructed an interval that contains the true prevalence.

This is typically stated as: with 95% confidence, between 4.5 and 8.3% of children of mothers who smoked throughout pregnancy become obese.

Note that this is typically not interesting in itself: you probably want to compare this to prevalence in children of mothers who didn't smoke (odds ratio, relative risk, etc.)

-
(This reply, which arrived here after a merger of two threads, is responding to a duplicate question framed in terms of a CI of a proportion.) –  whuber Oct 7 '11 at 14:37

If the true mean difference is outside of this interval, then there is only a 5% chance that the mean difference from our experiment would be so far away from the true mean difference.

-
What do you mean by "this far away"? Is this the upper bound of the CI that is far away or the observed mean? –  probabilityislogic Jun 16 '11 at 9:44
The distance between the true mean and the observed mean is what I mean by "this far away". I'm going to change it to "so far away"; I think that is a little more clear. –  Thomas Levine Jun 16 '11 at 19:50

My Interpretation: If you conduct the experiment N times ( where N tends to infinity) then out of these large number of experiments 95% of the experiments will have confidence intervals which lie within these 95% limits. More clearly, lets say those limits are "a" and "b" then 95 out of 100 times your sample mean difference will lie between "a" and "b".I assume that you understand that different experiment can have different samples to cover out of the whole population.

-
@ Ayush. thanks. That is helpful. Sorry I don't quite follow your final sentence. –  Anne Jun 13 '11 at 5:21
@anne -- Ok. What I mean is if you want to test the mean between two samples and lets say each sample has 1000 people, you can define infinite samples out of it ( of lets say 40 people from each)..I had wrote this to tell why do the different experiments differ from each other..The experiments where we are observing confidence interval. –  ayush biyani Jun 13 '11 at 5:33
@ayush - this is not the correct interpretation in your second last sentence. Or at least you should add subscripts to "a" and "b", which makes it clear that it is these quantities which are varying over the 100 times. Your current notation makes it seem like "a" and "b" are fixed quantities. –  probabilityislogic Jun 13 '11 at 6:06
@probabilityislogic -- agree..the subscripts are necessary. –  ayush biyani Jun 13 '11 at 6:33
@Ayush (-1) The characterization that currently appears in your reply can be interpreted in several ways, most of which (therefore) are incorrect. For example, confidence intervals $[a,b]$ are usually constructed so as to contain the "sample mean difference," implying that this difference will lie between the limits 100% of the time no matter what. –  whuber Jun 13 '11 at 13:28