2
$\begingroup$

Suppose that in R, we did a t-test with some sample data:

> t.test(1:10, y = c(7:20))

    Welch Two Sample t-test

data:  1:10 and c(7:20)
t = -5.4349, df = 21.982, p-value = 1.855e-05
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -11.052802  -4.947198
sample estimates:
mean of x mean of y 
      5.5      13.5 

The above yields a $95\%$ confidence interval. My general understanding of a level $\alpha$ confidence interval is that if we repeated an experiment an infinitely large number of times, $(1-\alpha)\%$ of those constructed intervals would contain the true population mean.

However, the above appears to be a "point estimate" of the confidence interval. What is the correct way to interpret the range, $(-11.052802, -4.947198)$ above? Thanks!

$\endgroup$
  • $\begingroup$ How could a point estimate ever "contain the mean"?? You need an interval to have a non-zero chance of containing the population parameter with a continuous distribution like this. $\endgroup$ – Glen_b Mar 20 '18 at 6:21
  • $\begingroup$ I am confused how exactly the reported interval above can be interpreted, do you have any thoughts? $\endgroup$ – user321627 Mar 20 '18 at 6:45
  • $\begingroup$ The correct interpretation is already given in the question. If that's all you're really asking there's nothing left to say. if you are asking anything other than that, then I don't understand the current phrasing of the question. What you mean needs to be clarified before it should be answered. $\endgroup$ – Glen_b Mar 20 '18 at 7:39
1
$\begingroup$

The Neyman-Pearson theory of Null Hypothesis Significance Testing has the goal of providing you with a decision rule which, when the null hypothesis is true, allows you to make the correct choice $95\%$ of times. However, it cannot tell you the confidence interval you computed based on your random sample drawn from the population (i.e., your realization $i=(−11.052802,−4.947198)$ of the random interval $I$) contains the true parameter or not: you just don't know.

Then why do you reject the null hypothesis $H_0$ in this specific case? You do it because you know that, if $H_0$ were true and you repeated this experiment a large number of times, then, following the Neyman-Pearson decision rule, which is:

  • accept $H_0$ if $i$ contains 0
  • reject $H_0$ if $i$ doesn't contain 0 (as in your case)

you would be wrong only $5\%$ of times. Thus the decision rule is a guide to control your error rate in the long run.

This is very relevant in manufacturing for example, in process quality control. If the manufacturing process is in control, you are effectively sampling multiple times from the same population, thus you expect $5\%$ of your confidence intervals not to contain the parameter of interest. Thus a process in control would raise an alarm $5\%$ of times, which can sound weird (actually, the $\alpha$ level used in quality control is usually much less than $5\%$).

You acutely asked in a comment why not to compute multiple confidence intervals, instead. First of all, in real life, you often can't afford the luxury of performing repeated sampling, because of time, budget, etc. constraints. Secondly, even if you could, it wouldn't make sense to create many such intervals and try to "intersect" them: there's no principled way to do that. Instead, you can gather all your $m$ random samples of size $n$ together and build a confidence interval based on your aggregated sample $\mathbf{x}=(x_{11},\dots,x_{1n},\dots,x_{1m},\dots,x_{nm})$. Since the width of a confidence interval decreases with the sample size $N$ (usually, as $O(\frac{1}{\sqrt{N}})$), the resulting confidence interval will be your most accurate inference (but you still won't be able to know with certainty if it contains the true parameter or not).

$\endgroup$
3
$\begingroup$

Your "general understanding" is precisely correct. Each one of your infinitely many experiments will give rise to a separate confidence interval for the parameter of interest (here: the difference in means). And if all preconditions are fulfilled, then 95% of these intervals will in fact contain the true parameter.

And yes, in practice, we only ever see a single CI, which will either contain the true parameter or not.

$\endgroup$
  • $\begingroup$ Thanks, in the context of a single t-test computed on a set of data, is there anything we can really say about the reported confidence interval? For example, for the above interval, $(-11.052802, -4.947198)$, is the only thing we can really say is that this interval represents one draw from a distribution of intervals? $\endgroup$ – user321627 Mar 20 '18 at 7:09
  • $\begingroup$ Yes, exactly. You can bootstrap the CI and plot, say, 20 CIs as horizontal lines to get an idea about the inherent variability. $\endgroup$ – Stephan Kolassa Mar 20 '18 at 7:11
  • $\begingroup$ Thanks, one more thing that confuses me is that hypothesis testing usually occurs here in the form of "since $0$ is not contained in the interval $(-11.052802, -4.947198)$, we reject the null that the mean differences is zero". Since this interval was a single draw, why is this reasoning valid? Shouldn't we instead be computing many such intervals and seeing if $0$ is inside? $\endgroup$ – user321627 Mar 20 '18 at 7:13
  • $\begingroup$ Wording suggestions: For statistic of interest read parameter of interest. For true parameter read true value of parameter. $\endgroup$ – Nick Cox Mar 20 '18 at 7:16
  • $\begingroup$ Yes, this is one of the problems with NHST. Note that this is just a reformulation of calculating a single p value, checking whether this is less than 0.05 and then declaring something as "statistically significant". What this disregards is that p values (and CI endpoints) are random variables in themselves. What NHST does is to hope that we are right 95% of the time. (Assuming preconditions are fulfilled.) $\endgroup$ – Stephan Kolassa Mar 20 '18 at 7:22
0
$\begingroup$

The confidence interval (CI) you have calculated is your measure of precision in estimating the population parameter. A wide CI would indicate low precision, and a narrow CI would indicate high precision. Interpretation of precision would depend on the nature of the scale or whatever measure you are using.

The 95% CI you have calculated from your sample data is one in an infinite sequence of potential confidence intervals. The 95% CI will differ each time a replication of your study is performed. Most likely the interval will capture the population parameter you want to estimate (where the point estimate is our best estimate of this parameter), but sometimes by chance the 95% CI will not contain the population parameter. However, in the long run, 95% of the CIs you calculate will contain the population parameter you are trying to estimate. This is why it is important to envision your study as just one observation among many, which at a later time can be meta-analysed to provide an estimate of the true population parameter.

The 95% CI is also a prediction interval. That is, on average, a 95% CI has about an 83% chance of capturing the mean of a near replication of your study. See Cumming, G., & Maillardet, R. (2006). Confidence intervals and replication: where will the next mean fall? Psychological Methods, 11(3), 217-227. doi:10.1037/1082-989x.11.3.217

If you are interested in confidence intervals I highly recommend the excellent introductory textbook: Cumming, G., & Calin-Jageman, R. (2016). Introduction to the New Statistics : Estimation, Open Science, and Beyond. London, Taylor and Francis.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.