I couldn't see a question addressing this. I think I'm just off the mark somewhere with my thinking. Here is a simple model:

mod <- lm(hp ~ factor(cyl), data=mtcars)

             Estimate Std. Error t value Pr(>|t|)    
(Intercept)     82.64      11.43   7.228 5.86e-08 ***
factor(cyl)6    39.65      18.33   2.163   0.0389 *  
factor(cyl)8   126.58      15.28   8.285 3.92e-09 ***

This tells me the hp is significantly higher in the 6 cylinders than the baseline (4 cylinders).

Now when I run the following, I'm confused by the results.

predict(mod, data.frame(cyl=as.factor(c(4,6,8))), interval="c")

        fit       lwr      upr
1  82.63636  59.25361 106.0191
2 122.28571  92.97388 151.5975
3 209.21429 188.48769 229.9409

The fit for the first is the same as the intercept, makes sense. But the upr for the first is higher than the lwr for the second, as in, the intervals overlap. When I see confidence intervals that overlap I think, 'not significantly different'.

I don't pretend to know EVERYTHING about the ins and outs of modeling, but I thought I had a pretty solid understanding of the basics, yet I'm clearly missing something as to why these intervals overlap...

Edit: Re the confusion around the terminology of 'predicting', I merely use the word predict as that is (explicit) in the function. Am aware I am using it in the 'expansion for confidence of estimation' framework.

Note, an idea of the data:

> aggregate(hp ~ cyl, data=mtcars, mean)
  cyl        hp
1   4  82.63636
2   6 122.28571
3   8 209.21429
> aggregate(hp ~ cyl, data=mtcars, sd)
  cyl       hp
1   4 20.93453
2   6 24.26049
3   8 50.97689
> aggregate(hp ~ cyl, data=mtcars, length)
  cyl hp
1   4 11
2   6  7
3   8 14


Based on first answer and a conversation in chat. I get that the p-value 0.039 is for whether the coefficient for 6cyl is different to 0. That would say, yes, the hp is 6cyl cars is different to that of 4cyl cars. Correct interpretation right? And then you say, "okay, so what is your estimate of the hp of a 4cyl car and the hp of a 6cyl car, please include your confidence", and then you get the second set of results, 60-106 for 4cyl and 92-151 for 6cyl. But they overlap, so how can you say they're different?

So on the one hand we say the estimate of the coefficient for 6cyl cars is sufficiently far away from that of 4 cyl cars to call them different, and on the other hand we calculate a predicted value for 4cyl and 6cyl cars and they overlap therefore are not different?

  • $\begingroup$ $0$ is not the mean for the 4cyl. The intercept is the mean for the 4cyl engine. Also, you should perhaps write p-value instead of 'PV' in your question just for clarity. Also, as I said in my answer, you cannot look at the individual confidence intervals to conclude anything as to whether the two parameters are different. $\endgroup$ – varty Nov 28 '11 at 2:18
  • $\begingroup$ Updated. I'm not looking at the confidence intervals for parameters (beta coefficients), I'm looking at confidence intervals for predictions. $\endgroup$ – nzcoops Nov 28 '11 at 2:24
  • $\begingroup$ The logic stays the same even if you are looking at predictions. You are looking at the confidence intervals for the individual predictions to conclude something about the true difference between the performance of the 4cl vs 6cl engine. So, the idea is the same as my answer. $\endgroup$ – varty Nov 28 '11 at 2:30
  • $\begingroup$ Just a terminology quibble...these are not prediction intervals. These involve estimation uncertainty only (interval="c"), and are not adding in fundamental uncertainty. $\endgroup$ – Ari B. Friedman Nov 28 '11 at 2:37

Your use of the confidence intervals associated with the individual parameters is incorrect.

To check if two parameters are significantly different from each other you cannot look if the confidence intervals for each parameter overlap. You have to check if the confidence interval associated with the difference between the two parameters contains $0$ or not.

More formally, you are checking if your hypothesis that $\beta_{4cl} = \beta_{6cl}$. The appropriate test statistic for the null hypothesis is: $t = \hat{\beta}_{4cl} - \hat{\beta}_{6cl}$ and thus the confidence interval is dependent on the standard error associated with this test statistic.

  • $\begingroup$ Thanks. Please see my addition to the question and see if you can clarify this a bit? I do follow what you're saying about the null of the test. It just seems to me at this stage (hopefully before some penny drops) that the two 'ways to look at the difference' are telling different stories. $\endgroup$ – nzcoops Nov 28 '11 at 2:15

I don't really see the conflict here. Your $\beta_{6cyl}$ is tested against 0. Your prediction interval is giving you joint confidence ($\beta_0+\beta_6cyl$). So since the interval (93,151) doesn't contain 83, they're saying the same thing.

I think you're less confused than you think you are. The only issue here is that it's not about whether one CI overlaps the other, but whether the one point estimate is contained within the other's confidence interval.

  • $\begingroup$ Hmm. What does it mean for one point estimate to be contained within another? :) $\endgroup$ – cardinal Nov 28 '11 at 3:02
  • 1
    $\begingroup$ Now you should consider whether your statement is true or not. :) $\endgroup$ – cardinal Nov 28 '11 at 3:22
  • $\begingroup$ Ha. Well, feel free to correct me if I'm way off base :-) $\endgroup$ – Ari B. Friedman Nov 28 '11 at 3:25
  • $\begingroup$ Consider a simplified version of the OP's question: Let $X_1,\ldots,X_n$ be iid $\mathcal N(\mu,1)$ and $Y_1,\ldots,Y_n$ be iid $\mathcal N(\theta,1)$ with the $\{X_i\}$ and $\{Y_i\}$ mutually independent. What can you say about the confidence intervals for $\mu$, $\theta$ and $\mu-\theta$, respectively? $\endgroup$ – cardinal Nov 28 '11 at 3:28
  • $\begingroup$ Thanks gsk3. Maybe it's a mindset issue for me, will keep thinking on this, am keen for others thoughts. $\endgroup$ – nzcoops Nov 28 '11 at 3:31

Okay. So the penny finally dropped when talking to a colleague about it. For the 'predicted' (read expanded) confidence intervals to NOT overlap, there would essentially be 4 SE's worth of difference, since each CI is the estimate +- 2SE. And of course 4SE would be fare more significant that 0.04.

Of course this was just a simplified example of something more complex I was working on and I was looking for the best way to explain this to non statisticians, who, like I naively feel into that trap, see overlying confidence intervals and think they mustn't be different.

Further reading available: http://www.sciencedirect.com/science?_ob=MiamiImageURL&_cid=272610&_user=554529&_pii=S0741521402000307&_check=y&_origin=&_coverDate=31-Jul-2002&view=c&wchp=dGLzVBA-zSkWz&md5=f47e0bb21d5a572a7e0c3299cd0796b7/1-s2.0-S0741521402000307-main.pdf


Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.