31
$\begingroup$

I am trying to understand the origin of the curved shaped of confidence bands associated with an OLS linear regression and how it relates to the confidence intervals of the regression parameters (slope and intercept), for example (using R):

require(visreg)
fit <- lm(Ozone ~ Solar.R,data=airquality)
visreg(fit)

enter image description here

It appears that the band is related to the limits of the lines calculated with the 2.5% intercept, and the 97.5% slope, as well as with the 97.5% intercept, and the 2.5% slope (although not quite):

xnew <- seq(0,400)
int <- confint(fit)
lines(xnew, (int[1,2]+int[2,1]*xnew))
lines(xnew, (int[1,1]+int[2,2]*xnew))

enter image description here

What I don't understand are two things:

  1. What about the combination of 2.5% slope & 2.5% intercept as well as 97.5% slope and 97.5% intercept? These give lines that are clearly outside the band plotted above. Maybe I don't understand the meaning of a confidence interval, but if in 95% of the cases my estimates are within the confidence interval, these seem like a possible outcome?
  2. What determines the minimum distance between the upper and lower limit (i.e. close to the point where the two lines added above intercept)?

I guess both questions arise because I don't know/understand how these bands are actually calculated.

How can I calculate the upper and lower limits using the confidence intervals of the regression parameters (without relying on predict() or a similar function, i.e. by hand)? I tried to decipher the predict.lm function in R, but the coding is beyond me. I'd appreciate any pointers towards relevant literature or explanations suitable for stats beginners.

Thanks.

$\endgroup$
16
$\begingroup$

The standard error of the regression line at point $X$ (i.e. $s_{\hat{Y}_{X}}$) is hand calculated (Yech!) using:

$s_{\hat{Y}_{X}} = s_{Y|X}\sqrt{\frac{1}{n}+\frac{\left(X-\bar{X}\right)^{2}}{\sum_{i=1}^{n}{\left(X_{i}-\bar{X}\right)^{2}}}}$,

where the standard error of the estimate (i.e. $s_{Y|X}$) is hand calculated (Double yech!) using:

$s_{Y|X} = \sqrt{\frac{\sum_{i=1}^{n}{\left(Y_{i}-\hat{Y}\right)^{2}}}{n-2}}$.

The confidence band about the regression line is then obtained as $\hat{Y} \pm t_{\nu=n-2, \alpha/2}s_{\hat{Y}}$.

Bear in mind that the confidence band about the regression line is not the same beast as the prediction band about the regression line (there is more uncertainty in predicting $Y$ given a value of $X$ than in estimating the regression line). And, as you are struggling to understand, the confidence intervals about the intercept and slope are yet other quantities.

Further, you do not understand confidence intervals: "if in 95% of the cases my estimates are within the confidence interval, these seem like a possible outcome?" Confidence intervals do not 'contain 95% of the estimates,' rather for each separate sample (produced by the same study design), 95% of the (separately calculated for each sample) 95% confidence intervals would contain the 'true population parameter' (i.e. the true slope, the true intercept, etc.) that $\hat{\beta}$ and $\hat{\alpha}$ are estimating.

$\endgroup$
  • $\begingroup$ Is there a textbook that explains where these formulas come from? $\endgroup$ – Michael Goerz Jan 2 '18 at 3:33
  • 1
    $\begingroup$ @MichaelGoerz Any introductory statistics, biostatistics, econometrics, etc. textbook that covers ordinary least squares linear regression should have. $\endgroup$ – Alexis Jan 2 '18 at 4:00
  • $\begingroup$ I have Wasserman - All of Statistics, James et al - An Introduction to Statistical Learning, and Hastie et al. - The Elements of Statistical Learning. I haven't been able to find the equations for the linear regression confidence bands in any of them. Do you have a chapter/eq number for any of these, or some other widely available book? $\endgroup$ – Michael Goerz Jan 2 '18 at 4:10
  • 1
    $\begingroup$ None of the books you mention are the kinds of books Alexis is discussing. Fox's book on Applied regression has it if I recall correctly. $\endgroup$ – Glen_b Jan 26 '18 at 13:53
  • $\begingroup$ @MichaelGoerz As do both Pagano, M. and Gauvreau, K. (2000). Principles of Biostatistics. Duxbury Press, Pacific Grove, CA, 2nd edition and Glantz, S. A. (2011). primer of biostatistics. McGraw-Hill Medical, New York, NY, 7th edition even though they are not regression-specific texts. $\endgroup$ – Alexis Jan 26 '18 at 19:17
15
$\begingroup$

Nice question. It's important to understand these concepts and they're not straightforward.

The 95% confidence bands you see around the regression line are generated by the 95% confidence intervals that the true value for $\bar y$ falls within that range for each individual x. So take a vertical slice, say at x = 50. The regression tells us that $\bar y$ at x = 50 is approximately 25. The confidence interval calculation tells us that we're 95% confident that the true value for $\bar y$ at that point is within the gray area of the graph (so approximately 15 and 35 for the graph above).

When we combine all of the confidence intervals, for every possible x, it gives us the gray bands that you see in the output.

What this functionally means is that we're 95% confident that the true regression line lies somewhere in that gray zone.

Because the confidence bands are calculated using the 95% confidence intervals for each individual point, it's very closely related to the 95% CI for the intercept. In fact, at x = 0 the edges of the gray zone will coincide exactly with the 95% CI for the intercept, because that's how we've generated the confidence bands. That's why the lines you've added above hit the edge of the gray band towards the left.

However, the slope is a little different. It does contribute to the limits, as you've seen above, but the slope and intercept are not separable in a linear regression. So, you can't really say "well what if the intercept was at the minimum of the CI range and the slope was also at the minimum?" This line would generate points that are well outside our 95% CI's for a lot of x's. This means that we're 95% confident that is not our true regression line.

To address your second question, out regression calculations are more precise for x values in the middle of our sample. In fact, the narrowest 95% CI will show up at $\bar x$. This is because, as you can see in the formula in Alexis's answer, $s_{{\hat y}_x}$, $(x - \bar x)$ is in the numerator of a fraction. When $x = \bar x$ this value is zero, so the standard error is smaller.

There's a decent powerpoint here that can help you visualize some of these things: http://www.stat.duke.edu/~tjl13/s101/slides/unit6lec3H.pdf

$\endgroup$
  • 2
    $\begingroup$ I think I fixed it - replaced the yhats with ybars. Is that more correct? I always screw up yhat. $\endgroup$ – Duncan Jun 5 '14 at 20:08
  • $\begingroup$ Ta. One thing that remains unclear to me is how to make the following two statements consistent: "What this functionally means is that we're 95% confident that the true regression line lies somewhere in that gray zone." vs "[...] the confidence intervals about the intercept and slope are yet other quantities." If the first statement is correct, there must be some (mathematical?) relation between the CIs of intercept and slope and the band plotted above? I guess this relates to a part of my question: How can I calculate (if possible) the band above using the CI's of slope and intercept? $\endgroup$ – David Jun 6 '14 at 8:21
  • 1
    $\begingroup$ You can't calculate the bands using just the CIs of slop and intercept because the bands are generated by calculating the CIs at each x. As the bands get tighter the closer you get to $\bar x$ they'll deviate from the lines that the extreme values of CIs for slope and intercept generate. $\endgroup$ – Duncan Jun 8 '14 at 12:17
  • $\begingroup$ Nice understandable post and a nice link! +1 $\endgroup$ – theforestecologist May 25 '18 at 3:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.