Understanding shape and calculation of confidence bands in linear regression

Question

I am trying to understand the origin of the curved shaped of confidence bands associated with an OLS linear regression and how it relates to the confidence intervals of the regression parameters (slope and intercept), for example (using R):

require(visreg)
fit <- lm(Ozone ~ Solar.R,data=airquality)
visreg(fit)

It appears that the band is related to the limits of the lines calculated with the 2.5% intercept, and the 97.5% slope, as well as with the 97.5% intercept, and the 2.5% slope (although not quite):

xnew <- seq(0,400)
int <- confint(fit)
lines(xnew, (int[1,2]+int[2,1]*xnew))
lines(xnew, (int[1,1]+int[2,2]*xnew))

What I don't understand are two things:

1. What about the combination of 2.5% slope & 2.5% intercept as well as 97.5% slope and 97.5% intercept? These give lines that are clearly outside the band plotted above. Maybe I don't understand the meaning of a confidence interval, but if in 95% of the cases my estimates are within the confidence interval, these seem like a possible outcome?
2. What determines the minimum distance between the upper and lower limit (i.e. close to the point where the two lines added above intercept)?

I guess both questions arise because I don't know/understand how these bands are actually calculated.

How can I calculate the upper and lower limits using the confidence intervals of the regression parameters (without relying on predict() or a similar function, i.e. by hand)? I tried to decipher the predict.lm function in R, but the coding is beyond me. I'd appreciate any pointers towards relevant literature or explanations suitable for stats beginners.

Thanks.

Answer 1

The standard error of the regression line at point $X$ (i.e. $s_{\widehat{Y}_{X}}$) is hand calculated (Yech!) using:

$s_{\widehat{Y}_{X}} = s_{Y|X}\sqrt{\frac{1}{n}+\frac{\left(X-\overline{X}\right)^{2}}{\sum_{i=1}^{n}{\left(X_{i}-\overline{X}\right)^{2}}}}$,

where the standard error of the estimate (i.e. $s_{Y|X}$) is hand calculated (Double yech!) using:

$s_{Y|X} = \sqrt{\frac{\sum_{i=1}^{n}{\left(Y_{i}-\widehat{Y}\right)^{2}}}{n-2}}$.

The confidence band about the regression line is then obtained as $\widehat{Y} \pm t_{\nu=n-2, \alpha/2}s_{\widehat{Y}}$.

Bear in mind that the confidence band about the regression line is not the same beast as the prediction band about the regression line (there is more uncertainty in predicting $Y$ given a value of $X$ than in estimating the regression line). And, as you are struggling to understand, the confidence intervals about the intercept and slope are yet other quantities.

Further, you do not understand confidence intervals: "if in 95% of the cases my estimates are within the confidence interval, these seem like a possible outcome?" Confidence intervals do not 'contain 95% of the estimates,' rather for each separate sample (produced by the same study design), 95% of the (separately calculated for each sample) 95% confidence intervals would contain the 'true population parameter' (i.e. the true slope, the true intercept, etc.) that $\widehat{\beta}$ and $\widehat{\alpha}$ are estimating.

Answer 2

Nice question. It's important to understand these concepts and they're not straightforward.

The 95% confidence bands you see around the regression line are generated by the 95% confidence intervals that the true value for $\bar y$ falls within that range for each individual x. So take a vertical slice, say at x = 50. The regression tells us that $\bar y$ at x = 50 is approximately 25. The confidence interval calculation tells us that we're 95% confident that the true value for $\bar y$ at that point is within the gray area of the graph (so approximately 15 and 35 for the graph above).

When we combine all of the confidence intervals, for every possible x, it gives us the gray bands that you see in the output.

What this functionally means is that we're 95% confident that the true regression line lies somewhere in that gray zone.

Because the confidence bands are calculated using the 95% confidence intervals for each individual point, it's very closely related to the 95% CI for the intercept. In fact, at x = 0 the edges of the gray zone will coincide exactly with the 95% CI for the intercept, because that's how we've generated the confidence bands. That's why the lines you've added above hit the edge of the gray band towards the left.

However, the slope is a little different. It does contribute to the limits, as you've seen above, but the slope and intercept are not separable in a linear regression. So, you can't really say "well what if the intercept was at the minimum of the CI range and the slope was also at the minimum?" This line would generate points that are well outside our 95% CI's for a lot of x's. This means that we're 95% confident that is not our true regression line.

To address your second question, out regression calculations are more precise for x values in the middle of our sample. In fact, the narrowest 95% CI will show up at $\bar x$. This is because, as you can see in the formula in Alexis's answer, $s_{{\hat y}_x}$, $(x - \bar x)$ is in the numerator of a fraction. When $x = \bar x$ this value is zero, so the standard error is smaller.

There's a decent powerpoint here that can help you visualize some of these things: http://www.stat.duke.edu/~tjl13/s101/slides/unit6lec3H.pdf