Confidence band for simple linear regression - why the curve? [duplicate]

Question

I am really struggling to understand why confidence bands for regression lines have a curve to them. A few example plots showing curved CIs, taken from this post: Shape of confidence interval for predicted values in linear regression

I have read a bit and understand that the standard error for the sampling distribution of $\hat y$ at a given point ($\hat y^*$) is as follows:

$$ MSE \sqrt{\frac{1}{n} + \frac{(x^*-\bar x)^2}{\sum_{i=1}^n (x_i-\bar x)^2}} $$

Because the above equation has $(x^* - \bar x)^2$ in the numerator, the further we are from $\bar x$, the bigger the standard error will be, hence the curve being 'thinnest' when $x = \bar x$, and fatter as $x$ increases/decreases.

But why does that make sense?

My thinking is this: if one of the assumptions for a linear regression is homogeneity of variance for $Y$ across values of $X$, and the residuals from our regression are also equally distributed across values of $X$, why would the variance of our sampling distributions of $\hat y$ not be the same at each value of $X$?

Clearly I am missing something!

Answer 1

Computing the sample variance of the estimate $\hat{y}$

The estimate for the mean $y$ (as a function of $x$) has the following function in terms of the predictions for coefficients $\alpha$ and $\beta$

$$\hat {y} = \hat{\alpha} + \hat{\beta} x$$

The standard error of $\hat{y}$ can be computed with the formula for the standard deviation or variance

$$Var(\hat\alpha + \hat\beta x) = Var(\hat\alpha) + x^2 Var(\hat\beta) - 2x Cov(\hat\alpha,\hat\beta)$$

So this is a quadratic function that has a minimum at $x = \frac{Cov(\hat\alpha,\hat\beta)}{Var(\hat\beta)} = \bar{x_i}$ and this creates that funnel shape with the minimum at the mean of the datapoints $x_i$.

Intuitive

Let's try out several fits. We use the following data

$X_i$ is normal distributed. $Y_i$ is $0.8$ times $X_i$ with some added noise.

$$\begin{array}{} X_i &\sim& N(0,1) \\ \epsilon_i &\sim& N(0,1) \\ Y_i &=& 0.8 X_i + 0.6 \epsilon_i \end{array}$$

Result of $25$ simulations with each $15$ data points

When we combine all those different lines in a single plot then we get:

So here we might see intuitively why the confidence band becomes 'fatter' at the ends. The confidence is due to errors in the height of the line (parameter $\alpha$) and the slope of the line (parameter $\beta$). It is this latter one that makes the error larger towards the ends.

---

^{$\frac{Cov(\hat\alpha,\hat\beta)}{Var(\hat\beta)} = \bar{x} $ follows from the covariance matrix for the $\alpha$ and $\beta$ which is $\sigma (X^TX)^{-1}$. Which you could work out further by filling in all the terms... But you could also argue that the minimum should be at $\bar{x}$ by transform the data matrix $X$ such that the the column vectors are perpendicular and the estimates $\hat\alpha$ and $\hat\beta$ have zero covariance.}

Answer 2

As you get farther from $\bar x,\bar y$ uncertainty increases. There's fewer and fewer observations when you reach out to distant regions of the domain of your function.

The main source of uncertainty is the one about the slope of the line. Take a look at the drawing here. With the given sample of observations you can say that the best fit line should be somewhere between these two grey lines. The uncertainty around $(\bar x,\bar y)$ is the smallest, but once you step away from where the observations are located, uncertainty increases.

Here's how we can intuitively "derive" the asymptotic confidence interval, i.e. where $x^*$ is very far away from your observations. The confidence given by model MSE will proportionally expand as $|x^*-\bar x|\to\infty$. Think of it as approximate equality of ratios $\frac{MSE}{\sqrt n\sigma_x}\approx \frac{CI(x^*)}{|x^*-\bar x|}$. That's the asymptotic of your formula: $$\lim_{x^*\to\infty} MSE \sqrt{\frac{1}{n} + \frac{(x^*-\bar x)^2}{\sum_{i=1}^n (x_i-\bar x)^2}} =\frac{MSE}{\sqrt n\sigma_x}|x^*-\bar x|$$

Answer 3

There is 2 uncertainties here. As you mentioned there is the uncertainty with the slope thus the spreading curve at ends, but there is also an uncertainty at the mean. Yes, the curve is thinnest at the mean but it is not zero. Thus the uncertainty of the slope passing through the mean's distribution causes the estimate to be non linear and generates the above examples.