6
$\begingroup$

I am really struggling to understand why confidence bands for regression lines have a curve to them. A few example plots showing curved CIs, taken from this post: Shape of confidence interval for predicted values in linear regression enter image description here

I have read a bit and understand that the standard error for the sampling distribution of $\hat y$ at a given point ($\hat y^*$) is as follows:

$$ MSE \sqrt{\frac{1}{n} + \frac{(x^*-\bar x)^2}{\sum_{i=1}^n (x_i-\bar x)^2}} $$

Because the above equation has $(x^* - \bar x)^2$ in the numerator, the further we are from $\bar x$, the bigger the standard error will be, hence the curve being 'thinnest' when $x = \bar x$, and fatter as $x$ increases/decreases.

But why does that make sense?

My thinking is this: if one of the assumptions for a linear regression is homogeneity of variance for $Y$ across values of $X$, and the residuals from our regression are also equally distributed across values of $X$, why would the variance of our sampling distributions of $\hat y$ not be the same at each value of $X$?

Clearly I am missing something!

$\endgroup$
9
  • 5
    $\begingroup$ Point of clarification: the variance of $y|x$ is constant in this model, and the curvature of the confidence bands does not reflect heteroscedasticity (non-constant of $y|x$). If you are looking for a purely intuitive idea: there is less nearby information informing estimates of $y$ that are far from the mean of $x$ (e.g., the estimate of $y$ for $\min (x)$ is determined by no values of $x$ below that point!). $\endgroup$
    – Alexis
    Nov 8 '21 at 1:13
  • 4
    $\begingroup$ Previously answered in some detail here: Shape of confidence interval for predicted values in linear regression $\endgroup$
    – Glen_b
    Nov 8 '21 at 3:21
  • 1
    $\begingroup$ If the variance of the sampling distribution of yhat were the same at each point, that means you are were only uncertain about the intercept, right? If you're uncertain about the intercept and the slope, the shape of needs to change. The homogeneity of variance of the residuals is still assumed, but that band is not about residuals. $\endgroup$
    – rep_ho
    Nov 8 '21 at 10:57
  • 3
    $\begingroup$ Does this answer your question? Shape of confidence interval for predicted values in linear regression $\endgroup$ Nov 8 '21 at 16:07
  • 1
    $\begingroup$ "But does this extend to mean that our predictions are less certain the further we move from $\bar{x}$?" That is exactly what the widening curvature means! Caveat: For predicted values of the regression line.. The prediction interval for a value of $y$ given a single $x$ value, the bands are much wider/much less certain (although they are still curved, and for exactly the same reason). $\endgroup$
    – Alexis
    Nov 9 '21 at 16:27
9
$\begingroup$

Computing the sample variance of the estimate $\hat{y}$

The estimate for the mean $y$ (as a function of $x$) has the following function in terms of the predictions for coefficients $\alpha$ and $\beta$

$$\hat {y} = \hat{\alpha} + \hat{\beta} x$$

The standard error of $\hat{y}$ can be computed with the formula for the standard deviation or variance

$$Var(\hat\alpha + \hat\beta x) = Var(\hat\alpha) + x^2 Var(\hat\beta) - 2x Cov(\hat\alpha,\hat\beta)$$

So this is a quadratic function that has a minimum at $x = \frac{Cov(\hat\alpha,\hat\beta)}{Var(\hat\beta)} = \bar{x_i}$ and this creates that funnel shape with the minimum at the mean of the datapoints $x_i$.

Intuitive

Let's try out several fits. We use the following data

$X_i$ is normal distributed. $Y_i$ is $0.8$ times $X_i$ with some added noise.

$$\begin{array}{} X_i &\sim& N(0,1) \\ \epsilon_i &\sim& N(0,1) \\ Y_i &=& 0.8 X_i + 0.6 \epsilon_i \end{array}$$

Result of $25$ simulations with each $15$ data points

25 trials

When we combine all those different lines in a single plot then we get:

25 lines

So here we might see intuitively why the confidence band becomes 'fatter' at the ends. The confidence is due to errors in the height of the line (parameter $\alpha$) and the slope of the line (parameter $\beta$). It is this latter one that makes the error larger towards the ends.


$\frac{Cov(\hat\alpha,\hat\beta)}{Var(\hat\beta)} = \bar{x} $ follows from the covariance matrix for the $\alpha$ and $\beta$ which is $\sigma (X^TX)^{-1}$. Which you could work out further by filling in all the terms... But you could also argue that the minimum should be at $\bar{x}$ by transform the data matrix $X$ such that the the column vectors are perpendicular and the estimates $\hat\alpha$ and $\hat\beta$ have zero covariance.

$\endgroup$
3
  • $\begingroup$ This is a great visual - thank you! But now it brings me to ask: Does this suggest that predictions for y have greater error if our input x is far from the mean of x? If our confidence bands are wider at the ends than in the middle, then predicting values of y at those spots is less certain than near the mean, no? $\endgroup$
    – dataphile
    Nov 9 '21 at 14:56
  • 1
    $\begingroup$ @dataphile the error in a prediction of a new datapoint $y$ has two sources of error. (1) The error in the estimate of the mean of the distribution of the data points. This error becomes larger further away from the mean as explained in the answers to the question here. (2) Error due to variation of the samples $y$ around the mean. This error can be (depending on the model) heterogeneous.... ...so yes, the increase of the bands further away from the mean will also happen for predictions, because the error in the estimate of the mean will influence the error of the predicitons. $\endgroup$ Nov 9 '21 at 15:00
  • $\begingroup$ Perfect - thank you so much! I just never made that connection before. $\endgroup$
    – dataphile
    Nov 9 '21 at 15:18
8
$\begingroup$

As you get farther from $\bar x,\bar y$ uncertainty increases. There's fewer and fewer observations when you reach out to distant regions of the domain of your function.

The main source of uncertainty is the one about the slope of the line. Take a look at the drawing here. With the given sample of observations you can say that the best fit line should be somewhere between these two grey lines. The uncertainty around $(\bar x,\bar y)$ is the smallest, but once you step away from where the observations are located, uncertainty increases.

enter image description here

Here's how we can intuitively "derive" the asymptotic confidence interval, i.e. where $x^*$ is very far away from your observations. The confidence given by model MSE will proportionally expand as $|x^*-\bar x|\to\infty$. Think of it as approximate equality of ratios $\frac{MSE}{\sqrt n\sigma_x}\approx \frac{CI(x^*)}{|x^*-\bar x|}$. That's the asymptotic of your formula: $$\lim_{x^*\to\infty} MSE \sqrt{\frac{1}{n} + \frac{(x^*-\bar x)^2}{\sum_{i=1}^n (x_i-\bar x)^2}} =\frac{MSE}{\sqrt n\sigma_x}|x^*-\bar x|$$

enter image description here

$\endgroup$
3
  • $\begingroup$ this is not true for linear regression which has a domain everywhere. Dave's answer is the correct one in this situation $\endgroup$
    – rep_ho
    Nov 8 '21 at 10:05
  • $\begingroup$ maybe you have data between [0.1] and [100,101], but you will still see uncertainty around 50, far from the data, to be small and uncertainty where you have data large $\endgroup$
    – rep_ho
    Nov 8 '21 at 10:51
  • $\begingroup$ @Aksakal, the point of rep_ho is that the reason "there's fewer observations" is not very clear, because the point $\bar{x}$ does not need to be the point with the most observations. You explain it in the comments, but in the answer it is unclear. $\endgroup$ Nov 8 '21 at 12:24
5
$\begingroup$

There is 2 uncertainties here. As you mentioned there is the uncertainty with the slope thus the spreading curve at ends, but there is also an uncertainty at the mean. Yes, the curve is thinnest at the mean but it is not zero. Thus the uncertainty of the slope passing through the mean's distribution causes the estimate to be non linear and generates the above examples.

$\endgroup$
1
  • $\begingroup$ Unveiling about the intercept doesn’t matter in big scheme of things. Consider the infinitely far away $x^*$ $\endgroup$
    – Aksakal
    Nov 8 '21 at 20:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.