1
$\begingroup$

Let's assume a simple linear regression $$ y = \beta_0 + \beta_1 x + \varepsilon $$ where $\varepsilon_i$ are independent and come from $\mathcal{N}(0, \sigma^2)$. We define residuals as $$ e = y - \hat y $$ I know that $$ Var(e_i) = \sigma^2(1-h_{ii}) $$ where $$ h_{ii} = (X(X^TX)^{-1}X^T)_{ii} $$ so residuals are not independent (we know that from the fact $X^T e = 0$ regardless of the presence of the intercept term) and do not have equal variances. Mathematically I understand all the formulas, but my question is: why is that? Why in reality, even for good data like this

x <- rnorm(100, 0, 1)
eps <- rnorm(100, 0, 1)

y <- 1 + 5*x + eps
mod <- lm(y ~ x)

X <- model.matrix(mod)

t(X) %*% solve(t(X) %*% X) %*% t(X)

variances aren't equal and residuals are dependent? What's the underlying reason for that? From the practical point of view.

EDIT: I am aware of the infulence, but the question is why the estimated errors (i.e. residuals) never have equal variances, regardless of how I specify the model matrix $X$.

$\endgroup$
5
  • 1
    $\begingroup$ It is instructive to look at intuitive, computationally simple cases. Consider, for instance, one value of $(0,0)$ and two (or more) values $(1,y_j).$ You don't have to do any arithmetic to know what the residuals are and to see why they have (radically) different variances. $\endgroup$
    – whuber
    Oct 19, 2023 at 19:09
  • 1
    $\begingroup$ You might benefit from considering the idea of influence. Start first with considering how much a small change to an observation influences its own fitted value in direct terms ($\frac{\partial \widehat{y}_i}{\partial y_i} = h_{ii}$), a sense of how directly a fitted value responds to the "pull" of the observation it approximates. Note that typically these are different for different observations -- values not near the mean in x-space have large $h_{ii}$. There's a somewhat related idea in regression diagnostics, DFFIT, where $\text{DFFIT} = \widehat{y}_i - \widehat{y}_{i(i)}$. $\endgroup$
    – Glen_b
    Oct 19, 2023 at 21:24
  • $\begingroup$ which is how much the fit changes if you leave out its observation. See also Ye, J. (1998), "On Measuring and Correcting the Effects of Data Mining and Model Selection", Journal of the American Statistical Association, 93 (441), 120–131, Eqn 7, and note that this general sense of model d.f. is the sum of these observation influences $\frac{\partial \widehat{y}_i}{\partial y_i}$. Different points "use up" more or less of the model's d.f. in pulling the fit toward them. A point with a strong influence on its own fit (under some model) will tend to have its fitted value pulled more toward it. $\endgroup$
    – Glen_b
    Oct 19, 2023 at 21:27
  • $\begingroup$ @whuber Thank you. But still, how can we specify variance for a single residual? We only have one point. $\endgroup$
    – thesecond
    Oct 20, 2023 at 8:34
  • $\begingroup$ (maybe not following the discussion), but you fix the x's and then generate multiple samples of the noise 'eps' for each x $\endgroup$
    – seanv507
    Oct 20, 2023 at 13:01

3 Answers 3

3
$\begingroup$

Say you start with an iid sequence $\{\varepsilon_i\}$. Now using OLS algebra, we have, uniformly in $i$ under OLS assumptions, that $$e_i-\varepsilon_i=O_p\left(N^{-1/2}\right),$$ implying that for a large sample the difference between $e$ and the model error term vanishes such that $e_i$ inhertis the properties of $\varepsilon$.

Note also that uncorrelatedness as noted in your post, does not imply independence. In fact, OLS chooses the model parameters such that the OLS residuals are orthogal to the explanatory variables; this is by deifinition. Further, the variance of $e_i$ in your post is the conditional variance and not the unconditional one (which is not in general tractable).

Addendum: \begin{align} e&=Y-\widehat{Y}\\ &=Y-X\widehat{\beta}\\ &=X\beta+\varepsilon-X\widehat{\beta}\\ &=X\left(\beta-\widehat{\beta}\right)+\varepsilon\\ \end{align} Now, under OLS assumptions we have $E(\beta-\widehat{\beta})=0$ and that $Var(\beta-\widehat{\beta})\propto \frac1N$, which implies $\beta-\widehat{\beta}=O_p(N^{-1/2})$.

$\endgroup$
2
  • $\begingroup$ Thank you. How did you get the formula for $e_i - \varepsilon_i$? And another question: although dependence on sample size seems very rational and intuitive, I don't quite get it. Let's say we have a sample of size $10$, with values $x_1^*, ..., x_{10}^*$ and then "add" $90$ different observations. Why would the corresponding residuals for $x_1^*, ..., x_{10}^*$ change? Is it due to the fact that as there is more data then the impact of each observation averages out? $\endgroup$
    – thesecond
    Oct 20, 2023 at 13:22
  • $\begingroup$ Please see the addendum. Note that "equal variance" would only make sense "asymptotically". In small sample you do not necessarily have the same variances. $\endgroup$
    – Math-fun
    Oct 20, 2023 at 16:05
1
$\begingroup$

Your claim is not true -- there does exist some design matrix $X$ that gives you all equal residual variances. One example is as follows: \begin{align*} X = \begin{bmatrix} 1 & 1 \\ 1 & 1 \\ 1 & -1 \\ 1 & -1 \end{bmatrix}. \end{align*} It is easy to verify that the diagonal elements of $H$ are $\{1/2, 1/2, 1/2, 1/2\}$, which means the variance of each residual is $\sigma^2(1 - 1/2) = \sigma^2/2$.

$\endgroup$
0
$\begingroup$

Not an answer, but reading your post I am puzzled by your conclusion that the estimated $e$ terms are NOT independent, given that the true residuals $\epsilon$ are? Could you explain that please? Thanks.

$\endgroup$
3
  • $\begingroup$ Residuals are only the estimates of unknown true errors. When you perform OLS, you seek for the linear combination $\hat y = Xb$ of columns of the design matrix $X$ such that the distance from $\hat y$ to your observed vector $y$ is minimized. That happens when the vector $e = \hat y - y$ is orthogonal to the column space of $X$. If the vector is orthogonal to the span of $X$, then it is orthogonal to each of the generators. $\endgroup$
    – thesecond
    Oct 23, 2023 at 12:56
  • $\begingroup$ Orthogonality happens when the inner product of two vector is $0$. Recall that for a model with the intercept you have a column of ones (denoted by $\mathbb{1}$) in $X$. Thus $\mathbb{1}e = 0$ what literally means $e_1 + ... + e_n = 0$, so $e_1, ..., e_{n-1} = -e_n$. If e.g. $e_1, ..., e_{n-1}$ are free to vary, then $e_n$ DEPENDS on the values of $e_1, ..., e_{n-1}$. $\endgroup$
    – thesecond
    Oct 23, 2023 at 12:58
  • $\begingroup$ OK Thanks, no I see what you meant. But isn't knowing this an answer to you initial question too? I mean, it's perfectly understandable then that/why such dependence exists, just like the different variances of the e residuals are evident. $\endgroup$
    – BenP
    Oct 23, 2023 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.