1
$\begingroup$

Consider a standard oneway ANOVA model given by $$ Y_{ij}=\theta_i+\varepsilon_{ij} $$ for $i=1,\ldots,k$ and $j=1,\ldots,n_i$, where $\varepsilon_{ij}$ are iid $N(0,\sigma^2)$ random variables. The residuals are defined as $$ e_{ij}=Y_{ij}-\frac1{n_i}\sum_{k=1}^{n_i}Y_{ik} $$ for $i=1,\ldots,k$ and $j=1,\ldots,n_i$. The variance of the residuals is given by $$ \operatorname{Var}e_{ij} =\sigma^2\Bigl(1-\frac1{n_i}\Bigr). $$ This means that the variance of the residuals increases if we increase the sample size $n_i$. Since the residuals $e_{ij}$ essentially estimate the errors $\varepsilon_{ij}$, I would expect the variance of the residuals to be larger than $\sigma^2$ and get closer and closer to $\sigma^2$ as the sample size $n_i$ increases. So the expression of the $\operatorname{Var}e_{ij}$ seems counterintuitive to me.

Why the variance of the residuals increases as the sample size $n_i$ grows?

Any help is much appreciated!

$\endgroup$
6
  • $\begingroup$ You are making a critical derivation after part 2. The key equation is $\text{var}(A-B) = \text{var}(A) + \text{var}(B) - 2 \text{cov}(A, B)$. $\endgroup$
    – AdamO
    May 14 at 19:16
  • $\begingroup$ @AdamO Are you saying that the expression of the variance is incorrect? $\endgroup$
    – Cm7F7Bb
    May 14 at 19:18
  • $\begingroup$ Absolutely 100%. The expression you have is the variance of a prediction. $\endgroup$
    – AdamO
    May 14 at 19:20
  • $\begingroup$ Another error you are making is not calculating the variance of a sum correctly. $\text{var} \sum_{i=1}^n Y_i = n \sigma^2$ for $Y_i$ iid with variance $\sigma^2$. $\endgroup$
    – AdamO
    May 14 at 19:22
  • $\begingroup$ @AdamO If $e_{ij}$'s are defined as $Y_{ij}-n_i^{-1}\sum_{k=1}^{n_i}Y_{ik}$, then the variance is equal to $\sigma^2(1-1/n_i)$. The same expression is given on page 776 of Applied Linear Statistical Models by Kutner et al. I am not asking if the expression is correct. It is correct. I am trying to intuitively understand why the variance increases as the sample size grows. $\endgroup$
    – Cm7F7Bb
    May 14 at 19:23
1
$\begingroup$

$\newcommand{\one}{\mathbf 1}\newcommand{\e}{\varepsilon}$We don't really need multiple groups here. It's enough to just consider $y_i = \theta + \e_i$ and then we can apply this to each group separately. In this case we have $\hat\theta = \bar y$ and $e = y - \bar y\one = (I - n^{-1}\one\one^T)y$ so $\text{Var}(e) = \sigma^2 (I-H)$ where I've set $H = n^{-1}\one\one^T$. This does indeed mean $\text{Var}(e_i) = \sigma^2 (1 - 1/n)$.

I think the key idea is that the variation of the $y_i$ around the sample mean $\bar y$ is less than around the true mean of $\theta$, since the sample mean is the minimizer of $\sum_{i=1}^n (y_i - c)^2$ over $c \in \mathbb R$. This is why $\text{Var}(e_i) < \text{Var}(\e_i)$. But as $n\to\infty$ we'll have $\bar y \to_{\text{as}} \theta$ so the variation of $y_i$ around $\bar y$ approaches the variation of $y_i$ around $\theta$, which is precisely $\e$. That's why $\text{Var}(e_i)\nearrow\text{Var}(\e_i)$.

Another aspect is that the residuals are not independent as $\text{Cov}(e_i,e_j) = -\sigma^2/n$ for $i\neq j$. This is because they are constrained to a subspace (the space orthogonal to $\one$, in this case) so knowing what $e_1$ is, say, reduces the possible variability in $e_j$ for $j\neq 1$. This effect also decreases with $n$ and the residuals are increasingly independent (since they're multivariate Gaussian here; more generally, they'll be increasingly uncorrelated).

$\endgroup$
2
  • $\begingroup$ Is the covariance between a single observation and the sample mean really 0? $\endgroup$
    – AdamO
    May 15 at 21:29
  • $\begingroup$ @AdamO no? Which part of my answer are you referring to? $\endgroup$
    – jld
    May 18 at 0:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.