5
$\begingroup$

My question is, as far as i am aware,
1. the residuals($\hat{u_i}$) are not independent of one another
2. the variance of ith residual is $\sigma\{(1-1/n-(X_i-\overline{X})/\Sigma(X_i-\overline{X})^2\}$

So since each $\hat{u_i}$ isn't independent and their variance does not equal $\sigma^2$, $\hat{u_i}^2/\sigma^2$ can't be a square of i.i.d standard normal variable. Then how can the sum of $\hat{u_i}^2/\sigma^2$ distributed chi squared?

(**I'm not familiar with matrix notations, so I will appriciate it if you could keep it within Simple Regression context)

$\endgroup$
3

1 Answer 1

7
$\begingroup$

The correct expression for the variance of the $i$th residual is explained here in detail.

I am using a slightly different notation but in the end it will all match with what you are working with.

Suppose we have a simple linear regression model $$y=\alpha+\beta x+\epsilon$$

where $\alpha+\beta x$ is the part of $y$ explained by $x$ and $\epsilon$ is the unexplained part or the error. Here $y$ is stochastic and $x$ is non-stochastic.

We consider the paired observations $(x_i,y_i)$ for $i=1,2,\ldots,n$ and assume that $\epsilon_i$ are i.i.d $\mathcal N(0,\sigma^2)$ for all $i$. This means we have some $Y_i\sim\mathcal N(\alpha+\beta x_i,\sigma^2)$, independently for all $i$.

Define $$s_{xx}=\sum_{i=1}^n (x_i-\bar x)^2\qquad,\qquad s_{yy}=\sum_{i=1}^n (y_i-\bar y)^2$$ and $$s_{xy}=\sum_{i=1}^n (x_i-\bar x)(y_i-\bar y)$$

From the normal equations, we have the least square estimates of $\alpha$ and $\beta$ :

$$\hat\alpha=\bar y-\hat\beta \bar x\qquad,\qquad\hat\beta=\frac{s_{xy}}{s_{xx}}$$

Let the residual variance be $$s^2=\frac{1}{n-2}\sum_{i=1}^n(y_i-\hat\alpha-\hat\beta x_i)^2$$

On simplification,

\begin{align} (n-2)s^2&=\sum_{i=1}^n (y_i-\hat\alpha-\hat\beta x_i)y_i \\&=\sum_{i=1}^n \left\{(y_i-\bar y)-\hat\beta(x_i-\bar x)\right\}y_i \\&=\sum_{i=1}^n (y_i-\bar y)y_i-\hat\beta \sum_{i=1}^n (x_i-\bar x)y_i \\&=\sum_{i=1}^n (y_i-\bar y)^2-\hat\beta\sum_{i=1}^n (x_i-\bar x)(y_i-\bar y) \\&=s_{yy}-\hat\beta s_{xy} \\&=s_{yy}-\hat\beta^2 s_{xx} \end{align}


Taking $\alpha'=\alpha+\beta\bar x$, joint pdf of $Y=(Y_1,\ldots,Y_n)$ for $(y_1,\ldots,y_n)\in\mathbb R^n$ is

$$f_{Y}(y_1,\ldots,y_n)=\frac{1}{(\sigma\sqrt{2\pi})^n}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n\left(y_i-\alpha'-\beta(x_i-\bar x)\right)^2\right]$$

Consider the orthogonal transformation $(y_1,\ldots,y_n)\to(z_1,\ldots,z_n)$ such that

$$\begin{pmatrix}z_1\\z_2\\\vdots\\z_n\end{pmatrix}=\mathbf Q\begin{pmatrix}y_1\\y_2\\\vdots\\y_n\end{pmatrix}\,,$$

where $$\mathbf Q=\left[\begin{matrix}\frac{1}{\sqrt n}&\frac{1}{\sqrt n}&\cdots&\frac{1}{\sqrt n}\\\frac{x_1-\bar x}{\sqrt{s_{xx}}}&\frac{x_2-\bar x}{\sqrt{s_{xx}}}&\cdots&\frac{x_n-\bar x}{\sqrt{s_{xx}}}\\\vdots&\vdots&\cdots&\vdots\end{matrix}\right]$$ is an $n\times n$ orthogonal matrix with its first two rows fixed.

Then, $$z_1=\frac{1}{\sqrt n}\sum_{i=1}^n y_i=\sqrt{n}\bar y$$ and $$z_2=\frac{\sum (x_i-\bar x)y_i}{\sqrt{s_{xx}}}=\frac{s_{xy}}{\sqrt{s_{xx}}}=\hat\beta\sqrt{s_{xx}}$$

Note that $\sum\limits_{i=1}^n y_i^2=\sum\limits_{i=1}^n z_i^2$ by virtue of orthogonal transformation, which leads to

\begin{align} \sum_{i=1}^n(y_i-\alpha'-\beta(x_i-\bar x))^2&=\sum_{i=1}^n y_i^2+n\alpha'^2+\beta^2\sum_{i=1}^n(x_i-\bar x)^2-2\alpha'n\bar y-2\beta\sum_{i=1}^n(x_i-\bar x)y_i \\&=\sum_{i=1}^n z_i^2 +n\alpha'^2+\beta^2 s_{xx}-2\alpha'\sqrt n z_1-2\beta z_2\sqrt{s_{xx}} \\&=(z_1-\sqrt n\alpha')^2+(z_2-\beta\sqrt{s_{xx}})^2+\sum_{i=3}^nz_i^2 \end{align}

For $(z_1,\ldots,z_n)\in\mathbb R^n$, joint density of $Z=(Z_1,\ldots,Z_n)$ becomes

$$f_{Z}(z_1,\ldots,z_n)=\frac{1}{(\sigma\sqrt{2\pi})^n}\exp\left[-\frac{1}{2\sigma^2}\left\{(z_1-\sqrt n\alpha')^2+(z_2-\beta\sqrt{s_{xx}})^2+\sum_{i=3}^nz_i^2\right\}\right]\,,$$

so that $Z_1,Z_2,\ldots,Z_n$ are independently distributed with

\begin{align} Z_1&\sim\mathcal N(\sqrt n\alpha',\sigma^2) \\Z_2&\sim\mathcal N(\beta\sqrt{s_{xx}},\sigma^2) \\Z_i&\sim\mathcal N(0,\sigma^2)\qquad,\,i=3,4,\ldots,n \end{align}

Now,

\begin{align} (n-2)s^2&=s_{yy}-\hat\beta^2s_{xx} \\&=\sum_{i=1}^ny_i^2-n\bar y^2-\hat\beta^2s_{xx} \\&=\sum_{i=1}^nz_i^2-z_1^2-z_2^2 \\&=\sum_{i=3}^nz_i^2 \end{align}

And you have that $Z_3,\ldots,Z_n\sim\mathcal N(0,\sigma^2)$, independently.

This implies $$\sum_{i=3}^n\frac{Z_i^2}{\sigma^2}\sim\chi^2_{n-2}$$

Or in other words, $$\frac{(n-2)s^2}{\sigma^2}\sim\chi^2_{n-2}$$

$\endgroup$
12
  • $\begingroup$ Oh wow...now i see that i should have first asked why the df is actually n-2 not n... Thank you so much! $\endgroup$ Aug 18, 2018 at 9:18
  • $\begingroup$ @user8931048 One could actually ask why $s^2$ is defined with $n-2$ instead of $n$. I think it has to do with $s^2$ being unbiased for $\sigma^2$. $\endgroup$ Aug 18, 2018 at 9:36
  • $\begingroup$ How was $z_3$ calculated? Its term was not mentioned in the transformation matrix? $\endgroup$ Nov 25, 2018 at 7:22
  • 1
    $\begingroup$ Hi @StubbornAtom, can you please tell me why did we multiple $\mathbf{y}$ by an orthogonal matrix $\mathbf{Q}$ and not any other matrix that is not necessarily orthogonal? I've read that the norm of a vector is invariant under multiplication by an orthogonal matrix? What is the intuition behind the orthogonal transformation here? Thanks in advance. $\endgroup$ May 1, 2020 at 8:12
  • 1
    $\begingroup$ @GoldmanClarck, Yes, orthogonal transformation preserves norm which is being used here: $$\sum_{i=1}^n Z_i^2=Z^TZ=(QY)^T(QY)=Y^TQ^TQY=Y^TY=\sum_{i=1}^n Y_i ^2$$ In addition, if $Y$ has a multivariate normal distribution $N(\mu,\sigma^2 I_n)$, then $QY\sim N(Q\mu,\sigma^2I_n)$ for orthogonal $Q$. So normality is also preserved. $\endgroup$ May 1, 2020 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.