4
$\begingroup$

Consider the least squares problem $Y=X\beta +\epsilon$ while $\epsilon$ is zero mean Gaussian with $E(\epsilon) = 0$ and variance $\sigma^2$. I need to prove that

$\frac{V(\hat{\beta})}{N-(n+m)}$ is an unbiased estimate of $\sigma^2$ with $V(\beta) = ||Y-X\beta||$ .

I wasn't able to find the answer online. I just got confused by a thousand different ways to write things down.

EDIT:
$Y = \begin{pmatrix} y(0)\\ \vdots \\ y(N-1)\end{pmatrix} \quad$ $X = \begin{pmatrix} x^T(0)\\ \vdots \\ x^T(N-1)\end{pmatrix}\quad $ $\beta = \begin{pmatrix} a_1\\ \vdots \\ a_n\\ b_1 \\\vdots \\ b_m \end{pmatrix}$

$\endgroup$
5
  • 1
    $\begingroup$ What are N, n, and m? $\endgroup$
    – ocram
    Nov 16, 2013 at 16:39
  • $\begingroup$ Sorry, I updated my question $\endgroup$ Nov 16, 2013 at 16:45
  • 1
    $\begingroup$ I still don't quite follow your $n$ & $m$, & the way you are representing the matrices is unusual for me. I'm more familiar w/:$$Y=\begin{pmatrix}y_1\\ \vdots\\ y_N\end{pmatrix},\quad X=\begin{pmatrix}1 &x_{11}&\cdots&x_{1p}\\ \vdots&\vdots&\ddots&\vdots\\ 1 &x_{N1}&\cdots&x_{Np}\end{pmatrix},\quad\beta=\begin{pmatrix}\beta_0\\ \vdots\\ \beta_p\end{pmatrix},\quad\varepsilon=\begin{pmatrix}\varepsilon_1\\ \vdots\\ \varepsilon_N\end{pmatrix}$$ $\endgroup$ Nov 16, 2013 at 17:05
  • $\begingroup$ The notation was given to me (at the university), but it is quite the same as x are vectors and p=m+n.. $\endgroup$ Nov 16, 2013 at 17:07
  • 1
    $\begingroup$ On the basis of this comment combined with details in your question, I've added the self-study tag. Please read its tag wiki info and understand what is expected for this sort of question and the limitations on the kinds of answers you should expect. While you can ask about course-related work (or even work you're just doing for your own study purposes), CV isn't a site to just do your study for you. $\endgroup$
    – Glen_b
    Nov 16, 2013 at 17:20

1 Answer 1

6
$\begingroup$

Following your notations, we have $$V(\hat{\beta}) = \|\hat{\epsilon}\|^2 = \text{RSS}$$ i.e., the Residual Sum of Squares.

It is a fact that (cf. here) $$\frac{\text{RSS}}{\sigma²} \sim \chi_{(N-p)}^2$$ with $N$ the total sample size and $p$ the number of parameters in $\beta$ (here, $p = n + m$).

The result follows from the fact that the expectation of a chi-square random variable equals its number of degrees of freedom, i.e., $$ \text{E}\left(\frac{\text{RSS}}{\sigma²}\right) = N - p $$ which can be rewritten as $$ \text{E}\left(\frac{\text{RSS}}{N-p}\right) = \sigma² $$ since $N-p$ and $\sigma²$ are both non-random.

$\endgroup$
2
  • $\begingroup$ I cant follow why $V(\hat{\beta})$ is $||\hat{\epsilon}||^2$. What exactly is $\hat{\epsilon}$? $\endgroup$ Nov 16, 2013 at 17:13
  • $\begingroup$ $\|v\| = \sum_{\ell=1}^L v_\ell^2$ for any vector $v=(v_1 \dotsc v_L)$. Thus $V(\hat{\beta}) = \|Y - X \hat{\beta}\|$ is the sum of squared residuals, which I have denoted by $\|\hat{\epsilon}\|$. $\endgroup$
    – ocram
    Nov 16, 2013 at 17:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.